Matchmoving is a technique that allows computer graphics to be inserted into live-action footage with correct position, scale, orientation, and motion. Also known as motion tracking, it's what allows movie monsters to run down Main Street and robots to run through crowds—and look real. Now this unique book from a top expert from Industrial Light and Magic teaches you the art of matchmoving.
With step-by-step tutorials and pages of examples, this book first explains the basics and then shows you professional techniques, from 3D calibration and tracking, to stereoscopy, and more.
- Explains concepts and teaches professional techniques for successful matchmoving
- Authored by a top matchmove specialist from Industrial Light and Magic, who walks you through step-by-step tutorials and impressive examples
- Covers matchmoving basics, 2D tracking, 3D calibration and tracking, automatic tracking, cameras, integrating matchmoves, and stereoscopy
Learn how studio visual effects professionals make all the right matchmoves with Matchmoving: The Invisible Art of Camera Tracking 2nd Edition.
Matchmoving is a technique that allows computer graphics to be inserted into live-action footage with correct position, scale, orientation, and motion. Also known as motion tracking, it's what allows movie monsters to run down Main Street and robots to run through crowds—and look real. Now this unique book from a top expert from Industrial Light and Magic teaches you the art of matchmoving.
With step-by-step tutorials and pages of examples, this book first explains the basics and then shows you professional techniques, from 3D calibration and tracking, to stereoscopy, and more.
- Explains concepts and teaches professional techniques for successful matchmoving
- Authored by a top matchmove specialist from Industrial Light and Magic, who walks you through step-by-step tutorials and impressive examples
- Covers matchmoving basics, 2D tracking, 3D calibration and tracking, automatic tracking, cameras, integrating matchmoves, and stereoscopy
Learn how studio visual effects professionals make all the right matchmoves with Matchmoving: The Invisible Art of Camera Tracking 2nd Edition.
eBook
Available on Compatible NOOK devices, the free NOOK App and in My Digital Library.
Related collections and offers
Overview
Matchmoving is a technique that allows computer graphics to be inserted into live-action footage with correct position, scale, orientation, and motion. Also known as motion tracking, it's what allows movie monsters to run down Main Street and robots to run through crowds—and look real. Now this unique book from a top expert from Industrial Light and Magic teaches you the art of matchmoving.
With step-by-step tutorials and pages of examples, this book first explains the basics and then shows you professional techniques, from 3D calibration and tracking, to stereoscopy, and more.
- Explains concepts and teaches professional techniques for successful matchmoving
- Authored by a top matchmove specialist from Industrial Light and Magic, who walks you through step-by-step tutorials and impressive examples
- Covers matchmoving basics, 2D tracking, 3D calibration and tracking, automatic tracking, cameras, integrating matchmoves, and stereoscopy
Learn how studio visual effects professionals make all the right matchmoves with Matchmoving: The Invisible Art of Camera Tracking 2nd Edition.
Product Details
| ISBN-13: | 9781118529669 |
|---|---|
| Publisher: | Wiley |
| Publication date: | 10/26/2012 |
| Sold by: | JOHN WILEY & SONS |
| Format: | eBook |
| Pages: | 336 |
| File size: | 59 MB |
| Note: | This product may take a few minutes to download. |
About the Author
Tim Dobbert is a layout lead at Industrial Light&Magic in San Francisco and has been in the visual effects industry for over ten years. He has worked as a matchmover and digital artist on over 25 feature films, including Avatar, Pirates of the Caribbean: On Stranger Tides, and Harry Potter and the Goblet of Fire. In addition to teaching university classes in matchmoving and image-based content creation, he has lectured all over the world on the subject.
Read an Excerpt
Matchmoving
By Tim Dobbert
John Wiley & Sons
ISBN: 0-7821-4403-9Chapter One
The Basics of MatchmovingAnytime a computer-generated (CG) element needs to be placed into a live-action sequence or vice versa, a matchmove is required. But what exactly is matchmoving?
Matchmoving is the process of matching CG elements into live-action footage. As a result, it's a crucial part of many visual effects shots. Despite its importance, it is completely invisible in the final shot-that is, if it's done right.
In this chapter, I explain the key steps involved in a matchmove and how matchmovers work with the rest of the visual effects team. I've also included a tutorial that is designed to help you get comfortable working with cameras and perspective.
A Typical Matchmove
In order to better understand what matchmovers do, let's consider a typical visual effects shot. The director has called for a CG creature to crash out of window of a building and run across the street and into an alley. Because the monster will need to interact with the window, the visual effects supervisor decides that in addition to the monster, the window-shattering effect should also be done in the computer.
On the day of shooting, the director makes artistic decisions as to how he wants to shoot the scene and eventually decides on a camera position that he likes. There is an opening on the building where the window should be, although the panes of glass are missing. The director and the cameraman practice the camera move a few times and watch the video playback to see how it looks. When they are filming, they move the camera as though it were following the monster crashing through the window and running across the street, even though the monster isn't there. Extras react to the imaginary beast, and props around the window are rigged with monofilament string (fishing line) to be pulled down on cue as though they were knocked over. Once the director is happy with the shot, the film is sent off to be digitized and then given to the visual effects artists to add the monster.
When the visual effects studio receives the digitized sequence (known as a plate), they decide that they will need an animator to animate the creature and a technical director (TD) to do the glass-shattering effect. And, of course, they'll need a matchmover to matchmove the plate.
The visual effects artists' goals are to make their 3D elements look as realistic as the scene that was filmed. The animator will need to make the creature move as though it were really crashing through a window, and the TD will need to make the window shatter like a real window. The matchmover needs to figure out where the camera was and how it was moving when the scene was filmed.
Matchmovers play an important role in this case, because in order for the creature and window to appear matched realistically with the scene, they need to make sure that the CG objects are "filmed" the same way with their CG camera as the real set was filmed with the real camera. Consider the window that needs to shatter-if the perspective of the window doesn't match the perspective in the plate, it will look out of place. Furthermore, if the real camera moves to the left and the CG window stays put, everyone will know it's a fake.
In our example effects shot, the visual effects supervisor measures key items on the set. For example, she measures the size of the opening of the window as well as its height off the ground. She measures the distance across the street and the size of the opening to the alleyway. She draws a rough picture of the set and makes notes about positions of certain props and lights that might be useful to know. She also measures how high the camera is off the ground, what lens is used, and how far it is from the window.
Typically the animator, TD, and matchmover all start at the same time. Since there are some measurements of the set, the animator knows how high the creature needs to jump to get through the window and how far to make it run across the street. The TD knows the size of the window that needs to shatter and how high it is off the ground. This is enough information to allow them to start setting up their scenes.
While they're doing that, the matchmover starts by first examining the footage to get an idea of how the camera was moving during the shot. He brings the footage into his matchmoving software and begins to track the 2D features in the scene as they move around the screen. 2D tracking usually involves identifying things in the scene that don't move (such as the corner of a building) and then letting the software follow that feature as the footage plays.
Once the matchmover has tracked a number of 2D tracks, the software analyzes these tracks and computes the position of the camera in relation to the items in the scene. At the end of this process, the matchmover exports a scene to his 3D-animation package that includes an animated camera and the 3D positions for all of the features he tracked.
Once the matchmover is happy with the camera he has generated, he goes about fitting that camera into a CG scene that is the same size as the one the animator and TD are using. When he's finished, he is able to look through his CG camera at the CG window and creature, and they appear in the right perspective and scale when compared to the original live-action footage.
Finally, he saves the scene with the matchmove camera in it. The animator and TD both use this camera to render their animations. If they've placed their window and creature in the right place in the environment, then they don't need to worry about whether the perspective matches or whether movement of the camera is the same. As long as they've rendered it using the matchmove camera, it will appear matched into the footage (Figure 1.1).
From 3D to 2D and Back Again
No discussion of matchmoving would be complete without discussing cameras. For matchmovers, it is important to understand how real-world cameras work and how they make the images we see on the screen. I'll discuss cameras more in-depth in Chapter 6, "Cameras," but for now there are a few basic concepts about real-world cameras that are important to know before getting started.
When a real camera films a scene, it is basically doing one thing: capturing the three-dimensional world as a two-dimensional image. That is, it gathers light from the 3D world around us and records it in a 2D form, such as a piece of film or a digital image. Let's consider for a moment exactly how this happens.
The light from the scene passes through the camera lens and is focused onto film that rests in the film gate on the back of the camera's inner chamber. The shutter closes, the film advances, the shutter opens again, and the process repeats. In the case of digital cameras, the film and film gate are replaced by a CCD (Charge-Coupled Device) that electronically captures the light information and records it to some sort of memory device.
The cameras in a 3D-animation program are based on real cameras, but they are represented in a slightly different manner. 3D-animation cameras represent a mathematically perfect model of the optics and construction of a real camera. Like real-world cameras, they have a focal length and a film back (the equivalent of a film gate). But rather than capturing light from the real world, they are simply capturing information of the synthetic, computer-generated environment in which they have been placed.
Whether you're dealing with exposed film or a 3D render, the resulting image is a projection. That is, the three-dimensional scene is flattened out into a two-dimensional representation of that scene. We have become so accustomed to these flattened images that we hardly notice them anymore, but every time we watch TV or a film, we are watching a flat recording of a three-dimensional scene (Figure 1.2).
The Matchmoving Process
So if a camera's purpose is to take the three-dimensional world and make a twodimensional image, a matchmover's job is the exact opposite. A matchmover must take a two-dimensional image and create a three-dimensional world. The portal between these two halves is the camera. If information about the camera can be reconstructed, it will go a long way toward figuring out how the 3D environment must have been at the time of filming. This information-the 3D environment and the camera-is what a matchmover will ultimately deliver to the animators to work with.
A matchmover's workflow (Figure 1.3) generally follows the same pattern for each shot, although there are a variety of ways to complete each task. The figure shows a typical matchmove pipeline. In the following sections, we'll examine each of these steps more closely.
Evaluate the Footage
This is perhaps the most important step in the matchmoving process, and unfortunately it is often overlooked. There are many ways to solve a matchmove, and careful scrutiny of the plate can help decide the appropriate tool to use, what pitfalls to watch out for, and how long it might take. This last item is particularly important, since clients and supervisors often put it at the top of their list of questions.
One reason the evaluation process is so often glossed over is that many of the things that determine the difficulty of matchmove require some experience to judge. Some typical questions asked during the evaluation of a shot could include:
What does the camera seem to be doing? Is it moving, and if so, how? Is it locked off or panning? How fast is it moving?
What is visible in the shot? Are there tracking markers? Is anything blocking the markers?
What format is the plate? Was it shot on film? DV? HD? Is there excessive compression, grain, or noise on the images?
What needs to be placed in the shot? How accurate does it have to be?
Who will be using the matchmove, and how will they be using it?
Of course, these only scratch the surface, but the more questions you ask, the more you will know what you need to do. The tutorials in this book are designed to help you learn what these key questions are and how to deal with their implications. A more thorough list of questions are included in Appendix A. The Evaluation Checklist there can be used as a guideline to help determine the difficulty of a matchmove.
Applying Information
As I've said, solving a matchmove can be like solving a puzzle (it's no coincidence that it's referred to as "solving" a matchmove). And as such, the more information there is, the easier it should be to achieve good results.
The amount of information given to matchmovers can run the gamut. There might only be an image sequence and nothing else, or perhaps someone was allowed on set to record all the camera information and take measurements. Usually it's somewhere in between. But the good news is that a surprisingly small amount of data can go a long way toward solving the matchmove.
The following are typical data a matchmover might include:
Camera information Such as focal length, aperture, and film type.
Set measurements Including camera height, focus distance, and measurements of various items in the shot.
Survey data This is very detailed measuring of the set, usually done by a professional surveyor.
Define the Camera
As stated before, the matchmover's job is to define all of the internal and external parameters of the camera, and there are many ways to do that. Knowing which method to use and under what circumstances comes with experience, but sometimes the only way to solve the matchmove is to experiment and see what works best. In broad terms, there are two major ways of solving for the camera: manual and automatic.
The manual methods harken back to the days before software existed to help matchmovers. This category would include perspective matching (matching the perspective of a single background image, rather than an image sequence, which is covered later in this chapter) and old-fashioned hand-tracking. This method of tracking a sequence involves making a speculation as to the camera's position and then refining it over many iterations until a match is achieved. Tracking a camera by hand is no small feat. It can often take weeks to truly figure out what is happening, because the process is nothing more than making educated guesses and tweaking them until they work.
In the past five or six years, software has emerged that allows a matchmover to track cameras somewhat automatically using a sophisticated technology known as photogrammetry. These software packages (which are covered in the next chapter) usually have a similar workflow to them. First, features in the image such as props in the scene or tape markers (commonly used on blue screen footage to mark points on the wall) are "tracked" as they move around the image in 2D. Then the software performs a calibration (or solve) for the camera by mathematically analyzing the movements of the 2D tracking markers. These packages usually generate an animated camera and 3D markers or nulls that represent the three-dimensional location of features that have been tracked in 2D. Matchmovers use this method most often since it is the easiest way to achieve a solution.
Some methods borrow from both manual and automatic techniques. Oftentimes, these are customized solutions that revolve around both types of workflows. For example, the 2D tracking information from matchmoving software could be used with a custom script that allows the matchmover to solve for pan shots.
Set Fitting
While cameras are the primary concern, they are only half of the process of matchmoving. Matchmovers must not only uncover all the facts about the camera, but they must also reconstruct the spatial layout of the environment on the live-action plate.
Figure 1.4 shows why this information is important. The first image shows an incorrect camera and building placement-what a mess. The second image has the correct camera, but the building is too close to the camera; therefore it doesn't match. But notice the third image. In this case, the building is in the correct position and distance from the camera, but since the camera isn't correct, the building still doesn't line up. The last image shows how it should be matched up with the correct camera and building placement. These images illustrate how matchmoving is not just solving cameras, and not just solving environments, but also figuring out the relationship between the two.
How much of the environment does the matchmover need to reproduce? That depends on what is being placed into the footage. If it is simply a character walking by, the animators and TDs might only need a simple ground plane. Other scenes might need rough geometry in order to cast 3D shadows. In some cases, such as digital set extensions, the matchmove might require an extremely accurate camera, detailed geometry, and spot-on positioning. Before beginning a matchmove, it is important to find out what type of 3D object is going into the scene and exactly where it will be placed.
3D environments might come from a variety of sources. Oftentimes, matchmovers create the rough geometry themselves or are provided a set to "fit" into the plate. And in some situations, the matchmover may provide a rough set from which a more detailed set is later constructed by a modeler. Many times, matchmovers use 3D markers they've calculated during the matchmove to provide information about the spatial relationships of the scene. But regardless of where the information comes from, it is often the matchmover's responsibility to establish the environment and set up the scene so that other artists further down the production pipeline don't need to worry about it.
Testing the Matchmove
Once the matchmove is solved, it needs to be tested for accuracy. A bad matchmove usually shows up as an obvious disconnect between the live-action plate and the CG elements. For example, the CG element seems to follow the motion and rotation of the live-action scene, but then suddenly the CG element pops to another location or gradually drifts away from the feature it's supposed to be resting on. Testing the matchmove consists of compositing the 3D objects over the image sequence and watching as the sequence moves to see if there are any unusual pops, drifts, or jitter.
(Continues...)
Excerpted from Matchmoving by Tim Dobbert Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Introduction xxiChapter 1 The Basics of Matchmoving 1
Exploring a Typical Matchmove 2
Moving from 3D to 2D and Back Again 4
Understanding the Matchmoving Process 5
Evaluating the Footage 5
Applying Information 6
Defining the Camera 6
Fitting the Set 7
Testing the Matchmove 9
Delivering the Scene 9
Matchmoving in the Production Pipeline 9
Perspective Matching Tutorial 10
Gathering the Data 11
Setting Up the Camera 13
Adding Rough Geometry and Refining the Camera 15
Creating a Camera Rig 16
Evaluating and Adjusting the Camera 18
Chapter 2 Matchmoving Software 23
Using Matchmoving Programs 24
Introducing Photogrammetry 24
Calibration 25
The Optics of Photography 26
Projection 28
Building a Better Camera 31
Parallax: A Matchmover’s Best Friend 38
Understanding How Matchmoving Programs Work 40
Requirements for Matchmoving Programs 41
Chapter 3 2D Tracking 43
Understanding the 2D Tracking Process 44
Exploring the Anatomy of a 2D Track 45
Track Placement: Making Every Track Count 47
Sample the 3D Space 47
Maintain the Minimum Number of Tracks 48
Track Stationary Objects 49
Track “True” 3D Positions 50
Getting Tracks to Stick 51
Following Hints for Effective 2D Tracking 52
Handling Plate Issues 54
Motion Blur 55
Soft Focus or Rack Focus 56
Occlusion 56
Noise 58
Low-Contrast Features 58
Lack of Trackable Features 59
Optimizing the Plate for 2D Tracking 61
2D Tracking Tutorial 62
Evaluating the Shot 63
Placing the First Tracks in SynthEyes 63
Choosing Your Tracks 66
Adding the Remaining Tracks 66
Chapter 4 3D Calibration 69
Understanding “Good” Calibration 70
Evaluating Error 70
Calibrating Your Cameras 72
Finding the Right Fit 72
Calibration Tutorial 73
Evaluating the Solution 77
Evaluating the 3D Nulls 77
Evaluating the 3D Space 78
Rendering the Matchmove 79
When Good Solutions Go Bad 80
Determining Camera Moves 81
Dolly/Crane/Moving Camera 81
Pan/Tilt Shots 82
Slight Dollies/Translation 82
Lock-offs 83
Zoom Shots 83
Shaky Footage/Handheld Footage 84
Setting Up a Coordinate System 84
Coordinate System Tutorial 85
Exporting the Scene 88
Chapter 5 Automatic Tracking 91
Tracking on Autopilot 92
Editing Automatic Tracks 93
Using Masks 96
Reaping the Benefits of Automatic Tracking 97
Noise Reduction 97
Environment Definition 98
Implementing the Automatic Tracking Process 99
Automatic Tracking Tutorial 99
Chapter 6 Cameras 107
How Film Cameras Work 108
Lenses 110
Focal Length 111
Focus 113
Nodality 113
Film Backs 115
Film Back and Focal Length 115
Digital Cameras 117
CMOS, CCD, and Rolling Shutter 119
Camera Information 121
Format 122
Lens Distortion 123
Anamorphic Distortion 127
Capture and Digitization 128
Video 128
Film 129
HD 130
Keeping It All Straight 130
Chapter 7 Set Fitting 133
Fitting the Camera 134
Checking the Matchmove 135
Proxy Geometry 135
Checkerboards 137
Fitting the Set 138
Moving the Camera 138
Scaling the Camera 139
Creating Camera Rigs 141
Set-Fitting Tutorial 141
Examining the Exported Scene 142
Adding Test Geometry 143
Moving the Camera 144
Texturing the Objects 149
Performing the First Tests 150
Placing the Matchmove in the Set 152
Altering the Coordinate System 156
Delivering the Scene 156
Tutorial Continued 157
Chapter 8 Being on Set 159
Getting the Right Information 160
Getting Camera Information 162
Before the Shoot 162
During the Shoot 162
After the Shoot 165
Marking Up the Set 165
Tracking Markers for a Green Screen or Blue Screen 166
Tracking Markers for Set Pieces 167
Tracking Markers for Exterior Scenes 168
Working with Others on Set 168
Building a Matchmoving Kit 172
Shooting Distortion Grids 173
Chapter 9 Matchamation 175
Understanding the Basic Technique 176
Establishing the Distance to the Camera 176
Setting the Initial Pose 177
Using Nonlinear Animation Techniques 178
Analyzing the Movement 178
Making the First Pass 179
Making Additional Passes/Tweaks 180
Making Model Changes 181
Working with Imperfect Data 181
Matchamation Tutorial 182
Setting Up the Camera 182
Creating the Proxy Object 184
Creating a Matchamation Rig 185
Analyzing the Motion 186
Making the First Pass 188
Making the Second Pass 191
Making Additional Passes 194
Chapter 10 Troubleshooting and Advanced Techniques 197
Performing Effective Troubleshooting 198
Following a Troubleshooting Checklist 199
Check the Camera’s Motion 199
Check Set Placement and Fitting 201
Scrutinize 2D Tracking Closely 202
Check for Plate Issues 202
Diagnosing the Problem 203
No nulls are visible after calibration 204
Nulls are all in a line or a single point 204
The camera path jumps suddenly to another position or is extremely jagged and chaotic 204
Everything lines up on the first frame but quickly slips off 205
Noise or jitter appears in the camera’s motion path 205
Objects appear to tear away from the plate feature during areas of heavy motion blur 206
Using Advanced Tools and Tricks 206
Using Advanced 2D Tracking Tools 206
Using Constraints 207
Working with Reference Photos 209
Forcing a Calibration 210
Faking the Focal Length 210
Using Survey Data 211
Chapter 11 Object Tracking 215
The Object-Tracking Process 216
Understanding Object Tracking vs Matchamation 216
Following the Object-Tracking Workflow 216
Converting a Camera Track to an Object Track 217
Object-Tracking Tutorial 219
Common Problems with Object Tracking 229
2D Tracking Problems 229
Difficult Object Motion 230
Object Not Truly Rigid 231
Tracking Human Faces 232
Geometry Tracking 233
Geometry-Tracking Tutorial 234
Common Problems with Geometry Tracking 237
Advanced Object Tracking 237
Moving Camera/Moving Object 238
Tracking Deforming Objects 239
Chapter 12 Multipurposing Matchmoving Data 241
Creating Special Cameras 242
Extending or Enhancing Matchmoves 242
Blending Matchmoves 243
Blending Cameras in Maya 245
Object Track as a Camera Track 251
Matchmoves for Paint Work: “Patchmoves” 252
Matchmoves for Compositors 253
Matchmoves for TD Work 254
Not the End of the Road 256
Chapter 13 Stereo 257
Understanding Stereoscopy 258
Interocular Distance 259
Convergence 261
How IO and Convergence Affect Stereo Depth 264
Adjusting Stereo Parameters in Postproduction 265
Stereo Postconversion 266
Filming Stereo Footage 266
Side-By-Side Rigs 266
Beam-Splitter Rigs 267
Dual-Lens Cameras 269
Viewing Stereo Footage 269
Anaglyphs 270
Polarized Lens Systems 271
Modern Stereo Projection 271
Active Shutter Systems 272
Matchmoving Stereo Footage 272
2D Tracking 273
Solving 273
Set Fitting 275
Evaluating the Matchmove 275
Overcoming Common Problems 276
Problems with 2D Tracking 276
Problems with 3D Tracking 277
Focal-Length Issues 277
Appendix A Resources 279
Appendix B Matchmoving Software and Useful Links 281
Matchmoving Software 282
Other Useful Programs 283
Books 283
Websites 284
Appendix C Film Formats 285
16 mm Formats 286
35 mm Formats 286
Large Formats 287
Digital Formats 287
Calculating the Film Back for Cropped Images 288
Glossary 289
Index 299