How does motion capture work?

Motion capture (mocap) records how a real person moves, so a computer character can move the same way. The most accurate kind is optical mocap. It happens in a capture volume: an empty stage ringed by special cameras, often 12 to 48 of them.

The performer wears a tight black suit with about 40 to 60 markers: small balls covered in retroreflective tape, the same stuff as on road signs and safety jackets. Each camera has a ring of infrared LEDs around its lens. The tape bounces that light straight back into the lens, so the markers shine brightly while everything else looks dark.

So a mocap camera does not film a picture of the actor. It sees only bright dots on black, finds the centre of each dot, and sends those 2D positions to the computer, about 120 times a second. Look at the four camera views: each camera sees a different pattern, and some dots are missing because the body is in the way.

A single camera cannot tell how far away a dot is. It only knows the direction: the marker is somewhere along a line, a ray, from the lens through that dot. Hold one eye shut and try to touch two pencil tips together: it is surprisingly hard.

A second camera, somewhere else, gives a second ray. Where the two rays cross is the marker. This is triangulation, the same idea as your two eyes judging distance. With more cameras the rays pin the point down better, and it helps when the cameras look from very different angles. Before a shoot, the team waves a calibration wand through the volume so the computer knows exactly where each camera is.

Because each dot's centre is found to a fraction of a pixel, a good optical system places a still marker to well under a millimetre. A lab test of one commercial system measured an average error of 0.15 mm. The big enemy is occlusion: when an arm or another actor blocks a marker, fewer cameras see it. Under two, it cannot be placed at all, and a gap appears in the data.

Triangulation gives a cloud of 3D dots, 120 times a second, but the computer does not yet know which dot is which. Step one is labelling: each dot gets a name like LKNE (left knee) or RWRA (right wrist). The software uses a template of the suit and follows each dot from frame to frame. When two dots pass close together, it can mix them up, a marker swap, which a person then fixes.

Step two is solving. The software fits a skeleton (a set of bones and joints sized to the actor) inside the labelled dots, frame by frame. The result is not dots any more but joint rotations: how much the hip, knee and elbow bend.

Step three is retargeting: copying those rotations onto a character with a different body. The character needs its own skeleton and rig (see Anim3DClear for rigging). If its legs are longer, each copied step is longer too. If the body still travels as far as the actor did, the planted foot must slip along the floor: foot slide. The fix is to scale the travel to the new leg length, and to lock the feet to the floor with IK (inverse kinematics).

Bodies are only half a performance. For the face, the actor wears a head-mounted camera (HMC): a small camera on a boom fixed to a helmet, pointing back at the face. Because it moves with the head, the face stays in the same spot in its picture, so only expressions make the dots move. Dots are painted or stuck on the skin: dozens, up to about 150.

The facial solver works backwards. It knows how the dots move for each basic expression, called a blend shape: jaw open, smile, brows up, blink. It finds the mix of blend shapes that best explains what the camera saw. Those numbers, 0 to 1 for each shape, then drive the creature’s own blend shapes. Artists can remap them: a creature with big ears could lift its ears when the actor lifts their brows.

Fingers are small and hide each other, so they are hard for body cameras. Performers often wear gloves with bend sensors along each finger, or tiny markers, or animators add fingers by hand.

Optical markers are the most accurate, but you need a stage full of cameras. There are other ways.

An inertial suit has about 17 small motion sensors (IMUs), one on each body segment. Each has a gyroscope, an accelerometer and a compass (magnetometer), like the ones in your phone. They measure how each bone turns, very well, and they work anywhere, even outdoors. But to know where you are, the suit must add up acceleration twice, and tiny errors grow fast: this is drift. Suits fight it by knowing that a planted foot is not moving. Steel in a floor can also upset the compass.

Markerless capture uses ordinary video and machine learning (pose estimation) to find keypoints like elbows and knees, with no suit at all. OpenPose (2017) did this in real time; Google’s BlazePose (2020) finds 33 body points on a phone. It is cheap and quick, but less precise, and it guesses what it cannot see.

Your phone can capture faces: a depth camera projects over 30,000 invisible dots, and the software gives 52 expression numbers, enough to drive a cartoon face live.

On a film, body, face and voice are often recorded together: performance capture. The director does not want to imagine the creature, so the stage streams the solved skeleton into a game engine in real time. A camera operator carries a virtual camera: a screen with markers on it. The system tracks it like a marker, and the screen shows the creature in its virtual world, framed from where the operator stands. The next step, filming real sets with virtual backgrounds on LED walls, is VirtualProdClear.

The live preview is rough. Afterwards the data is cleaned. Gaps where a marker was hidden are filled with a smooth curve. Jitter, the small shake of every measurement, is removed with a smoothing filter. Filter too hard and the motion goes soft: a quick hand flick or the top of a jump gets cut down.

Then animators polish: fixing feet, adding fingers, pushing a pose to be more readable, or blending in keyframes (see Anim3DClear). Games record short moves (walk, turn, jump) that loop and blend live at 30 to 60 frames a second, so they must join seamlessly. Films capture whole scenes, and each shot gets hours of care. Creatures are then finished with muscles, skin and lighting (see VFXClear).