How does virtual production work?

In virtual production the actors stand on a small real set, and the rest of the world is shown around them on a giant curved wall of LED screens, plus a screen ceiling. This room of screens is called an LED volume. The picture on it is a real 3D place, built by artists and drawn live by a game engine such as Unreal Engine (see CGIClear for how real-time rendering works).

The trick is at the seam. Real sand, rocks and props on the floor are painted to match the ground on the wall, so the camera can't tell where the set ends and the screen begins. The camera films it all at once: what you see through the lens is the final shot, which is why this is also called in-camera VFX.

The older way is a green screen (see VFXClear): film the actor in front of flat green, then replace the green with the world months later. It is cheaper and flexible, but the actor sees nothing, the light is wrong, and green light spills onto skin and shiny things. An LED wall gives real light, real reflections and a world the actors can react to, but everything must be built and decided before the shoot.

A picture on a flat wall has one problem: move the camera and a real mountain far away should hardly move, while a rock close by should slide past. A printed backdrop can't do that. An LED wall can, because it is redrawn for the camera 24 or more times a second.

The camera is tracked. Infrared cameras on the truss watch a crown of reflective balls on it (as in MocapClear), so the engine always knows exactly where the lens is and where it points. The engine then draws the world from that very spot, and paints the result onto the patch of wall the lens can see. This patch is the inner frustum (a frustum is the pyramid of space a camera sees). Every LED in it shows whatever lies behind it along the line to the lens, so through the camera the wall vanishes and a deep world with true parallax appears.

The rest of the wall, the outer frustum, shows the world from one fixed point at lower quality. The camera never looks at it directly: its job is to light the set. The inner patch is drawn a bit bigger than the lens needs (overscan), so quick moves don't show its edge.

Each LED is a tiny coloured lamp, and a wall has millions of them. So the wall doesn't just show the world: it lights the set with it. A sunset on the screens throws warm orange light on the actor's face; a city at night throws blue and pink neon. Nobody has to fake it with filters (see LightingClear for how film lights are normally set).

Shiny things show it best. A chrome helmet, a car door or a pair of glasses reflects the wall, so the reflections are correct in camera. With a green screen, that same helmet would reflect a big green wall, and artists would have to paint every reflection out later.

Change the location and the light changes instantly: a sunset can last all day. There are limits. Behind the camera there is no wall, so the face gets little light from the front, and a wall is far dimmer than the real sun. Crews still add film lights for fill and for hard sunlight.

Up close, an LED wall is a grid of tiny red, green and blue dots on a black face. The gap from one dot to the next is the pixel pitch, around 2.3 to 2.9 mm on film stages. A camera sensor is a grid too. When the two grids line up badly, you get moiré: wavy rainbow stripes that were never there.

The cure is focus. The actors are sharp and the wall is a few metres behind them, slightly out of focus. The blur washes out the dots before they reach the sensor, so the wall looks like a smooth picture. Focus on the wall itself, or stand too far back with a wide lens, and the dots come back.

The wall is also a little late. Tracking, drawing and sending pixels to the screens takes several frames (about 8 on early stages). When the camera whips round, the inner frustum trails behind, and overscan must be big enough to hide the lag.

Last, a wall has limits: about 1,500 to 1,800 nits at most, which is nothing next to the sun, and LEDs make light in three narrow colours, which can make skin and fabrics look slightly off. So hard sunlight and some colours still come from film lights.

Just off the stage sits the brain bar: a row of desks and screens where engine operators run the virtual world. They are the stage’s crew for everything that isn't physical.

If the director says “move that mountain left” or “give me the sunset again”, an engine operator drags the mountain or turns the sun, and the wall changes in seconds. The world itself was built weeks earlier by the virtual art department (VAD): artists who model, texture and light the locations, often from photos and scans of real places, and check them in VR before the shoot.

Other seats run the tracking system (where the camera is) and the LED processors that feed the screens. Everything must tick together. Genlock is a shared timing pulse, so the camera shutter, the engine and every LED panel start each frame at the same instant. Without it, the wall can change picture while the shutter is open, and a tear or dark band rolls through the shot.

Putting actors in a car and moving the world behind them is an old trick. From the 1930s, studios used rear projection: a projector behind a translucent screen played a filmed road behind a car on the stage. It was dim and flat, and the car's windows and paint reflected only a dark studio.

LED volumes do this far better. The street scrolls past on the wall, and the car's glass, chrome and paint reflect passing street lights and signs, with light sweeping across the actors' faces. Crews call these process shots or car process, and they are now one of the most common uses of LED stages.

LED volumes are used for series (The Mandalorian, shot on ILM's StageCraft stage from 2019, made the idea famous), films, ads and music videos. Since 2020 hundreds of stages have been built worldwide. India has them too: Annapurna Studios in Hyderabad opened an LED stage with Qube Cinema in 2022 to 2023, and more have followed in Mumbai and elsewhere.

Is it cheaper than a location? Sometimes. You skip travel, weather and waiting for the light, but you pay for the stage and weeks of building the world first. It works best for places that are hard, dangerous or impossible to reach, and for scenes you need to repeat.