How do AI video models work?

A video is not one moving thing. It is a quick run of still pictures, called frames. Show them fast enough and your brain sees smooth motion. Cinema settled on 24 frames per second (fps) when sound arrived in the late 1920s, and it still uses it (see FilmClear). TV and phones often use 25, 30 or 60.

Stack the frames one behind another and you get a block: two directions of space and one of time. This is the space-time cube, and it is exactly what an AI video model has to make. Cut the cube across and you get one frame. Cut it along time, one pixel row through every frame, and the ball's motion becomes a curved path.

Each frame is a grid of pixels, and each pixel is 3 numbers: red, green and blue. So one second of Full HD video at 24 fps is 1920 × 1080 × 3 × 24 ≈ 149 million numbers. Reading this page, you take in about 5 tokens of text a second. That gap, millions of times, is why making video is so much harder than making text (see LLMClear for tokens).

Real files are compressed, so a 1080p stream is closer to 5 megabits a second. But a model has to decide every pixel of every frame, and keep them all agreeing with each other.

Most AI video and image models today are diffusion models. The trick is surprising: they learn to remove noise. In training, you take a real picture, bury it under random static, and ask a neural network to guess the clean picture. Do that millions of times at every amount of noise, and the network gets very good at it (see NeuralNetClear for how networks learn).

To make something new, start from pure noise. Ask for a guess of the clean picture. The first guess is a blurry average, because it could be anything. Mix that guess back with a bit less noise, and ask again. Step by step, the picture sharpens out of the static. Real models take roughly 20 to 50 steps.

This one is real and runs right here. Its "memory" is 288 small pictures of shapes. Its guess is the mathematically best one: a weighted average of those 288 pictures, weighted by how well each fits the noise. The board shows which pictures it leans on at each step.

What is simplified: our denoiser is perfect for its 288 pictures, so it can only ever end on a copy of one of them. A real network is trained to approximate this guess, for billions of pictures. Its small, smooth errors are what let it mix ideas and make pictures nobody has seen. The same flaw in reverse is memorisation: real models sometimes reproduce training images too closely.

Denoising 149 million numbers a second directly would be far too slow. So video models first squeeze the video. An autoencoder has two halves: an encoder that turns each patch of pixels into a few numbers, and a decoder that turns them back. The small version is called the latent. All the diffusion happens in latent space, and the decoder paints the final pixels at the end.

Ours is real and was trained a moment ago on every frame in the toy clips. It turns each 4×4 block of pixels (48 numbers) into k numbers. Slide k down and watch the picture blur: the squeeze keeps the big shapes and throws away detail. Open video models reportedly shrink time by 4 and each side by 8, keeping 16 numbers per spot: about 48 times smaller.

Next, the latent cube is cut into space-time patches, a few cells wide and a few frames long. Each patch becomes one token, just like a word piece in a language model. OpenAI's Sora report (2024) described this idea as "spacetime patches".

A transformer then lets every token look at every other one with attention (see LLMClear). Click a cell: its token looks for tokens with similar content. The red ball's token finds the ball in other frames. That is how a model keeps things consistent over time. The catch: attention compares every pair, so 100,000 tokens means 10 billion pairs.

So far the model makes something. To make what you ask for, the prompt has to take part in every step. First a text encoder, a language model like the ones in LLMClear, turns your words into a list of numbers. The denoiser reads those numbers alongside the noisy picture, usually through attention. Training pairs millions of videos with their captions, so the network learns which pictures go with which words.

That link is never perfect. So models use a trick called classifier-free guidance. At each step the network makes two guesses: one with the prompt and one without. The difference between them is "what the prompt adds". Guidance multiplies that difference by a number w and adds it back: guess = without + w × (with − without).

Our toy's text encoder is just a lookup for two words, and its grasp of them is deliberately weak, like a partly trained model. At w = 0 it ignores you. At w = 1 it follows the prompt only some of the time. Push w higher and it obeys more often. Real models usually run at about 5 to 7. Too high and pictures turn harsh and samey, with less variety.

The seed picks the starting noise. Same seed and same prompt give the same result every time. Same seed, different prompt: a different result, often with a similar layout, because the noise already leans towards certain places.

A video model has a harder job than an image model. Every frame must look right, and every frame must also agree with the ones around it: the same ball, the same colour, moving smoothly. When that fails you see flicker, morphing (a thing slowly turns into something else) and things that pop in or out of existence.

Here are three real ways to make an 8-frame clip of the same prompt. Frame by frame: each frame is made alone, from its own noise. Each frame is fine, but the ball teleports and can even change colour. Same noise every frame: steady, but frozen, because the same noise always gives the same picture. Whole clip together: a video model denoises all 8 frames as one space-time block, so each frame's guess depends on the others. The motion comes out smooth.

That third way is what modern video models do, with attention across the whole space-time cube (chapter 3). Early text-to-video systems (2022) started from image models and added layers that look across time.

Why it is still hard. The model has no physics engine inside. It learned what videos usually look like, not the rules behind them. So it can get hands wrong, let objects pass through each other, or forget a detail after something walks in front of it. OpenAI itself listed failures like a cookie showing no bite mark after being bitten. Longer videos make it worse: more tokens, more to keep straight.

Cost. Video is the heaviest thing these models make. In 2025, MIT Technology Review measured open models: one 5-second video took about 3.4 million joules (roughly 1 kWh), against about 2,300 to 4,400 J for one picture and roughly 100 to 7,000 J for a text reply. A Hugging Face study found the energy grows roughly with the square of the length: twice as long, about four times the energy. Big commercial models do not publish their figures, so treat all of these as estimates. Running models is its own topic (see InferenceClear, being built).

Trust. A realistic fake video of a real person is a deepfake. Making one without the person's consent can hurt them badly, and it is illegal in many places. In India, a viral deepfake of an actor in November 2023 led MeitY to issue an advisory to platforms that December. Amended IT Rules, reported as notified in February 2026, require AI-made audio and video to carry a prominent label, and platforms to remove unlawful content within 3 hours of an official order. The EU and China have labelling rules too.

Two tools help. A watermark hides a faint signal in the pixels that a detector can find, like Google DeepMind's SynthID. Content credentials (the C2PA standard, started in 2021 by Adobe, Microsoft, the BBC and others) attach a signed record of how a file was made. Neither is perfect: watermarks can be weakened, and credentials can be stripped. A video with no label is not proof it is real.

Use. In film, these tools help with storyboards and previs, rough moving sketches before a shoot (see StoryboardClear), concept ideas and some effects work (see VFXClear), with shots still edited and graded by people (see GradeClear). They differ from CGI rendering, which computes light in a 3D scene exactly as built (see CGIClear): a video model predicts pixels it thinks are likely. That makes it fast and flexible, but harder to control. People are still debating training on artists' work without permission, and the effect on jobs.