How does image recognition work?

You look at this mango and just see a mango. A computer never does. A camera's sensor (CameraClear shows how) measures light in a grid of tiny squares called pixels, and each pixel becomes three numbers: how much red, green and blue light arrived, each from 0 to 255.

So this 32 × 24 picture is 2,304 numbers in a row. A phone photo of 12 megapixels is about 36 million. Somewhere in those numbers is a mango, but no single number says "mango".

The picture on the right is made live by a tiny renderer in this page: for every pixel it shoots rays from the camera into the 3D scene on the left, finds what they hit, and works out the light. Click any pixel to read its numbers.

Now the hard part. Turn the mango, move the camera or dim the lamp, and most of the numbers change. A different angle, size or light gives a completely different grid, for the same mango. Recognising things means finding what stays the same while all the numbers move. Your eyes and brain do this without effort (EyeClear and BrainClear show how). Computers had to learn it.

If a picture is just numbers, how do you find a mango in them? Start small. Look at a 3 × 3 patch, multiply each pixel by a weight, and add up. This little grid of weights is a filter (or kernel), and sliding it across the whole picture is called convolution. NeuralNetClear shows it on a digit; here it runs on a colour scene.

The right weights make the sum big only at an edge, where brightness jumps. Combine an up-down and a side-to-side edge filter and you get outlines. Where edges run two ways at once, you have a corner, like the tip of a chilli. Other filters answer to texture: busy, fine patterns. Each answer, spot by spot, makes a new picture called a feature map.

Pooling then shrinks each map: keep only the biggest number in every 2 × 2 block. Four numbers become one, so the next layer looks at a wider area, and a small shift of the object hardly changes the answer.

A convolutional neural network (CNN) stacks these steps, but nobody sets its weights by hand: it learns them from examples. Open "Inside the trained CNN" to see the network this box trains in your browser. Its first layer learns edge and colour filters on its own. Its second layer combines them into bigger parts. Big networks such as ResNet repeat this 50 or more times, climbing from edges to textures to parts to whole objects. The idea was loosely inspired by cells in the cat's visual cortex (see BrainClear), but a CNN is maths, not a brain.

Now let's teach a network to recognise things. When you opened this box, the page drew 1,200 made-up pictures, 240 each of a mango, a banana, a chilli, a coin and nothing, each 20 × 20 pixels with its own colour, size, angle and background. Each one comes with its right answer, a label.

Training works as in NeuralNetClear: show a batch of 16 pictures, measure how wrong the answers are, and nudge all 21,081 weights a little to be less wrong (backpropagation). One pass through all 1,200 is an epoch. This network gets 8.

The real test is pictures it has never seen. Every half pass, it is tested on 300 "wild" pictures: turned any way, in dim or bright light, with colour casts and more noise. The confusion matrix shows what it mixes up: each row is the true answer, each column its guess.

Data augmentation is a cheap trick with a big effect. Each time a training picture is used, it is randomly turned, flipped, zoomed, shifted, re-lit and made noisy. The network never sees exactly the same picture twice, so it can't just memorise. Turn augmentation off and train again: training accuracy shoots towards 100%, but on wild pictures it does clearly worse. It learned the pictures, not the fruit.

A recogniser says what is in a picture. A detector must also say where, for every object, even when there are several. The simplest way: slide the recogniser across the scene like a magnifying glass. This one checks 897 windows of 20 × 20 pixels, each 2 pixels from the last, and each window guesses a class, a confidence and a tight box.

Keep only windows above a confidence threshold. Here each box floats above the picture at a height equal to its confidence, and the glass sheet is the threshold. Lower it and you find more objects, but also more junk.

One mango sets off many overlapping windows. Non-max suppression (NMS) cleans up: take the most confident box, delete every box that overlaps it too much, and repeat. Overlap is measured by IoU, intersection over union: the shared area divided by the total area covered. 1 means identical, 0 means no overlap. A detection usually counts as right if its IoU with the true box is at least 0.5.

Boxes are rough. Segmentation labels every single pixel instead: a second small network here scores each pixel "object or not", and each detection paints its own object's pixels. Modern detectors such as YOLO look at the whole picture once instead of sliding, which is how they run at video speed in cars and phones. This little detector trained in your browser in seconds, so it misses some objects and raises some false alarms: the readout counts both honestly.

A network that scores 90% in a test can still fail in strange ways, because it never "understands" a mango. It found number patterns that happened to work. Here are four ways that goes wrong, three of them run live on this box's own networks.

Adversarial noise. Using the network's own maths, you can work out a tiny change for every pixel that pushes the answer the wrong way. Each number moves by only a few steps out of 255, too little for you to notice, yet the label flips. Random noise of the same size usually does nothing. Researchers have shown the same with stickers on real stop signs.

Shortcuts. A second network here was trained where every banana sat on green and every mango on blue. It scored almost 100%. Swap the backgrounds and it follows the colour behind the fruit, because that was the easiest pattern. The saliency map shows which pixels its answer depends on. Real models have done this too: one reading chest X-rays was found to pick up clues about which hospital took the picture, which was linked to how sick patients there tended to be.

Light and data bias. A network only knows the world its training pictures showed. Dim the light far below anything it trained on and accuracy falls. The same thing happens with people. The 2018 Gender Shades study tested three commercial face-analysis systems: they got the gender of lighter-skinned men wrong at most 0.8% of the time, but of darker-skinned women up to 34.7%. Much of the gap came from unbalanced training data. The companies improved their systems after the study, and a large 2019 test by the US standards agency NIST found error gaps between groups in many (not all) algorithms.

Privacy and surveillance. Face recognition can find missing children, unlock phones and speed up airport queues, like India's DigiYatra. It can also track people without their consent, and a false match can wrongly accuse someone. Countries are drawing different lines: the EU's AI Act bans most live face recognition by police in public places, while others are expanding its use. Where to draw the line is a debate for everyone, not only engineers.

The same ideas, pixels in, filters, layers, an answer out, run in many places around you. Pick one.

Phone photo search. Type "mango" in a gallery app and a network like this box's finds the pictures, without anyone having labelled them. The search here is live: this box's own network looks through 16 new pictures.

Eye screening. People with diabetes can slowly lose sight from diabetic retinopathy. In a 2019 study of 3,049 patients at Aravind Eye Hospital and Sankara Nethralaya in India, a network spotted cases that needed a doctor about as well as, or better than, human graders. It is a screening aid: doctors still decide.

Crops. Phone apps can suggest a leaf disease from a photo, and millions of farmers in India have reportedly tried them. But be careful: one well-known network scored 99% on tidy lab photos and only about 31% on photos taken elsewhere. Chapter 5's lesson again.

Roads and factories. Traffic cameras count vehicles and read number plates for fines. Factory cameras check every tablet, circuit board or coin for flaws, far faster than people.

UPI QR codes. Surprise: scanning a QR code mostly uses no learning. The scanner hunts for the three corner squares, whose stripes always come in the ratio 1:1:3:1:1, then straightens the grid and fixes errors with clever maths. Classic vision is still everywhere.

What's next. Vision transformers cut a picture into patches and treat them like words (see LLMClear). Vision-language models learn from hundreds of millions of pictures with captions, so you can ask a question about a photo and get an answer in words. They can also describe things wrongly and confidently, so check what matters. Video models (VideoModelClear) and running models fast on phones (InferenceClear) are the next steps.