Vision transformers
Lesson 5 of 7
5Vision transformers
The big pictureA vision transformer cuts an image into small patches, treats each patch like a word in a sentence, then applies the same attention mechanism you will learn in the language track. It assumes nothing about images, so it needs far more data than convolution, but wins when that data exists.
What the picture shows: The image becomes a sentence: each 16 × 16 patch is a word of 768 numbers, and the number under it is its position. Without a position encoding, attention cannot tell which patch is at the top and which at the bottom.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A vision transformer cuts the image into small squares like puzzle pieces, then treats each piece as a word in a sentence and lets the pieces confer. It is very strong when you have a huge number of images, but with few images convolution usually wins.
ViT splits the image into 16×16 patches, embeds each linearly, adds positional encoding and a [CLS] token, then runs standard transformer blocks. It lacks convolution's built-in bias towards locality, so it needs more data or large pretraining.
Before You Start
🔲Convolution
You know convolution's two assumptions, that patterns are local and matter everywhere, and you will compare ViT against them.
🧊Tensors
You can follow reshape and transpose across dimensions, since cutting into patches is all reshaping.
The Core Idea
A vision transformer reads an image like a sentence. Here you cut an image into patches, watch attention between patches, see why it needs more data, then the price of high resolution.
1Patches as words
We cut a 224 by 224 image into 16 by 16 patches, giving 14 × 14 = 196 patches. Each patch is 16 × 16 × 3 = 768 numbers, flattened into a vector.
Look at the picture: the image became a sentence of 196 words, then we add a special class token.
import numpy as np
image = np.arange(3 * 224 * 224, dtype=np.float32).reshape(3, 224, 224)
p = 16
c, h, w = image.shape
patches = image.reshape(c, h // p, p, w // p, p).transpose(1, 3, 0, 2, 4).reshape(-1, c * p * p)
print("patches:", patches.shape)
print("tokens per image:", patches.shape[0], "+ 1 class token =", patches.shape[0] + 1)
print("attention pairs per layer:", (patches.shape[0] + 1) ** 2)
reshape splits each spatial dimension into (patch count, patch size), transpose groups the patch dimensions together, and the last reshape flattens each patch. The result is 196 vectors of length 768, and models then add a special classification token for 197.
Each vector goes through a linear layer, then gets a position encoding. Without it the model cannot tell which patch is at the top and which at the bottom, since attention ignores order.
The class token is an extra learned vector that gathers information from all patches, and the final prediction comes from it.
2Attention between patches
Every patch decides which other patches matter to it, even very distant ones, from the first layer. Convolution needs many layers to link two ends of an image.
Look at the picture: each row is one patch, and each cell its attention weight for another patch. A row's weights sum to 1.
Five dummy patches with random vectors. This is the core attention computation you will unpack in the language track: a similarity score for each pair, then softmax so they become weights summing to one.
import numpy as np
rng = np.random.default_rng(0)
tokens = rng.normal(size=(5, 8))
Wq, Wk = rng.normal(size=(8, 8)), rng.normal(size=(8, 8))
scores = (tokens @ Wq) @ (tokens @ Wk).T / np.sqrt(8)
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
print("patch 0 attends to:", weights[0].round(2))
print("rows sum to 1:", np.allclose(weights.sum(axis=1), 1))
The first patch gave 89% of its attention to the third. In a trained model these weights mean something: a patch on a cat's ear attends to a patch on its eye, even if far away. Here the weights are random because Wq and Wk are untrained.
3Why it needs more data
Convolution knows in advance that neighbouring pixels are related and that a pattern matters wherever it appears. The transformer assumes nothing, so it learns that from data.
Look at the picture, which shows only the general shape. With little data convolution leads, and with huge data or a pretrained model the transformer catches up or passes it.
4The price of high resolution
The patch count grows with the square of the image size, and attention grows with the square of the patch count. So doubling the size multiplies the cost by roughly 16.
Look at the picture: the token count alone jumps fast, and the pairs much faster.
for size, patch in [(224, 16), (224, 32), (384, 16), (512, 16)]:
tokens = (size // patch) ** 2 + 1
print(f"{size}px / patch {patch:2}: {tokens:5} tokens, {tokens ** 2:>11,} attention pairs")
From 224 to 512 pixels: tokens go from 197 to 1025, and attention pairs from 39 thousand to over a million, 27 times more. That is why most vision transformers start at low resolution and use tricks like Swin's local windows when high resolution is needed.
How It Works, Step by Step: how a vision transformer reads an image
Cut the 224 × 224 image into 16 × 16 patches, giving 14 × 14 = 196 patches.
Flatten each patch into 768 numbers, project it to a vector with a linear layer, and add its position encoding.
Add the class token at the start so the sequence is 197 long, and pass it through ordinary transformer blocks.
In each block every patch decides which patches matter to it, then the prediction comes from the class token.
Worked example
A 224 by 224 image cut into 16 by 16 patches gives 14 × 14 = 196 patches. Each patch is 16 by 16 by 3 channels, 768 numbers, flattened into a vector. The image became a "sentence" of 196 "words".
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Transformers do not work for vision, since a CNN beats them. | From scratch on a few thousand images a CNN usually wins. The fair comparison is a pretrained ViT against a pretrained CNN, both fine-tuned on your data. |
| Smaller patches are always better because they catch finer detail. | The patch count grows with the square of the shrink factor and attention with the square of the patch count. Patches of 16 instead of 32 on a 224 image mean 197 tokens instead of 50. |
| Attention knows the order of the patches by itself. | Attention operates on a set: shuffled patches give the same results in another order. Position comes only from the position encoding. |
Try it yourself
A 256 by 256 image with 16 by 16 patches. How many patches?
256 / 16 = 16 each way, so 16 × 16 = 256 patches, each 16 × 16 × 3 = 768 numbers.
You have 3,000 product images and no pretrained model is allowed. Which architecture do you start with?
Convolution's assumptions (locality and weight sharing) make up for little data. A ViT from scratch on 3,000 images will memorise more than it learns. A linear network would need millions of parameters with no sense of space.
Quick review cards
Check Your Understanding
Why does a ViT need a position encoding while a convolutional network does not?
Because attention is an operation on a set: it scores every pair of patches, and shuffling the patches gives the same results in a different order. So without a position encoding, a cut-up and shuffled image equals the original. Convolution knows position implicitly because the filter slides over the pixel grid in order, so a neighbour is always a neighbour.
You raised resolution from 224 to 512 pixels with 16-pixel patches. Why did the cost grow about 27 times?
Because the tokens went from 197 to 1025, and attention compares every pair, so attention pairs went from about 39 thousand to over a million.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.