Layers and normalisation
Lesson 4 of 8
4Layers and normalisation
The big pictureA neural network is a stack of four recurring pieces: a linear layer that mixes inputs, an activation that breaks linearity, dropout that prevents memorising, and normalisation that keeps numbers in a sane range. Understand each piece's job and you will read any architecture like a sentence.
What the picture shows: The input passes four pieces in order: a linear mix, ReLU breaking linearity, dropout against memorising, then a linear mix to 10 classes. The marked piece is ReLU, because without it the two linear layers equal a single one.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A neural network is floors stacked on each other. Each floor takes numbers from the one below, mixes them in a way it learned, then decides what to pass up. Some floors only keep order: they keep numbers at a sensible size, or switch off random cells so the model never leans on a single one.
nn.Linear is a linear map, and an activation like ReLU adds the non-linearity without which layers collapse into one. Dropout turns off a share of units in training to cut overfitting, and BatchNorm and LayerNorm steady the activation distribution, making training faster and more stable.
Before You Start
📈Linear regression
You know y = Wx + b, and a linear layer is the same thing with many outputs.
⚙️A training loop by hand
You know the five lines, and here you learn what sits inside model on the first one.
The Core Idea
Every network is built from four recurring pieces. In this lesson you count a linear layer's parameters, then see why we need an activation, then try dropout in both modes, then compare the two normalisations.
1The linear layer: a weighted mix
A linear layer multiplies the input by a weight matrix and adds a bias, so each output is a weighted mix of all inputs. Look at the picture: every unit on the right connects to every unit on the left, and every line is a weight.
So it has in × out + out parameters. Anything called a "dense" or "fully connected" layer is this layer.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
from torch import nn
model = nn.Sequential(
nn.Linear(784, 128),
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(128, 10),
)
for name, p in model.named_parameters():
print(f"{name:10} {tuple(p.shape)}")
print("total:", sum(p.numel() for p in model.parameters()))
Linear(784, 128) holds a 128 × 784 matrix, 100,352 weights, plus 128 biases. The second layer is 10 × 128 + 10. The total is 101,770. ReLU and Dropout have no parameters. Count parameters by hand once for each new architecture; it tells you the memory and the risk of memorising.
2The activation breaks linearity
Ten linear layers with nothing between them equal one linear layer. The activation is what lets a network learn curves. ReLU zeroes negatives and passes positives, and GELU, a smooth version, is used in most transformers.
import numpy as np
z = np.array([-3.0, -0.5, 0.0, 0.5, 3.0])
relu = np.maximum(0, z)
sigmoid = 1 / (1 + np.exp(-z))
tanh = np.tanh(z)
print("z ", z)
print("relu ", relu)
print("sigmoid", sigmoid.round(3))
print("tanh ", tanh.round(3))
print("sigmoid slope at 3:", round(float(sigmoid[-1] * (1 - sigmoid[-1])), 3))
The sigmoid at 3 gives 0.953, and its slope there is only 0.045. In a deep network the gradient gets multiplied by such a small number at every layer and vanishes before reaching the first layers. That is "vanishing gradients", and why ReLU, with slope 1 for every positive value, replaced the sigmoid in hidden layers.
3Dropout and the two modes
Dropout randomly zeroes a fraction of units during training only, so the network cannot lean on any single unit. Usual rates are 0.1 to 0.5. Look at the picture: eight ones after Dropout(0.5) in training.
import torch
from torch import nn
torch.manual_seed(0)
drop = nn.Dropout(p=0.5)
x = torch.ones(8)
drop.train()
print("train:", drop(x))
drop.eval()
print("eval: ", drop(x))
In training, half the values became zero and the rest became 2, not 1. PyTorch scales the survivors by 1 / (1 − p) so the mean stays the same. In evaluation everything passes unchanged. Forget model.eval() and your predictions are partly random every time.
Some layers behave differently in training and evaluation. That is why you call model.train() before training and model.eval() before evaluating.
4Batch norm versus layer norm
Normalisation keeps the range of outputs stable between layers, allowing larger learning rates and deeper networks. The two kinds differ in direction. Look at the picture: batch norm takes one column at a time, one feature across all the examples.
Three examples, each with three features on very different scales. Compute both normalisations in NumPy to see the difference in direction.
import numpy as np
batch = np.array([[1.0, 200.0, 3000.0],
[2.0, 180.0, 1000.0],
[3.0, 220.0, 2000.0]])
col_mean, col_std = batch.mean(axis=0), batch.std(axis=0)
batch_norm = (batch - col_mean) / col_std
row_mean, row_std = batch.mean(axis=1, keepdims=True), batch.std(axis=1, keepdims=True)
layer_norm = (batch - row_mean) / row_std
print("batch norm (per column):\n", batch_norm.round(2))
print("layer norm (per row):\n", layer_norm.round(2))
Batch norm gave each column mean zero, so features became comparable. Layer norm gave each row mean zero, but the third column stayed the largest in every row since its values are in the thousands. In transformers the features are embedding dimensions of similar meaning, so layer norm fits, and it does not depend on batch size, working even with a single example.
Batch norm is common in vision, layer norm in transformers. Batch norm also behaves differently in eval mode: it uses stored statistics instead of the batch's own.
How It Works, Step by Step: how an input passes through the layer stack
Linear(784, 128) multiplies the input by a 128 × 784 matrix and adds a bias, so each output is a weighted mix of all inputs.
ReLU zeroes negative values and passes positive ones, letting the network learn curves.
Dropout(0.2) randomly zeroes a fifth of the units during training only, and scales the rest up so the mean stays the same.
Linear(128, 10) outputs one number per class, and for evaluation you call model.eval() so dropout switches off.
Worked example
One linear layer is a linear regression: y = Wx + b. Stack ten linear layers with nothing between them and the result is still linear, since composing linear functions gives a linear function. The activation is what lets a network learn curves.
model.eval() when the model has Dropout or BatchNorm. Numbers change between two runs on the same data and look worse than reality, and you may spend a day hunting a data bug.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Every extra layer adds capacity. | Three linear layers with no activation between them equal one linear layer. The extra parameters cost without adding capacity. |
| Dropout and normalisation improve any network, so add them from the start. | Each piece fixes a specific problem. Add dropout when you see a gap between training and validation, and normalisation when deep training turns unstable. |
| Sigmoid is fine in hidden layers because it is smooth. | Its slope at 3 is only 0.045, and multiplying by such a small number at every layer makes the gradient vanish. That is why ReLU replaced it. |
Try it yourself
How many parameters does nn.Linear(256, 64) have?
The weight matrix is 64 × 256 = 16384 and the bias has 64 values, totalling 16448. The first option forgets the bias.
Evaluation results change slightly on every run with the same validation data and weights. Most likely?
Dropout is random in training mode, so each run zeroes different units. model.eval() turns it off and makes results stable. Reading order does not change the overall accuracy across all the data.
Quick review cards
in × out + out parameters.eval mode.Check Your Understanding
A network of three consecutive Linear layers with no activation between them. How many "layers" does it effectively learn?
One linear layer. W3(W2(W1x + b1) + b2) + b3 can be written as Wx + b with a single matrix W = W3·W2·W1. The extra parameters add no expressive power, only cost and time. The activation between layers is what makes depth useful.
In training with Dropout(p=0.5) the surviving values became 2 instead of 1. Why?
Because PyTorch scales the survivors by 1 / (1 − p) so the mean stays the same, and the output range does not shift between training and evaluation.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs4
Done with this lesson?
Sign in to save your progress.