Debugging training
Lesson 6 of 8
6Debugging training
The big pictureThree faults cover most of what will happen to you: a loss that will not fall, a loss that turns into NaN, and a model that memorises training and fails elsewhere. Each fault has a signature on the loss curve, and each signature has a first thing to try.
What the picture shows: Up to epoch 100 both curves fall together, then training keeps falling while validation climbs from 0.112 to 0.135. The dashed line is the model you keep, and the gap after it is memorised noise, not learning.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A model that will not learn is like a car that will not move: it might be the fuel (data), the engine (the model), or a handbrake you forgot (a code bug). You check in order from simplest: can it memorise just ten examples? If it cannot, the fault is in the code, not the data.
Three faults: a loss that will not drop (wrong learning rate, a forgotten zero_grad, shuffled labels), a loss going NaN (too large a rate, division by zero, log(0)), and overfitting (a gap between training and validation loss). The first test is always to overfit one batch on purpose.
Before You Start
⚙️A training loop by hand
You know the five lines, and debugging means knowing which of them to inspect.
✂️Splitting the data
You know why we measure on separate validation data, since the training curve alone is not enough.
The Core Idea
Three faults cover most of what will happen to you, and each has a signature. In this lesson you check the first loss, then overfit one batch, then find where NaN comes from, then read both curves together, then change one thing at a time.
1Check the first loss
A classifier with random weights spreads its probability evenly, so its first loss is near ln(number of classes). Look at the picture: these are the numbers to expect before any training.
If the first loss of a ten-class classifier is 15 or 0.01, something is wrong before training even starts. Do not wait a thousand steps to find out.
2Overfit one small batch
The most important test: can the model memorise 16 examples? If it cannot, the bug is in the code. If it can, the problem is the data, the settings or the model size.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(16, 10)
y = torch.randint(0, 3, (16,))
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 3))
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for step in range(201):
loss = nn.functional.cross_entropy(model(X), y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if step % 50 == 0:
print(f"step {step:3} loss {loss.item():.4f}")
Sixteen examples with completely random labels, and the loss still falls from 1.07 to 0.0003. The network memorised them, which is what we want here: it proves forward, loss, backward and step are all wired correctly. If the loss stayed at ln(3) ≈ 1.10, look for a missing zero_grad or an output detached from the graph.
3Where NaN comes from
NaN usually comes from three places: a learning rate large enough to explode the weights, an unstable operation like log(0) or exp of a big number, or NaN already in the data. Look at the picture: how two ordinary numbers become NaN.
Softmax computes exp of each value. With large values exp overflows the largest number the computer can represent, becoming infinity, and infinity over infinity is NaN.
import numpy as np
logits = np.array([1000.0, 1001.0])
with np.errstate(over="ignore", invalid="ignore"):
naive = np.exp(logits) / np.exp(logits).sum()
shifted = logits - logits.max()
stable = np.exp(shifted) / np.exp(shifted).sum()
print("exp(709) =", np.exp(np.float64(709)), "(float64 tops out near here)")
print("naive softmax: ", naive)
print("stable softmax:", stable.round(3))
The simple fix: subtract the maximum before exp. Mathematically identical, but the largest exponent becomes zero. PyTorch does this inside cross_entropy and log_softmax, which is why you should use them instead of writing softmax yourself followed by log.
4Learning or memorising: read both curves
The training curve alone cannot tell learning from memorising. The gap between training loss and validation loss is the information.
A large network (two layers of 128 units) on only 60 points of a noisy sine curve. Watch both numbers together.
import numpy as np
from sklearn.neural_network import MLPRegressor
rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, size=(60, 1))
y = np.sin(X[:, 0]) + rng.normal(scale=0.3, size=60)
X_val = rng.uniform(-3, 3, size=(200, 1))
y_val = np.sin(X_val[:, 0]) + rng.normal(scale=0.3, size=200)
model = MLPRegressor(hidden_layer_sizes=(128, 128), learning_rate_init=0.01, random_state=0)
for epoch in range(1, 601):
model.partial_fit(X, y)
if epoch in (10, 100, 300, 600):
train = ((model.predict(X) - y) ** 2).mean()
val = ((model.predict(X_val) - y_val) ** 2).mean()
print(f"epoch {epoch:4} train={train:.3f} val={val:.3f}")
Until epoch 100 both fall: the model is learning the sine. After that, training loss keeps dropping to 0.042, below the variance of the noise itself (0.09), while validation loss climbs from 0.112 to 0.135. The model is now memorising noise. The best model was at epoch 100, and early stopping means exactly that: keep the model at the lowest validation loss.
When you see this signature, the usual remedies are more data, a smaller model, regularisation such as Dropout, or early stopping.
5Change one thing at a time
Work like a doctor. Look at the picture: a symptom, then a hypothesis, then one change that tests it, then you log the result, then back to the next symptom.
Changing five settings at once might fix it, but you will not know which was the cause, and you will repeat the mistake on the next project. Log every experiment, even in a plain text file.
How It Works, Step by Step: how to diagnose a failing training run
Compare the first loss with what you expect: ln(number of classes) for a classifier with random weights, so a number far from it means a bug before training starts.
Overfit one small batch, say 16 examples. If the loss does not get near zero, the bug is in the code.
Plot training and validation loss together and read the signature: flat, oscillating, NaN, or a widening gap.
Change one thing to match the signature, like dividing the rate by 10 or stopping early, and log the experiment and its result.
Going deeper
| Symptom | Likely cause | First move |
|---|---|---|
| Loss flat from the start | A code bug or a tiny learning rate | Overfit one batch, then raise the rate ×10 |
| Loss jumps and oscillates | Learning rate too high | Divide the rate by 10 |
| NaN after some steps | Exploding weights or an unstable op | Lower the rate, clip gradients, check the data |
| Train falls, validation rises | Memorising | Early stopping, Dropout, more data |
| Both high and close | Model too small for the task | A bigger model or better features |
Worked example
The most important test in debugging training: can the model memorise one small batch? If it cannot drive the loss near zero on 16 examples, the problem is in the code, not the data or the size.
optimizer.step(): torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0). It stops a single freak step from wrecking weeks of training, and is standard in most transformer training.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| If the loss will not fall, the model needs to be bigger. | A loss flat from the first step is not fixed by a bigger model. The one-batch test settles it in a minute. |
| NaN always means the learning rate is too high. | It can come from an unstable operation like log(0) or exp of a big number, or from NaN already in the data. |
| A very low training loss means an excellent model. | In the sine example training loss fell to 0.042, below the noise variance of 0.09, while validation rose. That is memorising. |
Try it yourself
Training loss is 0.05 and validation loss is 0.90 and rising. Which move does not help?
This is clear memorising: the model has more capacity than the data supports. Making it bigger adds memorising capacity and widens the gap. The other three all reduce memorising or make it harder.
The loss became NaN at step 3000 after healthy training. First move?
Training was healthy for three thousand steps, so the rate is not too high in general. Most likely one batch held an outlier or a missing value that produced a huge gradient. Save the batch when NaN appears and inspect it; gradient clipping protects you from a repeat.
Quick review cards
log(0) or exp of a big number, or NaN already in the data.Check Your Understanding
The first loss on a five-class classifier was 0.02 before any training. What does that tell you?
That something is wrong: a model with random weights knows nothing, so it should give roughly 1/5 to each class, a loss near ln(5) ≈ 1.61. A loss of 0.02 means it "knows" the answer already; most likely the label leaked into the inputs, or every label in the batch is the same and the model happens to output a large value for it.
Why subtract the largest value from the inputs before exp in softmax?
Because exp of values like 1000 overflows to infinity and then NaN. The subtraction does not change the result mathematically, but it makes the largest exponent zero.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.