Did it actually improve

Two ways into the same idea

Before You Start

The Core Idea

A lower validation loss does not mean a better model at your task. In this lesson you build a frozen test set, compare before and after with a confidence interval, catch forgetting, and then decide: adopt or roll back.

1The frozen set

The frozen set is test questions written before training, never seen by it, and unchanged across experiments. Without it two versions cannot be compared.

Look at the picture: the set is written and kept in Git before training, and is not touched until the final evaluation.

2Before and after, with an interval

You run both models on the same questions and compare question by question. Then you resample the same questions for both models thousands of times, and take the 95% interval of the difference.

Look at the picture: accuracy rose from 64.0% to 70.5%. The real question is whether the interval excludes zero.

import numpy as np

rng = np.random.default_rng(0)
n = 200
before = rng.random(n) < 0.71          # 1 = correct on the frozen eval set
after = before.copy()
flip_up = rng.choice(np.where(~before)[0], 22, replace=False)
flip_down = rng.choice(np.where(before)[0], 9, replace=False)
after[flip_up], after[flip_down] = True, False

diffs = []
for _ in range(2000):                  # paired bootstrap: resample the same questions for both
    idx = rng.integers(0, n, n)
    diffs.append(after[idx].mean() - before[idx].mean())
low, high = np.percentile(diffs, [2.5, 97.5])
print(f"before: {before.mean():.1%}   after: {after.mean():.1%}")
print(f"improvement: {after.mean() - before.mean():+.1%}, 95% interval [{low:+.1%}, {high:+.1%}]")
print("questions fixed:", len(flip_up), "  questions broken:", len(flip_down))

The improvement is 6.5 points, with an interval from +1% to +12%, so it is probably real since the interval excludes zero, but its size is uncertain. The last line matters just as much: 22 questions were fixed and 9 broken. Read those nine: if they are all one kind, tuning broke something specific you need to fix.

3Catastrophic forgetting

Tuning on a narrow task can weaken general abilities the model had, and you never see it in the task's loss. A small general set run before and after reveals it.

Look at the picture: the old skill collapsed from 0.99 to 0.58 with the new task alone, and stayed at 0.96 when we mixed in old examples.

A network learns an "old" task, then is trained on a new one. Once on the new task alone, once mixing in some old examples.

import warnings
warnings.filterwarnings("ignore")   # small runs stop before sklearn calls them converged

import numpy as np
from sklearn.neural_network import MLPClassifier

rng = np.random.default_rng(0)
def task(kind, n=600):
    X = rng.normal(size=(n, 6))
    X[:, 5] = 1.0 if kind == "old" else -1.0          # which task the input belongs to
    y = (X[:, 0] + X[:, 1] > 0) if kind == "old" else (X[:, 2] - X[:, 3] > 0)
    return X, y.astype(int)

old_train, new_train = task("old"), task("new")
old_test, new_test = task("old"), task("new")

def run(replay):
    m = MLPClassifier(hidden_layer_sizes=(32,), learning_rate_init=0.01, random_state=0)
    for _ in range(60):
        m.partial_fit(*old_train, classes=[0, 1])
    old_before = m.score(*old_test)
    X, y = new_train
    if replay:                                         # mix in a quarter as many old examples
        X = np.vstack([X, old_train[0][:150]])
        y = np.concatenate([y, old_train[1][:150]])
    for _ in range(60):
        m.partial_fit(X, y)
    return old_before, m.score(*old_test), m.score(*new_test)

for name, replay in [("new task only", False), ("new task + old examples", True)]:
    before, after, new = run(replay)
    print(f"{name:24} old skill {before:.2f} -> {after:.2f}   new task {new:.2f}")

Training on the new task alone collapsed the old one from 0.99 to 0.58, close to guessing, while the new one succeeded. Mixing in just 150 old examples kept the old task at 0.96 and the new at 0.98. The same phenomenon happens in language models, and the new task's loss never shows it. Only the general-abilities set reveals it.

4When to roll back

The original model always stays an option. You roll back when the gain does not clear the confidence interval, breaks important questions, or general abilities drop more than you accept.

Look at the picture: two questions decide. Adopting needs an interval above zero and a general set that held.

And read every broken question yourself. If they are all one kind, tuning broke something specific you need to fix.

How It Works, Step by Step: how to compare the model before and after tuning

Worked example

The frozen set is the test file from the data lesson: training never saw it, and it has not changed since it was written. We run the original and the tuned model on it and compare question by question.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding