Building and evaluating

Two ways into the same idea

Before You Start

The Core Idea

Building goes through four steps: a first version that works end to end, then an experiment table with one change per row, then error analysis by slice, then a decision to stop made by numbers.

1The first version: end to end

Start with the simplest thing that works all the way through within days, even if it is weak. Look at the picture: a real input goes in, a result comes out, and a number is computed on the eval set.

The eval set is frozen and carries tagged slices such as dialect, length and request type. That tells you where the system fails, not only how much.

2The experiment table: one change per row

Each experiment is a row: what changed, why, the number and the time. One change at a time, so you know what had the effect.

Look at the picture: four experiments on an imbalanced problem from the worked example, with F1 on the validation set.

Balancing the weights doubled the number, the forest won because the relationship is non-linear, and both ideas together did best. You choose on validation and open the test once at the end.

3Error analysis by slice

The overall number tells you how often the system fails; error analysis tells you what to do next. Read the errors themselves and sort them into slices before any new experiment.

Look at the picture: where 23 errors in 120 messages come from.

Eval results for 120 messages, each tagged with its slice when the set was built. The overall accuracy hides the story, so we split it.

from collections import defaultdict

# eval results: (text, slice, correct?) - slices were tagged when the eval set was built
results = (
    [("رسالة فصحى قصيرة", "msa_short", True)] * 46 + [("رسالة فصحى قصيرة", "msa_short", False)] * 4
    + [("رسالة عامية قصيرة", "dialect_short", True)] * 33 + [("رسالة عامية قصيرة", "dialect_short", False)] * 7
    + [("رسالة طويلة فيها أكثر من طلب", "multi_intent", True)] * 9 + [("رسالة طويلة فيها أكثر من طلب", "multi_intent", False)] * 11
    + [("English message", "english", True)] * 9 + [("English message", "english", False)] * 1
)

overall = sum(ok for *_, ok in results) / len(results)
print(f"overall accuracy: {overall:.0%} on {len(results)} examples\n")

by_slice = defaultdict(list)
for _, s, ok in results:
    by_slice[s].append(ok)

total_errors = sum(not ok for *_, ok in results)
print(f"{'slice':<14} {'n':>3} {'acc':>5} {'share of errors':>16}")
for s, oks in sorted(by_slice.items(), key=lambda kv: sum(kv[1]) / len(kv[1])):
    errors = len(oks) - sum(oks)
    print(f"{s:<14} {len(oks):>3} {sum(oks) / len(oks):>5.0%} {errors / total_errors:>16.0%}")

81% overall looks acceptable, but messages with more than one request fail more than half the time, and alone they are nearly half the errors though they are a sixth of the set. So the next experiment is clear: not hyperparameter tuning, but splitting the message into requests before classifying, or handing it to a human. Dialect comes second, so it is the target of the experiment after that.

4When to stop

Stopping is a decision made by numbers, not by fatigue. Look at the picture: two questions lead you to it.

If you passed the bar by a clear margin, stop. If three experiments in a row moved the number by less than the error range, more of them will change nothing, so stop too and rethink the problem.

How It Works, Step by Step: how to run experiments until you stop

Going deeper

In language model projects, an experiment might be a change to the instructions, to the number of retrieved chunks, or to the model itself. Same rule: a row in the table for each change, the instruction version saved in git, and the evaluation run automatically over the whole set, not on five examples you remember. And judging with a language model needs to be checked by hand on a sample first, as you saw in the evaluation lesson.

Worked example

The experiment table is your project's memory. In two weeks you will not remember why you chose this setting, or which version gave the best number. Here are four experiments on an imbalanced classification problem, each logged to a file as one line.

1. Four experiments and a table

import json
import time
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score

X, y = make_classification(n_samples=2000, n_features=20, n_informative=6,
                           weights=[0.8, 0.2], random_state=7)
X_tr, X_rest, y_tr, y_rest = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
X_val, X_te, y_val, y_te = train_test_split(X_rest, y_rest, test_size=0.5, stratify=y_rest, random_state=0)

log = []

def run(name, model, note):
    t0 = time.perf_counter()
    model.fit(X_tr, y_tr)
    entry = dict(run=len(log) + 1, name=name, note=note,
                 val_f1=round(f1_score(y_val, model.predict(X_val)), 3),
                 seconds=round(time.perf_counter() - t0, 2))
    log.append(entry)
    with open("runs.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps(entry, ensure_ascii=False) + "\n")
    return model

models = {
    "logreg": run("logreg", LogisticRegression(max_iter=1000), "first version"),
    "logreg_bal": run("logreg_bal", LogisticRegression(max_iter=1000, class_weight="balanced"), "minority class is rare"),
    "forest": run("forest", RandomForestClassifier(n_estimators=200, random_state=0), "non-linear"),
    "forest_bal": run("forest_bal", RandomForestClassifier(n_estimators=200, class_weight="balanced", random_state=0), "both ideas"),
}

print(f"{'run':>3}  {'name':<11} {'val F1':>6}  note")
for e in log:
    print(f"{e['run']:>3}  {e['name']:<11} {e['val_f1']:>6}  {e['note']}")

best = max(log, key=lambda e: e["val_f1"])
test_f1 = f1_score(y_te, models[best["name"]].predict(X_te))
print(f"\nchosen on validation: {best['name']}; test F1, measured once: {test_f1:.3f}")

Each experiment changed one thing and recorded why. Logistic regression started at 0.21 because the rare class is ignored, balancing the weights doubled the number, the random forest won because the relationship is non-linear, and combining both ideas gave the best. Then we chose on validation and measured on test exactly once: 0.738, a little below 0.769 as expected, because the act of choosing makes the validation number optimistic.

Example: Logging experiments with MLflow

When the table grows, a tool like MLflow stores parameters, metrics and the model itself and shows them in a UI. It needs pip install mlflow, so it is here for reading.

import mlflow
from sklearn.metrics import f1_score

mlflow.set_experiment("shop-support-classifier")

with mlflow.start_run(run_name="forest_bal"):
    mlflow.log_params({"model": "random_forest", "n_estimators": 200,
                       "class_weight": "balanced", "data_version": "eval-v1-frozen"})
    mlflow.set_tag("why", "minority class ignored + non-linear signal")
    model.fit(X_tr, y_tr)
    mlflow.log_metric("val_f1", f1_score(y_val, model.predict(X_val)))
    mlflow.sklearn.log_model(model, "model")

mlflow ui opens a table where you compare and sort runs, and the data version is recorded with each run so you know whether a comparison is fair. But a plain JSONL file like the first example is entirely enough for a solo project; the habit matters, not the tool.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding