Monitoring models and drift
Lesson 2 of 3
2Monitoring models and drift
The big pictureThe server runs, the reply is 200 and the model answers, but its answers slowly worsen because customers changed. Traditional monitoring cannot see this. Model monitoring can.
What the picture shows: The monitoring loop: comparing what the model sees today with what it learned from, and retraining when they drift apart.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A city map you drew five years ago: it was accurate, but new streets opened and others closed. The map was not wrong; the world changed. A model is a map of the data it saw, and it needs updating when the city changes.
Data drift (covariate shift): the input distribution changes (user ages, a new language, a new device). Concept drift: the relationship itself changes (what was fraud no longer is). Measuring without labels: PSI, KS or Jensen-Shannon distance between training and production distributions per feature, plus the distribution of outputs and confidence. Measuring with labels: late labels (did the customer actually pay?) or a sample reviewed by people. Policy: alert thresholds (PSI > 0.2 means major drift), automatic retraining through a recorded pipeline, evaluation on recent data, then a shadow deploy (the new model predicts without being used) or a canary (5% of traffic) before everyone. For language models: monitor hallucination, refusals, reply length and cost with evaluation samples.
Before You Start
🧪Training pipelines
The previous lesson.
📏Applied statistics
Distributions and comparison.
The Core Idea
1PSI: measuring drift
The Population Stability Index compares each bucket's share between training and production. Below 0.1 is stable, 0.1 to 0.2 means watch, above 0.2 is major drift:
import math
def psi(expected, actual):
total = 0.0
for e, a in zip(expected, actual):
e, a = max(e, 1e-4), max(a, 1e-4)
total += (a - e) * math.log(a / e)
return round(total, 3)
# share of users per age bucket: 18-24, 25-34, 35-44, 45+
training = [0.30, 0.40, 0.20, 0.10]
week_1 = [0.29, 0.41, 0.20, 0.10]
week_9 = [0.12, 0.30, 0.28, 0.30]
for name, prod in [("week 1", week_1), ("week 9", week_9)]:
score = psi(training, prod)
status = "stable" if score < 0.1 else "watch" if score < 0.2 else "DRIFT"
print(name, score, status)2Monitoring without labels
Most of the time you do not know the right answer straight away. Watch what you can: the input distribution, the share of each prediction, and the mean confidence. If the model always said "fraud" for 2% and now says 9%, something changed:
from statistics import mean
baseline = {"fraud_rate": 0.02, "confidence": 0.91}
today = [{"pred": "fraud", "conf": 0.55}] * 9 + [{"pred": "ok", "conf": 0.80}] * 91
fraud_rate = sum(p["pred"] == "fraud" for p in today) / len(today)
confidence = mean(p["conf"] for p in today)
if fraud_rate > baseline["fraud_rate"] * 2:
print(f"ALERT: fraud rate {fraud_rate:.0%} vs {baseline['fraud_rate']:.0%} baseline")
if confidence < baseline["confidence"] - 0.1:
print(f"ALERT: mean confidence {confidence:.2f} vs {baseline['confidence']:.2f}")3Shadow and canary deploys
Before the new model replaces the old one: in shadow it receives the same requests and logs its predictions without using them, so you compare them with the old model and with the labels as they arrive. Then a canary: a small share of users see the new model, growing gradually while the metrics stay healthy.
Worked example
Problem: a support-ticket classifier was 92% accurate at launch, and 3 months later the team complains of odd labels. Measurement: PSI on text length is 0.31, and the word distribution shows terms from a new product absent from training. Fix: collect 2000 recent labelled tickets, retrain with the recorded pipeline, evaluate on last month's data (90%), shadow-deploy for a week, then switch over. Plus a standing alert on PSI > 0.2.
Myth vs Reality
| Myth | Reality |
|---|---|
| Retrain every month and you are done. | You may retrain for nothing or lag a sudden drift. Let the metrics decide. |
| Test accuracy stays the same in production. | Test data is a snapshot of the past. Production changes every day. |
Test your understanding
What is concept drift?
The same input now deserves a different answer.
PSI = 0.27 for a key feature. What does it mean?
Above 0.2 is usually considered major drift.
What is a shadow deploy?
You compare safely before any user is affected.
Quick review cards
Check Your Understanding
Why is server monitoring not enough?
Because a degraded model replies fast with no technical errors, just wrong answers.
How do you monitor without labels?
Through the input distribution (PSI), prediction shares and mean confidence.
When do you retrain?
When drift crosses a threshold or metrics worsen on recent labels.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.