Metrics
Lesson 4 of 8
4Metrics
The big pictureAccuracy lies when classes are imbalanced, which is the case in most problems worth solving: fraud, disease, faults. Precision, recall and the confusion matrix tell you what actually happened.
What the picture shows: Twelve predictions fall into four cells: 3 true positives, 2 false alarms, 1 miss and 6 true negatives. Every classification metric comes from these cells: precision is 3 of 5 alerts, recall is 3 of 4 real cases.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
If one person in a hundred is ill, a model that always says "everyone is healthy" is right 99% of the time, yet catches no one. That is why "percent correct" is not enough. You ask: of the ill, how many did we catch? And of those we called ill, how many really were?
A confusion matrix counts true and false positives and negatives. precision = TP / (TP + FP), recall = TP / (TP + FN), and F1 is their harmonic mean. ROC-AUC and PR-AUC measure performance across all thresholds, the second more honest with rare classes.
Before You Start
✂️Splitting data
You know from the splits lesson why any setting is chosen on validation, not on test.
🔀Logistic regression
You know predict_proba returns a probability and predict compares it with a 0.5 threshold.
The Core Idea
Accuracy lies when classes are imbalanced. Here you start with a case where it lies, replace it with two questions, compute them from four cells, then move the threshold between them.
1Why accuracy misleads
A hospital runs 100 tests, and five are truly sick patients. A lazy model says "healthy" to everyone.
import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score
y_true = np.array([0] * 95 + [1] * 5)
lazy = np.zeros(100, dtype=int)
print("accuracy :", accuracy_score(y_true, lazy))
print("recall :", recall_score(y_true, lazy))
print("precision:", precision_score(y_true, lazy, zero_division=0))
95% accuracy and zero recall: it did not catch a single patient. The first number says "excellent"; the second tells the truth.
Look at the picture: accuracy treats every error as equal, so the rare class drowns in the majority.
2Two questions instead of one
Every prediction lands in one of four cells. Look at the picture: the model said positive or negative, and the truth was positive or negative.
Recall: of all truly positive cases, how many did we catch? Precision: of everything we flagged positive, how many really were? The first punishes misses; the second punishes false alarms.
Precision = TP / (TP + FP), and every false alarm lowers it. Recall = TP / (TP + FN), and every miss lowers it, which is often the costlier error.
3From the matrix to the metrics
The confusion matrix is these four cells in a table. Every other classification metric is computed from it.
Twelve predictions, four truly positive cases. Count the four cells, then derive the metrics from them.
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
y_pred = [1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"precision={precision_score(y_true, y_pred):.2f}")
print(f"recall ={recall_score(y_true, y_pred):.2f}")
print(f"f1 ={f1_score(y_true, y_pred):.2f}")
Precision 3 / (3 + 2) = 0.60: of five alerts, three were right. Recall 3 / (3 + 1) = 0.75: of four cases, we caught three. Work them out from the cells by hand once and you will remember them for good. Mind the order of ravel() in scikit-learn: tn, fp, fn, tp.
F1 folds the two into one number, but it assumes both errors matter equally, which is rare.
4The threshold moves the trade-off
The model gives a probability, and the threshold decides when we say "positive". Look at the picture: as the threshold rises, precision goes up and recall goes down.
One model, three thresholds. The probabilities never change, only the point at which we say "positive".
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, weights=[0.9], random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=1, stratify=y)
proba = LogisticRegression().fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
for t in [0.2, 0.5, 0.8]:
pred = (proba >= t).astype(int)
print(f"threshold={t} precision={precision_score(y_te, pred):.2f} recall={recall_score(y_te, pred):.2f}")
At 0.2 we catch 88% of cases but a third of our alerts are false. At 0.8 our alerts are 93% right but we miss half the cases. There is no mathematically "correct" threshold, only the right one for the cost of your errors. And because you are choosing it, choose on validation, then confirm on test.
How It Works, Step by Step: how to choose a classification threshold
Compute the positive class probability for every validation row with predict_proba.
Try several thresholds, and at each one compute precision and recall from the confusion matrix cells.
Estimate the real cost of each error: what a miss costs and what a false alarm costs.
Pick the threshold that fits those costs, then confirm it once on the test set without changing it.
Going deeper
| Truth: positive | Truth: negative | |
|---|---|---|
| Predicted: positive | TP true positive | FP false alarm |
| Predicted: negative | FN miss | TN true negative |
Worked example
A hospital runs 100 tests; five are truly sick patients. A lazy model says "healthy" to everyone.
average_precision_score.
classification_report after every training run. One line per class with precision, recall and sample count, so a bad class cannot hide behind a good average.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| High accuracy means a good model. | A lazy model saying "healthy" to 100 tests with five sick patients scores 95% with zero recall. |
| 0.5 is the mathematically correct threshold. | It is only a default. There is no mathematically correct threshold, only one that fits the cost of your errors. |
| F1 settles the choice between the two metrics. | F1 is handy as one number, but it assumes both errors matter equally, which is rare in practice. |
Try it yourself
A manufacturing defect model raised 40 alerts, 30 of them real defects, and missed 10 defects. What are precision and recall?
TP = 30 and FP = 10 (forty alerts minus thirty), so precision is 30/40 = 0.75. FN = 10, so recall is 30/(30+10) = 0.75. They match only because FP and FN are both 10.
You lower the threshold from 0.5 to 0.3. What do you expect?
A lower threshold means more alerts: we catch extra cases whose probabilities were between 0.3 and 0.5, so recall rises, but some of those extra alerts are wrong, so precision falls. Recall can never drop when you lower the threshold.
Quick review cards
Check Your Understanding
A loan screening system sends suspicious applications to a human reviewer who can check only 50 a day. Which metric matters most?
Precision among the top 50 alerts, known as precision@50. The reviewer cannot see more than 50, so high recall with a thousand alerts is useless. What matters is that the first fifty, ranked by probability, are full of real cases. The practical constraint picks the metric.
In the lesson's matrix, why is accuracy 0.75 while precision is 0.60?
Accuracy counts every correct answer, positive and negative: (3 + 6) out of 12 = 0.75. Precision looks only at the alerts: 3 out of 5.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.