Project: imbalanced classification
Lesson 8 of 8
8Project: imbalanced classification
The big pictureThis track's project gathers everything so far into one deliverable: a naive baseline, a model that beats it, metrics fit for imbalanced data, a threshold chosen by cost, and a short report that persuades with numbers, not adjectives.
What the picture shows: On the test set, with a threshold of 0.034 chosen on validation, the model caught 66 of 96 frauds and missed 30, at the cost of 33 false alarms. That line in counts is what a manager understands, not PR-AUC.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
In this project you look for rare cases, like one fraudulent transaction among thousands of good ones. You start with the simplest possible solution to know your starting point, then build a model that beats it, and explain the difference in numbers a manager who knows no machine learning can follow.
A DummyClassifier baseline, then logistic regression, then boosting, with class_weight or resampling, and the threshold chosen on the precision-recall curve. The report compares PR-AUC and recall at a fixed precision, and prices the errors in money.
Before You Start
⚖️Metrics and thresholds
You compute precision and recall and choose a threshold on validation, from the metrics lesson.
🌲Gradient boosting
You know from the trees lesson why boosting usually wins on tabular data.
✂️Splitting
You split into training, validation and test with stratify and a fixed seed.
The Core Idea
Here you gather the whole track into one deliverable. You start with the baseline, compare two models on the right metric, choose the threshold on validation, then write a report that persuades with numbers.
1The baseline first
Only 3% of transactions are fraud. Look at the picture: the test set holds 96 cases out of 3000, a thin strip beside the majority.
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.metrics import average_precision_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=10000, n_features=12, n_informative=6, weights=[0.97], flip_y=0.005, class_sep=1.2, random_state=7)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=7, stratify=y)
print(f"positives in test: {y_te.sum()} of {len(y_te)}")
baseline = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
print(f"baseline accuracy: {baseline.score(X_te, y_te):.3f}")
print(f"baseline PR-AUC: {average_precision_score(y_te, baseline.predict_proba(X_te)[:, 1]):.3f}")
An accuracy of 96.8% with no intelligence at all. The area under the precision-recall curve, PR-AUC, is 0.032, exactly the positive share, which is always this metric's baseline. From now on every model is compared against 0.032, not 96.8%.
Write this number on the first line of the report. PR-AUC measures how well the rare cases are ranked, whatever the threshold, so it shows at once whether a model does anything.
2Two models on the same metrics
Do not try twenty models; try two well: logistic regression as an interpretable reference, and gradient boosting as the strong candidate.
The same split, two models, and a version of logistic regression with balanced class weights, the most famous trick for imbalanced data.
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=10000, n_features=12, n_informative=6, weights=[0.97], flip_y=0.005, class_sep=1.2, random_state=7)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=7, stratify=y)
models = {
"logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"logistic balanced": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, class_weight="balanced")),
"boosting": HistGradientBoostingClassifier(random_state=7),
}
print(f"{'model':18} PR-AUC precision recall")
for name, model in models.items():
model.fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]
pred = (proba >= 0.5).astype(int)
print(f"{name:18} {average_precision_score(y_te, proba):.3f} {precision_score(y_te, pred, zero_division=0):.2f} {recall_score(y_te, pred):.2f}")
Three stories in three lines. Plain logistic regression is perfectly precise when it alerts but catches only 7% of cases. Balanced weights lifted recall to 57%, but 94% of its alerts are false, and worse, PR-AUC dropped: the trick did not improve the ranking, it only moved the threshold. Gradient boosting reaches a PR-AUC of 0.729, more than twice the best linear model, because the data has non-linear relationships.
Look at the picture: balanced weights lowered PR-AUC, and boosting is more than double the best linear model.
3The threshold on validation
The threshold is a setting like any other, so you choose it on validation under a written constraint, then apply it to test unchanged. Look at the picture: the order never changes.
The business constraint: at least 70% of frauds must be caught. Among all thresholds meeting that on validation, pick the one with the best precision, then apply it to test unchanged.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import precision_recall_curve
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=10000, n_features=12, n_informative=6, weights=[0.97], flip_y=0.005, class_sep=1.2, random_state=7)
X_rest, X_te, y_rest, y_te = train_test_split(X, y, test_size=0.3, random_state=7, stratify=y)
X_tr, X_val, y_tr, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=7, stratify=y_rest)
model = HistGradientBoostingClassifier(random_state=7).fit(X_tr, y_tr)
precision, recall, thresholds = precision_recall_curve(y_val, model.predict_proba(X_val)[:, 1])
ok = recall[:-1] >= 0.7
best = np.argmax(np.where(ok, precision[:-1], -1))
t = thresholds[best]
print(f"threshold chosen on validation: {t:.3f}")
pred = (model.predict_proba(X_te)[:, 1] >= t).astype(int)
tp = int(((pred == 1) & (y_te == 1)).sum())
fp = int(((pred == 1) & (y_te == 0)).sum())
fn = int(((pred == 0) & (y_te == 1)).sum())
print(f"test: caught {tp} of {tp + fn} positives, {fp} false alarms")
The chosen threshold is 0.034, far from 0.5, which is normal with rare classes because the model is rarely confident about them. On test: we caught 66 of 96 frauds (69%, close to the 70% target) against 33 false alarms out of about 2,900 good transactions. That exact line goes into the report, because a manager understands "66 of 96 with 33 false alarms" and does not understand "PR-AUC 0.729".
4The one-page report
A convincing report is specific numbers in comparison, not adjectives. Look at the picture: five parts on one page.
Write the line a manager understands: we caught 66 of 96 frauds with 33 false alarms. Then give at least two cases where the model failed, with a likely explanation.
How It Works, Step by Step: how to run the project from start to report
Compute the baseline with DummyClassifier: 96.8% accuracy and a PR-AUC of 0.032, and write it as the first line.
Train logistic regression as an interpretable reference and gradient boosting as the strong candidate, and compare them by PR-AUC.
Choose the threshold on validation under a written constraint, such as at least 70% recall with the best possible precision.
Apply the threshold to test once, and write the counts: how many caught, how many missed, how many false alarms.
Worked example
The problem: only 3% of transactions are fraud. The deliverable is a notebook or script that runs the whole experiment from data to final table, plus a one-page report. We start with the step everyone skips.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Balanced class weights improve the model on imbalanced data. | In the lesson they raised recall from 7% to 57% but lowered PR-AUC. They effectively moved the threshold without improving how cases were ranked. |
| The default 0.5 threshold works here too. | With a rare class the model is rarely confident, so the chosen threshold came out at 0.034. That is normal, not a fault. |
| Try as many models as you can to find the best. | Two models studied well beat twenty. Every extra attempt on the same data adds luck to the winning number. |
Try it yourself
class_weight="balanced" raised recall from 7% to 57% but lowered PR-AUC. How do you explain it?
Recall at 0.5 changed because all probabilities shifted up for the rare class, an effect much like lowering the threshold. PR-AUC measures ranking across all thresholds and did not improve. You can get the same effect by moving the threshold on the plain model, an important lesson: imbalance "tricks" often change the threshold, not the quality.
Which line best fits the top of the report?
It says what happened in counts any reader understands, and compares against a reference. 98% accuracy is misleading here, an algorithm name says nothing about the result, and PR-AUC is correct but technical, so it belongs in the table, not the first sentence.
Quick review cards
Check Your Understanding
Your manager asks: "Should we replace the current rules with the model?" What does the report need to answer?
The current rules' result on the same test set with the same metrics, since that is the real baseline, not DummyClassifier. Then the difference in counts: how many extra frauds caught, how many more or fewer false alarms, and the cost of each in riyals if possible. Finally the limits: on which kind of transaction the model fails, and how we will monitor it after release.
Why is the baseline PR-AUC exactly 0.032?
Because a solution that does not rank cases at all has precision equal to the share of positives at any recall, which is 3.2% in the test set.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.