Splitting the data

Two ways into the same idea

Before You Start

The Core Idea

A number measured on data the model has seen says nothing about the future. Here you build a three-part split, see how the answer leaks, measure safely with cross-validation, and split by time.

1Three parts, one job each

Training learns the weights, validation picks the model and settings, and test gives one final number. Look at the picture: the row count of each part of the breast cancer data.

Training is what the model learns from. Validation is where you choose between models and settings. Test is opened once at the end to get the number you will report. We split with train_test_split twice, with stratify so the class balance stays the same in all three parts.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
for name, part in [("train", y_train), ("val", y_val), ("test", y_test)]:
    print(f"{name:5} rows={len(part):3} positive={part.mean():.2f}")

60%, 20% and 20%, with the positive share at 0.62 to 0.63 in every part. Without stratify a small part can land with a different balance by chance, and you compare models on data unlike reality.

2Why two parts are not enough

Choosing is a kind of training too. Look at the picture: you try many settings and pick the best on validation, and only then reach the test set, once.

If you try twenty settings and pick the best on the test data, you have "trained" your choice on it. The winning number is optimistic because you selected it from twenty attempts. Validation absorbs that selection, and the test set stays clean.

3Leakage: a model out of noise

Leakage is information from evaluation data reaching training. Its most dangerous form is processing done before the split.

Completely random data: two thousand columns of random numbers and a random output. There is no real pattern, so any honest model should land near 50%. Watch what happens when we pick the "best" ten columns before splitting.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 2000))
y = rng.integers(0, 2, size=100)


def top_features(X, y, k=10):
    corr = np.abs([np.corrcoef(X[:, j], y)[0, 1] for j in range(X.shape[1])])
    return np.argsort(corr)[-k:]


best = top_features(X, y)
X_tr, X_te, y_tr, y_te = train_test_split(X[:, best], y, test_size=0.5, random_state=0)
leaky = LogisticRegression().fit(X_tr, y_tr).score(X_te, y_te)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, random_state=0)
best = top_features(X_tr, y_tr)
honest = LogisticRegression().fit(X_tr[:, best], y_tr).score(X_te[:, best], y_te)

print(f"leaky accuracy on pure noise:  {leaky:.2f}")
print(f"honest accuracy on pure noise: {honest:.2f}")

74% on pure noise. The reason: feature selection saw the test set, so it chose columns that happen to correlate with the test outputs. The honest way, selecting from the training half only, gives 46%, chance, as it should. Ship the first model and it collapses immediately.

Look at the picture again: the code runs and the number is great, and nothing says there is a bug. So you hunt for leakage on purpose, and the rule is simple: split first, then do everything else on training only.

4Cross-validation

One number from one split may be luck. Cross-validation cuts the data into five folds and makes each fold the validation set once.

make_pipeline ties preprocessing to the model, so scaling is fitted inside each training fold separately. It is the only safe way to cross-validate with any preprocessing.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X, y, cv=5)
print("folds:", [round(float(s), 3) for s in scores])
print(f"mean={scores.mean():.3f} std={scores.std():.3f}")

Five results between 0.974 and 0.991. The 0.007 spread says a gap under about 0.01 between two models may be split luck rather than real improvement. Always report the mean and the spread together.

5Splitting by time

If the model will be used on the future, like sales or fraud, a random split shows it tomorrow to predict yesterday. Train on the past and evaluate on what follows.

import pandas as pd

days = pd.date_range("2026-01-01", periods=10, freq="D")
df = pd.DataFrame({"day": days, "sales": [12, 15, 14, 18, 20, 19, 23, 25, 24, 28]})
cut = int(len(df) * 0.7)
train, test = df.iloc[:cut], df.iloc[cut:]
print("train:", train.day.min().date(), "->", train.day.max().date())
print("test: ", test.day.min().date(), "->", test.day.max().date())
print("no future row in train:", train.day.max() < test.day.min())

Sort by time, then cut, no shuffling. For time-aware cross-validation scikit-learn offers TimeSeriesSplit, which always trains on the past and evaluates on what follows.

How It Works, Step by Step: how to split data without leakage

Worked example

Picture a student who saw the exam questions beforehand. A perfect score tells you nothing about understanding. A model is the same: you must hold back data it never saw to measure it on.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding