Feature engineering
Lesson 6 of 8
6Feature engineering
The big pictureA model only sees the numbers you give it. Turning cities, dates and text into meaningful numbers, and putting them on a common scale, often helps more than changing the algorithm. And every new feature has to prove it deserves its place.
What the picture shows: Three real features and only 120 rows. As noise features are added, validation accuracy falls: 89.2%, then 87.5% with twenty, then 63.3% with two hundred. Every meaningless feature is another chance for the model to memorise a phantom pattern.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A model does not understand the word "Riyadh" or a written date; it understands only numbers. Feature engineering translates information into numbers that carry meaning: a city into yes-or-no columns, a date into "day of week" and "is it a holiday". One good feature can help more than a more complex model.
one-hot encoding for few categories, target encoding for many with care about leakage, and StandardScaler for scale-sensitive models. You measure each new feature on the validation set and drop what does not move the number, since it is noise that adds overfitting.
Before You Start
✂️Splits and leakage
You know why every preprocessing step is fitted on training alone, and you use make_pipeline as in the splits lesson.
🌲Trees
You know from the trees lesson that a tree asks "is the value above a cut-off?".
The Core Idea
A model sees only the numbers you give it. Here you turn cities into numbers, put columns on one scale, pull features out of dates and text, and see why every feature has to earn its place.
1One column per category
Write Riyadh = 1, Jeddah = 2 and Dammam = 3, and a linear model reads Dammam as "three times" Riyadh. Look at the picture: each city becomes three numbers, and only one of them is 1.
One-hot encoding makes a column per city holding 1 if it is this row's city and 0 otherwise. No fake order, no invented distances.
import pandas as pd
df = pd.DataFrame({"city": ["Riyadh", "Jeddah", "Dammam", "Jeddah"], "rooms": [3, 2, 4, 3]})
print(pd.get_dummies(df, columns=["city"], dtype=int).to_string(index=False))
Four rows, three new columns. In production use OneHotEncoder(handle_unknown="ignore") inside a pipeline, because get_dummies does not remember its columns and a city unseen in training will break the shape.
An ordinal number fits only when the order is real, like small, medium and large.
2One scale for every column
Salary is in thousands and age in tens. A distance-based model sees a thousand-riyal gap as far more important than a ten-year gap, just because the number is bigger.
Standardising subtracts the mean and divides by the standard deviation, so every column has mean zero and spread one.
The output depends only on age (over forty or not). Salary is noise with big numbers.
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(0)
n = 600
salary = rng.normal(9000, 3000, n)
age = rng.uniform(20, 60, n)
y = (age > 40).astype(int)
X = np.column_stack([salary, age])
raw = cross_val_score(KNeighborsClassifier(), X, y, cv=5).mean()
scaled = cross_val_score(make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y, cv=5).mean()
print(f"kNN raw: {raw:.3f}")
print(f"kNN scaled: {scaled:.3f}")
Without scaling, 57.8%, near chance, because the distance is almost all salary. With scaling, 98.7%. Same data, same model, one line apart. And note the scaler sits inside make_pipeline so it does not leak from validation folds.
3Features from dates and text
A raw date as a number barely helps a model. Look at the picture: one date yields the hour, the weekday, and whether it is a weekend.
import pandas as pd
orders = pd.DataFrame({
"placed_at": pd.to_datetime(["2026-03-05 08:10", "2026-03-06 21:40", "2026-03-07 13:05"]),
"note": ["عاجل جدا", "", "الرجاء الاتصال قبل الوصول"],
})
orders["hour"] = orders.placed_at.dt.hour
orders["weekday"] = orders.placed_at.dt.day_name()
orders["is_weekend"] = orders.placed_at.dt.day_of_week.isin([4, 5]).astype(int)
orders["note_words"] = orders.note.str.split().str.len()
orders["is_urgent"] = orders.note.str.contains("عاجل").astype(int)
print(orders.drop(columns=["placed_at", "note"]).to_string(index=False))
One date column became three features and one text column became two. day_of_week starts at Monday = 0, so Friday is 4 and Saturday is 5. That is local knowledge the model lacks: elsewhere the weekend falls on different days.
Text gives simple, strong features too: word count, or whether a word like "urgent" appears. These come long before embeddings.
4Every feature earns its place
A feature that is noise teaches the model phantom patterns, especially with little data. So every feature is measured on validation before it stays.
Three real features, then we add completely random ones. With only 120 rows, watch what happens.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(1)
n = 120
signal = rng.normal(size=(n, 3))
y = (signal.sum(axis=1) + rng.normal(scale=0.8, size=n) > 0).astype(int)
for extra in [0, 20, 200]:
X = np.hstack([signal, rng.normal(size=(n, extra))])
score = cross_val_score(LogisticRegression(max_iter=2000), X, y, cv=5).mean()
print(f"3 real + {extra:3} noise features -> {score:.3f}")
From 89.2% to 87.5% with twenty noise features, then 63.3% with two hundred. The model found "patterns" in the noise that fit training and nothing else. The less data you have, the more each extra feature is a gamble.
How It Works, Step by Step: how to decide a new feature earns its place
Record the current model's validation number before adding anything.
Add one feature only, computed from data available at prediction time and from training alone.
Rerun cross-validation with the same folds and compare the new number with the old one.
Keep the feature if the number improves by more than the spread across folds, drop it if not, and log the decision in a table.
Worked example
A city is text and the model needs numbers. Write Riyadh = 1, Jeddah = 2, Dammam = 3, and a linear model reads Dammam as "twice" Riyadh, which is nonsense.
2. Different scales fool distances
Salary is in thousands and age in tens. A distance-based model like nearest neighbours will treat a thousand-riyal gap as a hundred times more important than a ten-year gap, just because the number is bigger.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Numbering categories 1, 2 and 3 is enough. | A linear model will read Dammam as "twice" Riyadh. Use one column per category unless the order is real. |
| More features mean a better model. | With 120 rows accuracy fell from 89.2% to 63.3% after adding two hundred noise features. |
| A customer ID is a useful feature because it is unique. | The model memorises that customer 4471 bought, which helps with no new customer, so it looks excellent in training only. |
Try it yourself
Which model's performance will barely change if you scale the inputs?
Trees ask "is the value above a cut-off?", and scaling does not change the order of values, so the possible questions are identical. Nearest neighbours relies on distances, and regression and networks on gradient descent, all scale-sensitive.
You added "customer's average monthly sales" computed over all the data and accuracy jumped from 70% to 96%. First suspicion?
An average over the whole period includes sales that happened after prediction time, possibly the very sales you are predicting. A sudden huge jump from one feature is a classic leakage signal. Compute the feature from each row's past only.
Quick review cards
Check Your Understanding
You have a "device type" column with 3 values and a "product code" column with 40,000. Do you one-hot both?
Device type yes: three columns, no problem. Product code no: forty thousand mostly-zero columns, and many products with one or two samples the model will memorise. Alternatives: group rare products into "other", use a higher-level feature like product category, or target encoding computed inside training folds only so it does not leak.
You have a "size" column with small, medium and large. Which encoding do you pick?
An ordinal encoding such as 0, 1 and 2, because the order is real here: large really is bigger than medium, so the numbers invent no false relationship.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.