Linear and logistic regression
Lesson 2 of 8
2Linear and logistic regression
The big pictureLinear and logistic regression are the two simplest models that learn from data, and everything after them in this roadmap, transformers included, is a variation on the same idea: weights adjusted to reduce a loss. Write them by hand once, then in three lines of scikit-learn.
What the picture shows: The line swings and then settles where the mean squared error is as small as possible. That is all training is: a weight and a bias adjusted step by step until the line sits close to the points, and then we read off the price of a 130 m² flat.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Linear regression is like drawing the best straight line through scattered points, so the line tells you the price of a house you have not seen from its size. Logistic regression takes the same idea and turns the output into a probability between zero and one, answering a yes-or-no question.
Linear regression y = w·x + b minimises mean squared error MSE. Logistic regression passes the output through a sigmoid and minimises cross-entropy loss. You write it by hand with gradient descent, then with sklearn.linear_model in three lines, reading the weights as each feature's effect.
Before You Start
📉Gradient descent
You wrote the descent loop by hand in the gradient lesson; here it repeats with two weights instead of one.
🎯Framing the problem
You tell regression from classification by the type of output, as in the first lesson of this track.
The Core Idea
Here you build the two simplest models that learn from data. You start with a line, then a number for how far it is from the points, then steps that bring it closer, then you turn the line into a probability and get a classifier.
1The model is a line
Linear regression learns one weight per input and one bias. With one input, the area, the model is a line.
Price ≈ w × area + b. The weight w is how much the price rises per extra square metre, and the bias b is the starting point. "Training" means finding the w and b that put the line as close to the points as possible.
Look at the picture: these are the values the model learns from the six flats. The weight is 4.291, the bias 49.82, and the 130 m² flat comes out at 607.7 thousand.
2The loss: mean squared error
We need one number that says how far the line is from the points. That number is the loss.
For each flat, the error is the gap between the true price and what the line says. Square the errors and average them, and you get one number: the mean squared error, MSE. Squaring makes a big error far costlier than a small one, and keeps the maths easy to differentiate.
Look at the picture: an error four times bigger costs 16 times more. Squaring also stops a positive error from cancelling a negative one.
3Stepping down the gradient
Training is a loop of four steps. Look at the picture: predict, compute the error, compute the gradient, move both weights a small step, and go again.
This is exactly what you did in the gradient lesson, now with two weights instead of one. We standardise area first (subtract the mean, divide by the spread) so the gradient is not huge.
import numpy as np
area = np.array([60, 80, 100, 120, 150, 180], dtype=float)
price = np.array([310, 390, 480, 560, 700, 820], dtype=float)
x = (area - area.mean()) / area.std()
w, b = 0.0, 0.0
for step in range(200):
pred = w * x + b
error = pred - price
w -= 0.1 * 2 * (error * x).mean()
b -= 0.1 * 2 * error.mean()
mse = ((w * x + b - price) ** 2).mean()
print(f"w={w:.2f} b={b:.2f} mse={mse:.2f}")
print("predict 130 m2:", round(w * (130 - area.mean()) / area.std() + b, 1))
Two hundred small steps brought us to a line that misses by under 4 thousand on average (the root of 14.68). The 130 m² flat is predicted at 607.7 thousand. Notice that b equals the mean price exactly, because standardised area has a mean of zero.
4The sigmoid and log loss
Logistic regression takes the same weighted sum and passes it through a sigmoid. Look at the picture: any number comes out between 0 and 1, and zero becomes exactly 0.5.
The sigmoid turns any number into a probability. Log loss measures how low the probability you gave the right answer was.
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def log_loss(y, p):
return -(y * np.log(p) + (1 - y) * np.log(1 - p))
print([round(float(sigmoid(z)), 3) for z in [-4, -1, 0, 1, 4]])
for p in [0.9, 0.6, 0.1]:
print(f"true=1 predicted={p} loss={log_loss(1, p):.3f}")
Zero becomes exactly 0.5, and large numbers approach 1 without reaching it. More important are the last three lines: a 0.9 probability on the right answer costs 0.105, 0.6 costs 0.511, but 0.1, high confidence in the wrong answer, costs 2.303, twenty-two times the first. Log loss punishes confident mistakes far more than it rewards being right.
That is why we do not use the square here. Log loss punishes confidence in the wrong answer very hard.
5From a probability to a decision
Despite its name, logistic regression is a classifier. It gives you a probability, and the decision comes from comparing it to a threshold.
import numpy as np
from sklearn.linear_model import LogisticRegression
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=float).reshape(-1, 1)
passed = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])
model = LogisticRegression().fit(hours, passed)
for h in [2, 5, 8]:
p = model.predict_proba([[h]])[0, 1]
print(f"{h} hours -> P(pass)={p:.2f} -> {model.predict([[h]])[0]}")
predict_proba gives the probability and predict compares it to a fixed 0.5 threshold. The student with five hours gets 0.39 and is labelled a fail, even though one student in the data passed with five hours. The 0.5 threshold is not sacred; in the metrics lesson you will choose it yourself from the cost of each error.
How It Works, Step by Step: how linear regression learns its line
Start with a zero weight and bias, and standardise the area so the gradient is not huge.
Predict each flat with w * x + b and compute the gap from the true price.
Square the errors and average them to get the MSE, and compute its gradient with respect to w and b.
Move w and b against the gradient with rate 0.1, and repeat two hundred times until the MSE is about 14.68.
Worked example
You have six flats with their areas and prices, and want the price of a seventh. Linear regression draws the best straight line through the points, then reads the price off the line.
Example: The same line in three lines
scikit-learn expects inputs as a 2-D table, one row per sample and one column per input, which is why we reshape(-1, 1) even with a single input.
import numpy as np
from sklearn.linear_model import LinearRegression
area = np.array([60, 80, 100, 120, 150, 180], dtype=float).reshape(-1, 1)
price = np.array([310, 390, 480, 560, 700, 820], dtype=float)
model = LinearRegression().fit(area, price)
print(f"slope={model.coef_[0]:.3f} intercept={model.intercept_:.2f}")
print("predict 130 m2:", round(model.predict([[130]])[0], 1))
Exactly the same prediction: 607.7. The slope of 4.291 means each extra square metre adds about 4.3 thousand. That interpretability is why linear regression wins when someone asks "why this price?".
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Logistic regression is a regression model. | Despite the name it is a classifier: it gives a class probability between 0 and 1, and the decision comes from comparing it with a threshold you choose. |
| A bigger weight means a more important feature. | Without standardised inputs a weight reflects the unit, not the importance: the weight on area in metres looks smaller than on number of rooms just because area numbers are bigger. |
| Simple models have no place in production. | When a complex model beats them by a small margin, the simple one is often the better choice: faster, clearer, fewer surprises. |
Try it yourself
You trained linear regression by gradient descent on an input ranging 0 to 100000 without standardising, and the loss became inf within a few steps. The likely cause?
The gradient multiplies the error by the input value, so an input around a hundred thousand gives a gradient in the hundreds of thousands, and a sensible learning rate for a small input becomes a huge leap. Standardising, or a much smaller rate, fixes it. It is the same explosion you saw with rate 1.1 in the gradient lesson.
A logistic model gave 0.02 to a case that was actually positive. What does log loss do?
The loss is -log(0.02), about 3.9, far larger than a confidently correct case. This is why confident mistakes move the weights most during training, and also why the model is sensitive to mislabelled data.
Quick review cards
Check Your Understanding
Why not use linear regression for classification directly, predicting 0 or 1 and rounding?
Because a straight line produces values outside the range, like 1.7 or -0.4, that are not probabilities, and because one far-off point, like a student who studied a hundred hours, drags the whole line and shifts the boundary between classes. The sigmoid keeps output between 0 and 1, and log loss cares more about points near the boundary than the obvious far ones.
At what value of z does the sigmoid give exactly 0.5, and what does that mean for the decision?
At z = 0. So with the default 0.5 threshold, the boundary between the classes is where the weighted sum equals zero.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.