Applied statistics
Lesson 7 of 9
7Applied statistics
The big pictureA model at 91% accuracy on 100 examples might really be 85% or 97%. Statistics tells you how far to trust the number, and whether the gap between two models is real or chance.
What the picture shows: From raw numbers to a grounded decision. Each step answers a question before the next.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Ask 10 people about your coffee and 8 like it: do 80% of all your customers like it? Maybe, or maybe you only asked friends. Statistics says: with 10 people the real number could be between 50% and 95%. With 1000 people the range narrows a lot. And if sales rose when it rained, it does not mean rain causes a love of coffee.
Description: the mean is sensitive to outliers, the median is not, and the standard deviation measures spread. The normal distribution appears in averages (the central limit theorem), and real data is often skewed. The standard error = σ/√n, and a 95% confidence interval ≈ the mean ± 1.96 × the standard error, or via the bootstrap with no assumptions. A/B testing: a null hypothesis, a p-value being the chance of a gap this large if there were no real gap, a sample size fixed before the experiment, and peeking repeatedly inflates false positives. Correlation (Pearson's coefficient) does not mean causation: hidden variables, reversed direction, and chance. And Simpson's paradox flips a result when data is aggregated.
Before You Start
🎲Probability
The probability lesson in this track.
🐍Python for data
This track's first lesson.
The Core Idea
1Describe before you model
A small team's salaries with one huge one. The mean misleads; the median describes the typical person:
import statistics as st
salaries = [8, 9, 9, 10, 11, 12, 95] # thousands
print("mean :", round(st.mean(salaries), 1))
print("median:", st.median(salaries))
print("stdev :", round(st.stdev(salaries), 1))
q1, q2, q3 = st.quantiles(salaries, n=4)
print("IQR :", q1, "to", q3)2How far do you trust your number?
A model's accuracy on a test sample is an estimate. The confidence interval says where the real number most likely lies, and it narrows as the sample grows:
import math
def accuracy_interval(correct, total, z=1.96):
p = correct / total
margin = z * math.sqrt(p * (1 - p) / total)
return round(p, 3), round(max(0, p - margin), 3), round(min(1, p + margin), 3)
for correct, total in [(91, 100), (910, 1000), (9100, 10000)]:
p, low, high = accuracy_interval(correct, total)
print(f"n={total:>5}: accuracy {p} (95% CI {low} to {high})")3An A/B test by permutation
Did the new design really raise conversion? A permutation test asks: if we shuffled the two groups at random, how often would we see a gap this large? We use a fixed seed for a reproducible result:
import random
a = [1] * 48 + [0] * 452 # old design: 48 of 500 converted
b = [1] * 70 + [0] * 430 # new design: 70 of 500 converted
observed = sum(b) / len(b) - sum(a) / len(a)
rng = random.Random(7)
pooled = a + b
extreme = 0
trials = 2000
for _ in range(trials):
rng.shuffle(pooled)
gap = sum(pooled[500:]) / 500 - sum(pooled[:500]) / 500
if gap >= observed:
extreme += 1
print(f"observed lift: {observed:.3f}")
print(f"p-value about: {extreme / trials:.3f}")Worked example
Problem: a new model at 88% accuracy against 86% for the old one, on 300 test examples. Do we ship it? Maths: a 95% interval for the new one is roughly 84% to 92%, and for the old 82% to 90%. They overlap. Decision: not enough evidence. Grow the test to 3000 examples, or compare on the same examples with a paired test (McNemar), before calling it better.
Myth vs Reality
| Myth | Reality |
|---|---|
| A p-value below 0.05 means the result matters. | It means it is probably not chance. The gap may be real and too small to be worth it. |
| More data solves everything. | More data with the same bias gives a wrong answer with more confidence. |
Test your understanding
Which measure is not swayed by a single outlier?
The median is the middle value after sorting.
What happens to the confidence interval as the sample grows?
The standard error shrinks with the square root of the sample size.
Umbrella sales and colds rise together. The likeliest explanation?
Correlation does not mean causation.
Quick review cards
Check Your Understanding
When do you use the median instead of the mean?
When the data is skewed or has outliers, like salaries and prices.
Why not stop an A/B test at the first nice gap?
Because chance creates temporary gaps, and repeated peeking raises false positives.
95% accuracy on 40 examples. Your remark?
The sample is small and the interval wide, so the true figure may be much lower.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.