Choosing the problem

Two ways into the same idea

Before You Start

The Core Idea

Before any code you write four things, in order: who the user is, what the success bar is, which baseline you must beat, and what you will not build.

1The user: a person with a name

Start from a real person you know and can sit with for an hour, not from "users in general". Learn what they do today without your system and how long it takes.

Look at the picture: Noura, a shop owner, reads support messages by hand, and 40% of them are the same question.

This is a problem she lives with every week, and it has a number you can measure: the time it costs. That is what makes it worth weeks of work.

2The success bar, before the code

Write the bar in the user's language first: "the system answers half the shipping messages alone, and gets no more than one in twenty wrong". Then translate it into a metric you compute.

Look at the picture: 50% coverage at 95% precision on the shipping class.

The translation reveals what you had not thought of: here precision matters more than coverage, because a wrong reply to a customer is worse than none.

You measure it on at least a hundred real cases from last month, labelled by the user, and you freeze them before you build.

3The baseline you must beat

Your model's number means nothing alone. Compare it with the simplest possible solution: if it does not beat it, the simple one is cheaper, faster and easier to explain.

Twenty real support messages from a small shop, labelled by the owner. Before any model, try the two simplest solutions: always answering the most common class, and a keyword rule you wrote in ten minutes.

from collections import Counter

# 20 real support messages from a small shop, labelled by the owner
messages = [
    ("وين طلبي؟ صار له أسبوع", "shipping"),
    ("كم سعر التوصيل للرياض؟", "shipping"),
    ("الشحنة متأخرة ومحد يرد", "shipping"),
    ("متى يوصل الطلب لجدة؟", "shipping"),
    ("رقم التتبع ما يشتغل", "shipping"),
    ("المندوب ما اتصل علي", "shipping"),
    ("هل التوصيل مجاني فوق ٢٠٠؟", "shipping"),
    ("طلبت من عشرة أيام وما وصل شي", "shipping"),
    ("أبي أرجع المنتج", "returns"),
    ("كيف أستبدل المقاس؟", "returns"),
    ("المنتج ما عجبني أبي فلوسي", "returns"),
    ("ابغى استرجاع المبلغ", "returns"),
    ("المنتج وصل مكسور", "damaged"),
    ("الكرتون مفتوح والقطعة ناقصة", "damaged"),
    ("اللون غير اللي في الصورة وفيه خدش", "damaged"),
    ("هل عندكم مقاس أكبر؟", "product"),
    ("القماش قطن ولا بوليستر؟", "product"),
    ("متى ينزل اللون الأسود؟", "product"),
    ("هل يناسب الغسالة؟", "product"),
    ("فيه ضمان على الجهاز؟", "product"),
]

labels = [label for _, label in messages]
majority, count = Counter(labels).most_common(1)[0]
print(f"majority baseline: always '{majority}' -> {count / len(labels):.0%}")

def keyword_rule(text):
    if any(w in text for w in ["طلب", "التوصيل", "الشحن", "يوصل"]):
        return "shipping"
    if any(w in text for w in ["أرجع", "استرجاع", "أستبدل"]):
        return "returns"
    if any(w in text for w in ["مكسور", "ناقصة", "خدش"]):
        return "damaged"
    return "product"

wrong = [(t, y, keyword_rule(t)) for t, y in messages if keyword_rule(t) != y]
print(f"keyword rule:      {1 - len(wrong) / len(messages):.0%}")
for text, truth, guess in wrong:
    print(f"  missed: {text}  (truth {truth}, rule said {guess})")

A ten-line rule reached 85%, and that number is optimistic because you wrote it while looking at the same messages. Still, it is the bar your model must beat on new messages, or else the rule is cheaper, faster and easier to explain. And look at the three misses: all of them are wording the rule did not expect ("tracking", "courier", "my money"). That is where a model might genuinely add value.

4The smallest scope, and what you will not build

Capstones die of scope, not of difficulty. Choose one task the user can try end to end within two weeks, not five half-done tasks.

Then put all of it on one page, with a list of the tempting things you deliberately postpone.

One page you write before the code and show the user for agreement. This is a filled template for the shop example.

# Scope: shipping replies for <shop name>

User:        Noura, owner, answers ~60 support messages a day by hand
Today:       ~2 hours a day; 40% of messages are "where is my order?"
Task:        classify incoming messages; for shipping status questions,
             draft a reply with the tracking status for Noura to send
Success bar: on 150 held-out real messages labelled by Noura:
             - shipping precision >= 95% (a wrong draft costs her trust)
             - drafts accepted without edits >= 60%
             - beats the keyword rule (85% on the dev messages) on new data
Baseline:    keyword rule, written and measured first
Data:        last month's messages, phone numbers and emails removed
             before they leave Noura's laptop
Not building: login, other message channels, returns, refunds,
             automatic sending without Noura's click, an app
Deadline:    first usable version in 2 weeks, stranger test in week 4

Every line can be checked later. "95% precision" is a number you measure, "60% without edits" is something you ask Noura about after a week, and "no automatic sending" is a decision that protects the user and reduces your risk. The "not building" list is what you return to every time a new feature tempts you.

How It Works, Step by Step: how to write the scope document before the code

Going deeper

Eval data comes from the user, not from you. Ask them for a hundred real cases from the last month, and to write the correct answer for each one themselves. If the data contains personal information, agree with them from the start on what gets removed and where it is stored. And freeze this set before you start building, so you neither add to it nor remove from it to improve your number.

Worked example

Before you write the success bar, you need to know how many examples it takes to measure it. "80% accuracy" on twenty examples is not a number but a very wide range. This code computes the real range for the same accuracy on eval sets of different sizes.

1. How many examples are enough?

import math

def wilson(correct, n, z=1.96):
    p = correct / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half

# the same 80% accuracy, measured on eval sets of different sizes
for n in [20, 50, 100, 300, 1000]:
    low, high = wilson(round(0.8 * n), n)
    print(f"n={n:5}: 80% means somewhere in {low:.0%} .. {high:.0%}  (width {high - low:.0%})")

On twenty examples, 80% means anything between 58% and 92%, so a weak system and an excellent one can give the same number. On a hundred examples the range narrows to 16 points, and on three hundred to 9. The practical rule for the capstone: at least a hundred examples in the eval set, and three hundred if you want to tell two close versions apart.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding