Evaluation first

Two ways into the same idea

Before You Start

The Core Idea

Before you change anything in your system you need a number to compare against. In this lesson you build a golden set, learn where its questions come from, calibrate a model judge, then expose its bias in pairwise comparison.

1The golden set

A golden set is real questions with what their right answers must contain, fixed across experiments. The simplest measure: does the answer contain the required phrases?

Look at the picture: two versions of the system on the same five questions, and each version now has a number.

golden = [
    {"q": "How many annual leave days?", "must": ["21"]},
    {"q": "Who approves sick leave?", "must": ["manager"]},
    {"q": "Can leave be carried over?", "must": ["10 days", "next year"]},
    {"q": "Notice period for annual leave?", "must": ["two weeks"]},
    {"q": "What is the parking fee?", "must": ["not covered"]},
]
v1 = ["You get 21 days.", "HR approves it.", "Yes, up to 10 days into next year.",
      "Two weeks notice.", "The fee is 50 riyals."]
v2 = ["21 working days.", "Your direct manager.", "Up to 10 days, used by March next year.",
      "At least two weeks before.", "Parking is not covered by these documents."]

def score(answers):
    hits = [all(m.lower() in a.lower() for m in g["must"]) for g, a in zip(golden, answers)]
    return sum(hits), hits

for name, answers in [("v1", v1), ("v2", v2)]:
    n, hits = score(answers)
    print(f"{name}: {n}/{len(golden)}  {['ok' if h else 'FAIL' for h in hits]}")

The first version got wrong who approves sick leave and invented a parking fee that does not exist. The second passed all five. The fifth question matters: it is outside the documents, and the right answer is "not covered". Every golden set needs questions like this, or it rewards a system that makes things up.

2Where the questions come from

A good set has questions from inside the documents and questions from outside them. Look at the picture: of fifty questions, ten are outside the documents, and their right answer is that the information is not there.

From real users, not your imagination. Collect questions from email, tickets or a session with five people, and write each right answer with whoever holds the knowledge. Fifty varied questions beat five hundred similar ones.

3A model as judge

Text matching is enough for short facts, and not when the right answer can be phrased many ways. That is where a language model grades the answer against a written rubric of yes-or-no questions.

But the judge needs calibrating. Look at the picture: twenty answers graded by a person and by the judge, who agreed on 17 and differed on 3.

Twenty answers graded by a person (1 good, 0 bad) and by the model.

from sklearn.metrics import cohen_kappa_score

human = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0]
judge = [1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1]

agree = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"raw agreement: {agree:.0%}")
print(f"Cohen's kappa: {cohen_kappa_score(human, judge):.2f}")
fp = sum(1 for h, j in zip(human, judge) if j == 1 and h == 0)
print("judge said 'good' where the human said 'bad':", fp, "times")

85% agreement looks excellent, but kappa of 0.67 is more precise because it removes agreement that happens by chance. More important: all three errors go one way, the judge said "good" to bad answers. A lenient judge makes you think your system is better than it is. Read those three cases yourself and adjust the rubric to catch them.

Calibrate the judge on 30 to 50 answers you graded yourself before you trust it. A kappa above about 0.6 is a good start; more important is knowing where it errs.

4Pairwise comparison and position bias

Instead of "score from 1 to 10", ask the judge: which of the two answers is better? This is more stable, but the judge leans towards whichever answer comes first when the two are close.

Look at the picture: quality is random for both sides, so A should win about 100 times out of 200. Yet it wins 139 times when always shown first.

A judge that picks the better answer when the gap is clear, and leans towards the first one when the two are close. This bias is documented in real models.

import random

random.seed(3)
def biased_judge(first, second):
    """Prefers the better answer, but also prefers whatever is shown first."""
    if abs(first["quality"] - second["quality"]) < 0.15:
        return "first" if random.random() < 0.75 else "second"
    return "first" if first["quality"] > second["quality"] else "second"

pairs = [({"name": "A", "quality": random.uniform(0.4, 0.9)},
          {"name": "B", "quality": random.uniform(0.4, 0.9)}) for _ in range(200)]

one_order = sum(biased_judge(a, b) == "first" for a, b in pairs)
consistent = ties = 0
for a, b in pairs:
    r1 = biased_judge(a, b) == "first"          # did A win when shown first?
    r2 = biased_judge(b, a) == "second"         # did A win when shown second?
    if r1 == r2:
        consistent += 1
    else:
        ties += 1
print("A wins when always shown first:", one_order, "of 200")
print("verdicts that survive swapping the order:", consistent)
print("verdicts that flip with the order (count as ties):", ties)

Quality is random for both sides, so A should win about half the time. But when always shown first it wins 139 times out of 200. When we judge twice in both orders, 72 verdicts flip with the order, and we count those as ties. The lesson: never compare two versions by always showing the new one first; you will always find it "better".

How It Works, Step by Step: how to build a golden set and use it

Worked example

A golden set needs no special tools. A file with a question and what the right answer must contain. We start with the simplest measure: does the answer contain the required phrases?

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding