Latency and cost
Lesson 2 of 6
2Latency and cost
The big pictureThe mean lies about latency. A service averaging 220 ms may leave one user in a hundred waiting a second and a half or more, and a user sending twenty requests in a session will likely hit that case. This lesson is about percentiles, time to first token, batching, and the cost of a thousand requests.
What the picture shows: The mean is 222 and the median 181, and both look fine. But p99 is nearly 1.3 seconds, so one request in a hundred is several times slower. That is why targets are written in percentiles, not the mean.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
If most customers wait one minute but one in twenty waits ten, the average looks fine, yet that customer will not come back. That is why we measure the slowest, not just the average. With language models it also matters when the first word appears.
p50 is the median, and p95 and p99 are the tail the worst-served users live in. For generation you measure time to first token TTFT and tokens per second. Cost is per token or per hardware second per request, and batching raises throughput at the expense of latency.
Before You Start
🌐From a notebook to a service
You have a service that takes requests, and now you measure how long they take.
🎲Probability
You know distributions and the median, and that a distribution can have a long tail.
The Core Idea
The mean lies about latency. Here you learn to read percentiles, then the two phases of a language model and time to first token, then batching, then the cost of a thousand requests.
1Percentiles, not the mean
p50 is the time half the requests finish within, and p95 the time 95% finish within. The second measures the unluckiest users' experience.
We simulate 10,000 requests: most around 180 ms, and a few hitting a slow path.
import numpy as np
rng = np.random.default_rng(0)
latency_ms = rng.lognormal(mean=np.log(180), sigma=0.45, size=10_000)
latency_ms[rng.choice(10_000, 150, replace=False)] *= 8 # 1.5% hit a slow path: a cold cache, a long input
print(f"mean: {latency_ms.mean():7.0f} ms")
for p in [50, 95, 99]:
print(f"p{p}: {np.percentile(latency_ms, p):7.0f} ms")
print(f"max: {latency_ms.max():7.0f} ms")
The mean is 222 and the median (p50) 181; both look fine. But p95 is twice the median, p99 over a second and a quarter, and the worst near five seconds. Watching the mean alone, you would never know one request in a hundred takes seven times the usual. Always watch p50, p95 and p99, and write your targets in them: "p95 under 500 ms".
2Two phases: prefill and decode
A language model reads the whole prompt at once in parallel, which is fast. Then it writes the answer token by token, each token needing a full pass, which is slow.
Look at the picture: reading 1,200 tokens is a thin sliver, and writing 300 tokens takes almost the whole bar.
prompt_tokens, answer_tokens = 1200, 300
prefill_tok_per_s = 6000 # reading the prompt is fast and parallel
decode_tok_per_s = 40 # writing is one token at a time
ttft = prompt_tokens / prefill_tok_per_s
total = ttft + answer_tokens / decode_tok_per_s
print(f"time to first token: {ttft:.2f} s")
print(f"time to full answer: {total:.2f} s")
print(f"with streaming, the user starts reading after {ttft:.2f} s instead of {total:.2f} s")
Reading 1,200 tokens takes 0.2 seconds, writing 300 takes 7.5. Without streaming the user stares at a blank screen for seven and a half seconds. With streaming they start reading after a fifth of a second, and the answer completes as they read. The same compute, a completely different experience.
That is why an output token costs more than an input token, and why a shorter answer is the fastest way to cut latency and cost together.
3Batching: throughput against latency
Batching processes several requests in one pass. Each pass has a fixed cost, so spreading it over more requests raises throughput and adds a little latency to each request.
Look at the picture: throughput rises fast then flattens, while latency starts slow then jumps. The mark is at a batch of 16.
Each pass has a fixed cost (40 ms) and a cost per request in the batch (6 ms). The card costs $1.20 an hour.
overhead_ms, per_item_ms = 40, 6 # each forward pass has a fixed cost plus a cost per request in the batch
gpu_cost_per_hour = 1.20
print("batch latency requests/s $ per 1000 requests")
for batch in [1, 4, 16, 64]:
latency = overhead_ms + per_item_ms * batch
throughput = batch / (latency / 1000)
cost = gpu_cost_per_hour / (throughput * 3600) * 1000
print(f"{batch:5} {latency:5.0f} ms {throughput:9.0f} {cost:10.4f}")
From a batch of 1 to 16, throughput goes from 22 to 118 requests a second and cost per thousand falls below a fifth, while latency goes from 46 to 136 ms. After that the gains shrink: a batch of 64 triples latency for only 28% more throughput. Choose the batch that meets your p95 target at the lowest cost, not the largest possible.
4Cost per thousand requests
Cost per thousand requests is the hourly price divided by requests served per hour, times 1000. You compare settings and providers with this number.
Look at the picture: the same card at $1.20 an hour, and the cost falls with batch size.
Choose the batch that meets your p95 target at the lowest cost, not the largest possible.
How It Works, Step by Step: how a language model's response time is made
The request waits in the queue if the card is busy, and may be batched with other requests.
The whole prompt is read at once in parallel, a fast phase: 1,200 tokens in 0.2 seconds in the example.
The answer is generated token by token, each token a full pass: 300 tokens in 7.5 seconds.
With streaming the user sees the first token after the reading phase, and starts reading before the answer is complete.
Worked example
Ten thousand requests with a realistic distribution: most around 180 ms, and a few hitting a slow path (a cold cache, a long input). Let us compare the mean with the percentiles.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| The mean is enough to describe a service's speed. | It hides the tail. In the example the mean was 222 and the worst near five seconds. |
| p99 is a rare case that does not matter. | A user sending fifty requests meets it with about 40% chance, so it is what most active users hit every few sessions. |
| The largest possible batch is best. | After a batch of 16 the gains shrink: 64 tripled latency for 28% more throughput. Pick what meets your p95 target at the lowest cost. |
Try it yourself
Which number reflects the unluckiest users' experience?
The mean and median hide the tail. p95 says: 5% of requests are slower than this.
A chat system takes 8 seconds for the full answer. What is the quickest improvement the user feels?
The compute does not change, but the felt wait drops from 8 seconds to under one. The next step: shorter answers.
Quick review cards
Check Your Understanding
The mean is 200 ms and p99 three seconds. A user sends 50 requests in a session. What is the chance they hit at least one very slow request?
About 40%: the chance all fifty fall within the 99% is 0.99 to the power 50, about 0.6. That is why p99 is not a "rare case" for your active users but something most of them meet every few sessions.
A card at $1.20 an hour serves 118 requests a second. How do you work out the cost per thousand requests?
Divide the hourly price by requests per hour (118 × 3600) and multiply by 1000, giving about $0.0028, as in the batching table.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs3
Done with this lesson?
Sign in to save your progress.