Before you fine-tune
Lesson 1 of 6
1Before you fine-tune
The big pictureFine-tuning changes the model's own weights, and it is the last rung on the ladder, not the first. Before it come three cheaper, faster rungs: a better prompt, then examples in the prompt, then retrieving information from your documents. Each rung solves a different kind of problem, and tuning solves a very specific kind.
What the picture shows: Few-shot prompting costs $180 every month, and the tuned model starts at $600 once, then $25 a month. The lines cross just before month four, and every month after that is a saving, provided quality is at least equal.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Before sending an employee on a long, expensive course, you try the easy things: clearer instructions, then examples to copy, then a reference file to read from. Fine-tuning is the expensive course, used only when nothing before it works.
The ladder in order: prompt engineering, then few-shot examples, then RAG for changing knowledge, then fine-tuning to teach a format, style or narrow stable task. Fine-tuning is poor at adding fresh knowledge and costs data, training and maintenance for every version.
Before You Start
🔎Retrieval
You have built a system that answers from your documents, and know it is the fix for changing and private information.
🧱Context engineering
You know how to improve the template and add examples to it, the two cheapest rungs on the ladder.
The Core Idea
Fine-tuning is the last rung of the ladder, not the first. In this lesson you see the ladder, which problems tuning fixes and which it does not, when it pays for itself, and its real cost.
1A ladder of four rungs
Before tuning come three cheaper, faster rungs: a better prompt, then examples in the prompt, then retrieval from your documents. Climb a rung only when you measure that the one below falls short.
Look at the picture: tuning sits at the end of the road. Each rung before it solves a different kind of problem.
2What tuning fixes, and what it does not
The first question: what exactly is the problem? If the model does not know a fact, like your company's policy or today's prices, the fix is retrieval. If it knows what to do but does not keep to the format or style, that is where tuning may help.
Look at the picture: the road to tuning goes through a format problem, not a missing fact.
Tuning fixes a consistent format, style, narrow repeated tasks, and moving a task a large model does well into a small one. It does not fix changing information, reasoning the base model cannot do, or errors caused by bad data.
3The break-even point
The second question: is it worth its price? Examples in the prompt are paid in tokens on every request, and tuning is paid once. So the monthly volume decides.
A system classifies 200 thousand requests a month and needs 12 examples in every prompt to work well. A tuned model needs no examples.
requests_per_month = 200_000
price_in = 0.50 / 1e6 # dollars per input token, an example price
few_shot_tokens = 1800 # 12 worked examples in every prompt
tuned_tokens = 250 # the tuned model needs only the instruction
tuning_cost = 600 # one-off: data preparation and the training run
monthly_few_shot = requests_per_month * few_shot_tokens * price_in
monthly_tuned = requests_per_month * tuned_tokens * price_in
print(f"few-shot prompting: ${monthly_few_shot:,.0f} per month")
print(f"tuned model: ${monthly_tuned:,.0f} per month + ${tuning_cost} once")
months = tuning_cost / (monthly_few_shot - monthly_tuned)
print(f"tuning pays for itself after {months:.1f} months, if quality is at least equal")
Tuning saves $155 a month here, paying for itself in under four months, provided its quality is at least equal, which the "did it actually improve" lesson measures. Beware that some providers price tuned models above the base ones, so plug in the real price. And at a thousand requests a month instead of 200 thousand, tuning would practically never pay for itself.
4The real cost
Training hours are the smallest part of the bill. Before them comes building and cleaning the data, and after them evaluation and hosting.
Look at the picture: the last part repeats. Every new base model may beat your tuned one with no tuning at all, so you redo everything.
So keep the data, the training script and the evaluation ready, to retune on the new model in a day rather than a month. And always compare against "the new model, untuned".
How It Works, Step by Step: how to decide whether to fine-tune
Name the problem: a fact the model does not know, or a format and style it does not follow consistently.
If it is changing or private information, the fix is retrieval, not tuning.
If it is format or a narrow task, first try a better template, then examples in the template, and measure each rung.
Work out the break-even point with your real volume and price, and write the sentence: I will tune because this was not solved by that, and I expect the metric to rise from this to that.
Going deeper
And remember a hidden cost: each new base model released every few months may beat your tuned model with no tuning at all. If you tune, keep the data, training script and evaluation ready, so you can retune on the new model in a day rather than a month, and always compare against "the new model, untuned".
Worked example
The first question: what exactly is the problem? If the model does not know a fact (your company's policy, today's prices), tuning will not reliably fix it, and retrieval will. If it knows what to do but does not consistently do it in the format or style you want, that is where tuning may help. Then the second question: is it worth its price?
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Fine-tuning teaches the model your company's information. | It memorises some, mixes up some, cites no source, and needs redoing with every change. Changing information belongs in retrieval. |
| The cost of tuning is the training hours. | The larger part is usually building and cleaning data, evaluation and hosting, then redoing it all with every new base model. |
| A tuned model stays the best. | A new base model may beat it with no tuning at all. Always compare against "the new model, untuned". |
Try it yourself
Which problem does fine-tuning solve better than retrieval?
A fixed format and a narrow repeated task are tuning's domain. The rest is changing knowledge that belongs in retrieval.
You have only a thousand requests a month. What is likely about tuning?
The saving per request is small, and multiplied by a thousand it does not cover the cost of data, training and evaluation.
Quick review cards
Check Your Understanding
The model answers questions about your company's policies with outdated information. Do you tune it on the new policies?
No. This is a changing-knowledge problem, and retrieval solves it: update the document and the answer updates immediately, with a citation you can check. Tuning would partly memorise the current policies, which would then go stale again.
In the break-even example, how many months would tuning need to pay for itself at a thousand requests a month instead of 200 thousand?
The monthly saving becomes about $0.78 instead of $155, so it would need over 770 months, meaning it practically never pays for itself.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.