1. Which lever adds fresh, changing facts most reliably?
- A longer style prompt
- Fine-tuning
- Retrieval (RAG)
- Raising temperature
Show answer
Answer: Retrieval (RAG)
Retrieval brings current, citable facts at question time.
27 questions from the Fine-tuning vs Prompting course, each with the correct answer and an explanation. Try to answer before you open the answer.
Answer: Retrieval (RAG)
Retrieval brings current, citable facts at question time.
Answer: It is faster, cheaper and reversible, and often enough
Use the cheapest step that meets the target.
Answer: Extracting invoice fields into a strict schema with thousands of labelled examples
Narrow, repeated, measurable tasks with plentiful examples suit training.
Answer: Staying with prompting
Fixed training costs are not recovered at low volume.
Lesson: The Economics: Break-Even Volume
Answer: The model adapts to a pattern from examples in the prompt, with no weights changed
Examples in the prompt steer behaviour only for that call.
Answer: A trained model can keep improving where example matching flattens
Check your own curve: if accuracy still climbs with data, training may pay off.
Lesson: Learning Curves: When Training Overtakes Examples in the Prompt
Answer: Prompt caching and selecting only the most relevant examples per request
Cheaper prompting often removes the cost argument for fine-tuning.
Answer: The weights, to reduce loss on desired example outputs
Training adjusts weights to make the target answers more likely.
Answer: The learning rate is too high (or the data/batch is bad)
Lower the learning rate and inspect the data and logs.
Answer: Small low-rank matrices added to frozen weights
The adapter learns a compact update while the base stays frozen.
Answer: Training performance keeps improving while validation performance stalls or worsens
Hold out data and watch the gap.
Answer: Pairs of answers labelled chosen versus rejected
Comparisons teach which behaviours are favoured.
Lesson: Preference Tuning: RLHF and DPO
Answer: On human-verified test data, not only agreement with the teacher
Matching the teacher's errors is not the goal; being correct is.
Lesson: Distillation: Teaching a Small Model From a Large One
Answer: Hundreds of clean, consistent examples
Quality and consistency matter more than raw volume.
Answer: Format errors are caught cheaply before a paid training run
Cheap local checks avoid wasted runs.
Answer: Related or duplicate rows on both sides of the split, inflating the score
Split by the unit that must remain unseen.
Answer: Replaying a sample of earlier data, a low learning rate or LoRA
Keep old behaviour in the loop and in the evaluation.
Answer: Facts can be updated, cited and access-controlled without retraining
Keep knowledge outside the weights.
Answer: Tuning can weaken the base model's safety behaviour
Check safety after every change to the weights.
Answer: The base model with your best prompt and examples on the same test set
Beat your best alternative, not a strawman.
Answer: A single headline score can hide regressions in other dimensions
Look at several dimensions together.
Lesson: Choosing Metrics for the Task
Answer: Small adapters can be swapped over one shared base model
Small per-task deltas over a shared base save memory.
Answer: Cheaper prompting was still over budget and the learning curve was still rising with data
Evidence from the ladder, not enthusiasm, justified the step.
Answer: You control the weights, data location and serving
Control comes with operational responsibility.
Answer: So the model can be reproduced, compared and rolled back
Reproducibility is what makes re-training safe.
Lesson: Versioning, Drift and Retirement
Answer: Skipping a strong prompted baseline and an evaluation set
Evidence against a fair baseline is the core discipline.
Lesson: Common Pitfalls
Answer: Build one before training anything
Without a measure you cannot know whether tuning helped.
Lesson: A Decision Checklist