Fine-Tuning Is a Poor Way to Add Facts

Style transfers, facts leak

Training on documents tends to teach the model how such text sounds more reliably than what is true in it. Facts learned from a few passes are stored unevenly, can be paraphrased wrongly with confidence, cannot be cited, and cannot be updated or deleted without retraining. Retrieval keeps facts outside the weights where they can be refreshed, sourced and access-controlled. A good division: tune for behaviour (format, tone, task skill) and retrieve for knowledge. If the model must also reason over your private data, do both and test them separately.

A trained chef versus a cookbook

Training teaches the chef technique; facts such as today's menu prices belong in a book that can be reopened and updated.

Test fact questions both ways

Compare a tuned model with and without retrieval on fact questions; retrieval usually wins on accuracy and on freshness.

Quick check: Why prefer retrieval for company facts that change?

• Facts can be updated, cited and access-controlled without retraining
• Retrieval needs no data
• Weights store facts perfectly
• Tuned models cannot answer questions