Versioning Prompts and Context Pipelines

Every change is a release

Small changes to wording, example selection, chunk count or ordering can shift quality in surprising ways. Store prompt templates and context settings in version control, give each version an ID, log that ID with every request, and run a regression evaluation before releasing a change. Keep the model name and version, retrieval settings and template version together, because a result is only reproducible with all of them. Roll out risky changes gradually and compare live metrics with the previous version.

What to record per request

Enough to reproduce and compare.

request_id:        r_8f12
template_version:  support-answer@v14
model:             <model id + version>
retrieval:         k=6, min_score=0.6, dedupe=jaccard@0.6
context_tokens:    5,812 (cached 3,940)
stop_reason:       end_turn
validation:        passed
latency_ms:        1,840

Diff the rendered prompt

In code review, show the rendered prompt before and after a template change, not only the template diff.

त्वरित जाँच: Why log the template version with each request?

• To hide changes from users
• To increase the context window
• Because models require it
• So results can be reproduced and compared across prompt changes