Comparing Prompt and Model Versions

A safe rollout recipe

A new prompt or a cheaper model should pass three gates. Offline: run the dataset experiment and compare scores with the current version, reading the examples that regressed rather than only the averages. Canary: send a small share of real traffic to the new version, tagging each trace with the variant (through a label, metadata or tag), and compare feedback, judge scores, latency and cost per variant. Full rollout: switch everything over, keep monitoring, and keep the old version ready for rollback. Look at quality, latency and cost together, because they trade off: a model that is half the price but triples the rate of negative feedback is not a saving. Make variant assignment sticky per user or session, so one conversation does not jump between prompts, which would make both its traces and the user experience confusing.

Sticky variant assignment

Hash the session ID so the same session always gets the same variant, then record the variant on the trace.

import hashlib

def variant_for(session_id: str, canary_share: float = 0.1) -> str:
    bucket = int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
    return "v5-canary" if bucket < canary_share else "v4"

for sid in ["chat-1", "chat-2", "chat-3", "chat-1"]:
    print(sid, variant_for(sid, canary_share=0.5))

Output:

chat-1 v4
chat-2 v5-canary
chat-3 v4
chat-1 v4

New dish on the restaurant menu

A restaurant first tries a new recipe in the kitchen (offline), then serves it to a few tables and watches their faces and leftovers (canary), and only then prints it on the menu. Each table keeps the same dish through the meal.

त्वरित जाँच: Why should canary variant assignment be sticky per session?

• So a single conversation does not switch prompts mid-way, keeping behaviour and comparisons consistent
• Because observability tools reject random assignment
• To reduce token cost
• So that every user receives the canary