# Comparing Prompt and Model Versions — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-prompt-ab

> Roll out a new prompt or model safely and decide between versions using traces, scores and cost.

## A safe rollout recipe

A new prompt or a cheaper model should pass three gates. **Offline**: run the dataset experiment and compare scores with the current version, reading the examples that regressed rather than only the averages. **Canary**: send a small share of real traffic to the new version, tagging each trace with the variant (through a label, metadata or tag), and compare feedback, judge scores, latency and cost per variant. **Full rollout**: switch everything over, keep monitoring, and keep the old version ready for rollback. Look at quality, latency and cost together, because they trade off: a model that is half the price but triples the rate of negative feedback is not a saving. Make variant assignment **sticky per user or session**, so one conversation does not jump between prompts, which would make both its traces and the user experience confusing.

## Sticky variant assignment

Hash the session ID so the same session always gets the same variant, then record the variant on the trace.

```python
import hashlib

def variant_for(session_id: str, canary_share: float = 0.1) -> str:
    bucket = int(hashlib.sha256(session_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
    return "v5-canary" if bucket < canary_share else "v4"

for sid in ["chat-1", "chat-2", "chat-3", "chat-1"]:
    print(sid, variant_for(sid, canary_share=0.5))
```

Output:

```
chat-1 v4
chat-2 v5-canary
chat-3 v4
chat-1 v4
```

## New dish on the restaurant menu

A restaurant first tries a new recipe in the kitchen (offline), then serves it to a few tables and watches their faces and leftovers (canary), and only then prints it on the menu. Each table keeps the same dish through the meal.

**Quiz:** Why should canary variant assignment be sticky per session?

- [x] So a single conversation does not switch prompts mid-way, keeping behaviour and comparisons consistent
- [ ] Because observability tools reject random assignment
- [ ] To reduce token cost
- [ ] So that every user receives the canary

*Answer:* So a single conversation does not switch prompts mid-way, keeping behaviour and comparisons consistent. Switching variants within a conversation confuses users and mixes two versions inside one session's traces, making the comparison unreliable.