Canary Releases and A/B Tests

Small slice, statistical check, expand or roll back

A canary release sends a small share of traffic (for example 1 to 5%) to the new model, compares key metrics with the current version, and expands gradually if they hold. Decide the metrics, thresholds and observation time in advance, and compare with statistics that account for small samples. An A/B test is a longer, controlled comparison of business outcomes. Automate rollback when guardrail metrics (errors, latency, complaint rates) breach limits.

Is the canary's error rate really higher? run

I ran this with Python 3, MLflow 3.16.1, scikit-learn 1.9.1, scipy 1.18.1 and numpy 2.5.3, using bundled or seeded synthetic data and a local SQLite tracking store. The current version has 240 errors in 20,000 requests (1.2%); the canary has 21 in 1,000 (2.1%). A two-proportion z-test gives z = 2.51 and a one-sided p-value of 0.006, so the rule says roll back. The counts are example numbers.

from math import sqrt
from scipy.stats import norm
control = (240, 20000)     # (errors, requests) on the current version
canary = (21, 1000)        # 5% of traffic on the new version
def two_prop_z(a, b):
    p1, n1 = a[0] / a[1], a[1]; p2, n2 = b[0] / b[1], b[1]
    p = (a[0] + b[0]) / (n1 + n2)
    z = (p2 - p1) / sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    return p1, p2, z, 1 - norm.cdf(z)
p1, p2, z, pval = two_prop_z(control, canary)
print(f"error rate control {p1:.3%} | canary {p2:.3%} | z {z:.2f} | one-sided p {pval:.3f}")
print("roll back" if pval < 0.05 else "keep watching / continue rollout")

Output:

error rate control 1.200% | canary 2.100% | z 2.51 | one-sided p 0.006
roll back

Fix the observation window in advance

Decide how long and how many requests a canary needs before peeking; stopping early on noise leads to wrong decisions.

Quick check: What is a canary release?

• Running models only offline
• Deploying to all users at once
• Sending a small share of traffic to a new version and expanding only if metrics hold
• Deleting the old version first