Safe Changes: Deployments, Flags and Rollbacks

Most outages start with a change

Industry incident reviews repeatedly find that a large share of outages are triggered by changes: deployments, configuration pushes, schema migrations, certificate or DNS updates. Reliability therefore depends heavily on how you change systems. Progressive delivery limits the blast radius: canary releases send a small percentage of traffic to the new version and compare its error rate and latency to the old one before widening; blue-green deployments keep the old environment ready for instant switch-back; feature flags separate deploying code from enabling behaviour and allow instant kill switches. Roll out zone by zone or region by region, never everywhere at once. Make rollback fast and rehearsed. Treat configuration as code with review and staged rollout. Make database migrations backward compatible (expand, migrate, contract) so old and new versions can run side by side.

An automated canary analysis rule

Promote only if the canary is not worse than the baseline.

canary:
  steps:
    - setWeight: 5          # 5% of traffic to the new version
    - pause: { duration: 10m }
    - analysis:
        metrics:
          - name: error-rate
            condition: canary_error_rate <= baseline_error_rate * 1.1
          - name: p99-latency
            condition: canary_p99_ms <= baseline_p99_ms * 1.2
        onFailure: rollback
    - setWeight: 25
    - pause: { duration: 10m }
    - setWeight: 100
# rollout order: one zone -> one region -> all regions

Expand, then contract

Renaming a column in one deploy breaks the old version still running during rollout. Add the new column, write to both, migrate reads, and only then remove the old column in a later release.

Quick check: What is the main reliability benefit of a canary release?

• A bad change affects only a small share of traffic and is detected before full rollout
• It makes builds faster
• It removes the need for testing
• It doubles capacity