# Safe Changes: Deployments, Flags and Rollbacks — Scalability, Availability & Reliability

Source: https://www.skillbyai.com/en/scalability/r-change

> Reduce change-related outages with progressive delivery and fast rollback.

## Most outages start with a change

Industry incident reviews repeatedly find that a large share of outages are triggered by **changes**: deployments, configuration pushes, schema migrations, certificate or DNS updates. Reliability therefore depends heavily on how you change systems. **Progressive delivery** limits the blast radius: **canary releases** send a small percentage of traffic to the new version and compare its error rate and latency to the old one before widening; **blue-green deployments** keep the old environment ready for instant switch-back; **feature flags** separate deploying code from enabling behaviour and allow instant kill switches. Roll out **zone by zone or region by region**, never everywhere at once. Make **rollback** fast and rehearsed. Treat configuration as code with review and staged rollout. Make database migrations **backward compatible** (expand, migrate, contract) so old and new versions can run side by side.

## An automated canary analysis rule

Promote only if the canary is not worse than the baseline.

```yaml
canary:
  steps:
    - setWeight: 5          # 5% of traffic to the new version
    - pause: { duration: 10m }
    - analysis:
        metrics:
          - name: error-rate
            condition: canary_error_rate <= baseline_error_rate * 1.1
          - name: p99-latency
            condition: canary_p99_ms <= baseline_p99_ms * 1.2
        onFailure: rollback
    - setWeight: 25
    - pause: { duration: 10m }
    - setWeight: 100
# rollout order: one zone -> one region -> all regions
```

## Expand, then contract

Renaming a column in one deploy breaks the old version still running during rollout. Add the new column, write to both, migrate reads, and only then remove the old column in a later release.

**Quiz:** What is the main reliability benefit of a canary release?

- [x] A bad change affects only a small share of traffic and is detected before full rollout
- [ ] It makes builds faster
- [ ] It removes the need for testing
- [ ] It doubles capacity

*Answer:* A bad change affects only a small share of traffic and is detected before full rollout. Canaries limit blast radius and provide a comparison against the stable version.