Kill Switches and Feature Flags for Resilience

Turning features off under pressure

During an incident, the fastest mitigation is often to turn something off: an expensive recommendation query, a new feature that misbehaves, a non-essential integration, or high-resolution images. Kill switches are feature flags designed for this: a runtime setting, changeable in seconds without a deploy, that disables a feature and activates its fallback. Prepare them in advance for every expensive or risky feature, document them in runbooks, and test them regularly so they work when needed. Flags can also implement brownouts: deliberately degrading optional features under high load (for example, when CPU or latency crosses a threshold) to protect critical paths. Make flag evaluation itself resilient: cache flag values locally with safe defaults, so an unavailable flag service does not become a new single point of failure.

A kill switch with a safe default

If the flag service is unreachable, the cached or default value is used.

SAFE_DEFAULTS = {
    "recommendations.enabled": True,
    "search.typo_tolerance": True,
    "images.high_res": True,
}

def flag(name):
    try:
        return flags_client.get(name, timeout=0.05)     # fast, locally cached by the SDK
    except Exception:
        return SAFE_DEFAULTS[name]                     # flag service down: known default

def product_page(product_id):
    page = render_core(product_id)
    if flag("recommendations.enabled"):
        page.recommendations = get_recommendations(product_id)
    return page

# incident runbook: set recommendations.enabled=false to cut DB load by ~30%

Write the kill switch into the runbook

A kill switch nobody remembers during an incident is useless. List each switch in the service runbook with its effect, and practise using it in game days.

Quick check: What is a brownout in this context?

• Deliberately degrading optional features under high load to protect critical paths
• A full power failure in a datacentre
• A database backup
• A type of rate limiter header