Detecting Flaky Tests

Rerun before you fix

A flaky test passes and fails without code changes, because of timing, ordering, randomness, shared state or external services. An automated fixer that "fixes" code for a flaky test introduces needless changes, or adds sleeps and retries that hide real problems. Before attempting a fix, rerun the failing test several times: consistent failure signals a real bug; intermittent failure means quarantine and investigate the test itself.

Classifying a test by reruns, run

I ran this with Python 3 (standard library) and, where it uses git, real git in a throwaway temporary repository. Candidate patches are written by hand to stand in for model output. Rerunning a test 20 times: one passes 15 of 20 times and is classified as flaky (quarantine it, do not change product code); the other fails all 20 times, a consistent signal of a real bug. Randomness simulates the flakiness.

import random
def flaky_test(rng):            # depends on timing / ordering in real life; here a 20% random failure
    return rng.random() > 0.2
def stable_failing_test(rng):
    return False
def classify(test, runs=20, seed=1):
    rng = random.Random(seed)
    results = [test(rng) for _ in range(runs)]
    passed = sum(results)
    if passed == runs: return f"pass ({passed}/{runs})"
    if passed == 0: return f"consistent failure ({passed}/{runs}) -> real bug signal"
    return f"FLAKY ({passed}/{runs}) -> quarantine, do not auto-fix code for it"
print("flaky_test         :", classify(flaky_test))
print("stable_failing_test:", classify(stable_failing_test))

Output:

flaky_test         : FLAKY (15/20) -> quarantine, do not auto-fix code for it
stable_failing_test: consistent failure (0/20) -> real bug signal

Track flake rates over time

Record pass/fail history per test in CI; tests with intermittent failures should be fixed or quarantined by owners.

Quick check: A test fails 5 times out of 20 with no code change. What should an auto-fixer do?

• Merge a random patch
• Rewrite the function under test
• Delete the test silently
• Treat it as flaky and not change product code for it