SkillByAIOpen interactive version →

Lesson 9 / 25

Race Conditions and Stale Sets

Recognise cache race conditions and fix them with ordering, leases and short TTLs.

When the cache ends up older than the database

Consider cache-aside with two requests. Reader A misses and reads the old value from the database. Writer B updates the database and deletes the cache key. Reader A then sets the old value into the cache. The cache now holds stale data until the TTL expires. This stale set race is rare but real at high traffic. Mitigations: keep TTLs bounded so damage is limited; delayed double delete (delete, then delete again shortly after the write, to remove values set by slow readers); leases, described in Facebook's "Scaling Memcache at Facebook" paper, where a miss hands the reader a token and a set is rejected if the key was invalidated after the token was issued; or version checks, storing the row version with the cached value and refusing to overwrite a newer version. For data that must be strictly current, such as stock during checkout or balances, read the source of truth instead of the cache.

The stale-set race, step by step

Reading top to bottom: the final cache value is older than the database.

time  reader A                         writer B                       cache        database
----  -------------------------------  -----------------------------  -----------  --------
t1    GET product:42 -> miss                                          (empty)      price=100
t2    SELECT price -> 100                                             (empty)      price=100
t3                                     UPDATE price=120               (empty)      price=120
t4                                     DEL product:42                 (empty)      price=120
t5    SET product:42 price=100                                        price=100    price=120
      -> stale until the TTL expires or the next invalidation

Use the source of truth for decisions

Display from the cache, decide from the database. Showing a price that is a few seconds old is fine; charging it is not. Re-read critical values inside the transaction that acts on them.

Quick check: In the stale-set race, what causes the cache to hold an old value?

  • The TTL is too long
  • Redis loses data
  • A slow reader writes the value it read before the update, after the writer has deleted the key
  • The database rolls back
Answer

A slow reader writes the value it read before the update, after the writer has deleted the key — The reader's set lands after the invalidation, reinstating an outdated value.