Design a News Feed

Push, pull and hybrid

In a push (fan-out on write) model, a new post is inserted into every follower's precomputed feed, so reads are fast but celebrities cause millions of writes. In a pull model, feeds are assembled at read time from followed accounts, so writes are cheap but reads are expensive. Real systems use a hybrid: push for ordinary accounts, pull for celebrities at read time, with feeds cached and ranked, and media served from a CDN.

The celebrity problem in numbers, run

I ran this with Python 3.12.3 using only the standard library; inputs are fixed or seeded, so the output is reproducible. With invented but plausible numbers, one celebrity posting four times a day creates 20 million feed inserts under the push model, while ordinary users create hundreds.

# News feed: fan-out on write vs on read, and the celebrity problem
followers = {"alice": 200, "bob": 350, "celebrity": 5_000_000}
posts_per_day = {"alice": 3, "bob": 1, "celebrity": 4}

for user, n in followers.items():
    writes = n * posts_per_day[user]
    print(f"{user:10} push model: {writes:>12,} feed inserts/day")

following = 300            # accounts a typical reader follows
reads_per_day = 20         # feed refreshes per reader per day
print(f"pull model: {following * reads_per_day:,} timeline lookups per reader per day")
print("hybrid: push for normal accounts, pull celebrity posts at read time")

Output:

alice      push model:          600 feed inserts/day
bob        push model:          350 feed inserts/day
celebrity  push model:   20,000,000 feed inserts/day
pull model: 6,000 timeline lookups per reader per day
hybrid: push for normal accounts, pull celebrity posts at read time

Separate post storage from feed storage

Feeds store post IDs only; post content is fetched and cached separately.

त्वरित जाँच: Why do large feeds use a hybrid fan-out?

• Pushing celebrity posts to millions of feeds is too expensive
• Pull is always faster
• Push cannot handle small accounts
• Feeds cannot be cached