HLD · Full design

News Feed / Timeline — End-to-End Design

Classic interview problem: design Twitter/Instagram home feed from scratch.

In simple words: A news feed shows recent posts from people you follow. The hard part is balancing fast reads with millions of writes when someone with many followers posts. Example: when a normal user posts, the system pushes that post into each follower's prebuilt feed cache; celebrity posts are merged at read time instead.

1. Requirements clarification

Functional

Users post tweets (text, optional media)
Users follow other users
Home timeline: recent posts from people I follow, reverse chronological
User profile shows their posts

Non-functional (sample interview assumptions)

Assume 300M daily active users and 500M posts/day (about 5,800 writes/second on average, with higher peaks)
Read:write ≈ 100:1 → heavy read optimization
Latency: feed load < 200ms p99
Eventual consistency OK (few seconds delay for new post in feed)

2. High-level architecture

Clients → API Gateway / LB
Post Service — create post, store metadata
Social Graph Service — follow/unfollow relationships
Feed Service — assemble timeline
Media Service — upload to object storage (S3), CDN serve
Notification Service — async fan-out on new post (optional)

3. Data model (core tables)

users(user_id, name, ...) posts(post_id, user_id, content, created_at, ...) follows(follower_id, followee_id, created_at) # fan-out table (see below) feed(user_id, post_id, rank/score, created_at)

4. Fan-out: push vs pull vs hybrid

Pull (read-time merge)

On feed request: get followee list → fetch recent posts from each → merge sort.

Pro: Simple writes. Con: Slow for users following thousands of accounts because the feed must merge many timelines at read time.

Push (write-time fan-out)

On new post: write post_id into every follower's precomputed feed cache (Redis/Cassandra).

Pro: Fast reads. Con: Slow write for celebrity with 50M followers (fan-out storm).

Hybrid (production approach)

Accounts below a chosen follower threshold: Push to followers' feed caches when they post
Very high-follower accounts: Skip the large push and merge their posts when followers read

Concrete example: A user follows 200 regular accounts and one account with 20 million followers. Regular posts are already in the user's feed cache; when the user opens the app, the Feed service also fetches recent posts from the high-follower account and merges both lists.

5. Feed generation flow (push path)

User posts → Post Service writes to posts DB
Publish PostCreated event to Kafka
Fan-out workers consume event, fetch follower list (cached)
Batch insert post_id into each follower's feed store (Redis sorted set by timestamp)
Follower opens app → Feed Service reads top N from Redis → hydrate post details (batch get from posts DB)

6. Storage choices

Data	Store	Why
Posts metadata	SQL or Cassandra (sharded by post_id)	Durability, range queries for profile
Social graph	Graph DB or adjacency lists in Cassandra	Fast follower/followee lookups
Precomputed feed	Redis sorted sets	Fast ordered reads; keep a bounded number of recent items
Media	S3 + CDN	Cheap blob storage, edge delivery

7. Ranking extension (pro level)

Beyond chronological: score = f(recency, engagement, affinity). Offline ML pipeline computes scores; feed sorted set uses score not just timestamp.

8. Caching & pagination

Cursor-based pagination: ?cursor=post_id:timestamp&limit=20. Cache first page of hot users in CDN edge (usually no — too personalized).

9. Failure & scale tactics

Fan-out workers scale with Kafka consumer groups
Rate limit posts per user (anti-spam)
Celebrity list maintained in config — hybrid routing
Degrade: show cached stale feed if feed service slow

10. Back-of-envelope

Under the sample assumptions, 500M posts/day × 200 followers per post = 100B feed writes/day. This estimate shows why the design should avoid blindly pushing every post to every follower. Options include a hybrid feed, fan-out only to active users, filtering before fan-out, and asynchronous batches.

Quick revision

Clarify: DAU, read:write ratio, latency, consistency.
Push: Fast read, bad for celebrities.
Pull: Simple write, slow read for heavy followees.
Hybrid: Push normal users; pull/merge celebrities at read.
Thresholds: Choose them from measured follower counts, fan-out cost, and latency.
Stack: Kafka fan-out, Redis feed cache, SQL/Cassandra posts, S3+CDN media.
Pagination: Cursor-based, not offset.