System Design · Design Question
Design Twitter Search
Search recent tweets at scale: ingest, inverted index, ranking, and fan-in query serving.
In simple words:
Stream new tweets into an inverted index, ask index shards for their best matches, and merge those answers into one ranked page.
Concrete example: Search "World Cup" on Twitter returns recent tweets containing those terms — each tweet is indexed within seconds, shard leaves return local top matches, and a root service merges them into one ranked timeline.
1. Requirements
Functional
- Full-text search over tweets (keywords, hashtags, from:user).
- Freshness: new tweets searchable within seconds.
- Rank by relevance + recency + engagement.
- High query QPS; huge write stream of tweets.
Non-functional
- Make new tweets searchable within seconds and serve high query QPS.
- Keep tweet creation available even when indexing falls behind.
Back-of-envelope
500M tweets/day ≈ 5.8k index writes/s average. At ~10KB indexed data/tweet,
that is ~5TB/day before replication; size query replicas for ~100k peak searches/s.
Scope: text search on tweets; not the whole Twitter product
Ask: only last 7–30 days early? archive later?
2. API
GET /v1/search?q=world%20cup&type=latest|top&cursor=
→ { results: [{ tweetId, user, text, ts, score }], nextCursor }
3. Data model
Source of truth: tweet store (already exists in Twitter design)
Search index (inverted):
term → postings list of (tweet_id, tf, ts, ...)
Hashtags / mentions as special terms
User filter: secondary index or posting metadata
4. High-level design
Tweet Write Path → Kafka
|
Indexer workers → Lucene/ES shards (inverted index)
|
optionally early "realtime" segment + later merge
Query Path:
Client → Search API → Query Planner
→ fan-in to relevant shards
→ merge top-K → hydrate tweet objects from cache/DB
5. Inverted index refresher
tweet1: "cats are great"
tweet2: "cats and dogs"
cats → (t1, t2)
great → (t1)
dogs → (t2)
AND query: intersect postings; OR: union
6. Freshness strategy
- Realtime index: small in-memory/recent segments for last N minutes.
- Batch index: larger compacted segments for older tweets.
- Query both and merge; promote realtime → batch via merge jobs.
7. Ranking
latest: sort by ts DESC after text match
top: BM25 / learned ranker + engagement + recency decay
Retrieve a candidate set (e.g. few thousand) then re-rank; do not score the entire corpus deeply.
8. Sharding
- Shard by time (recent vs archive) and/or by term hash / tweet_id.
- Queries may fan-in to many shards — cap and prefer recent shards.
- Hot terms (“a”, “the”) need skip lists / careful query rewriting.
9. Scaling & ops
- Index replicas for query HA.
- Cache hot queries briefly (seconds) carefully — freshness trade-off.
- Backpressure indexers on Kafka lag; never block tweet create path.
10. Edge cases
- Deleted tweets → tombstones in index; filter on hydrate.
- Spam → quality filters in ranker / index exclusion.
- Unicode / tokenization for i18n.
11. Query planning examples
q = "cats filter:media since:2024-01-01"
1) tokenize "cats"
2) constrain postings to docs with media flag
3) time bound → only recent shards
4) retrieve → re-rank → hydrate
q = "from:alice dogs"
use author index / filter early to cut postings
12. Abuse and quality
- Rate-limit search scrapers.
- Downrank spam clusters and bot-like engagement.
- Sensitive queries may need safety filters before results render.
13. Failure modes
- Indexer outage → Kafka retains tweets; freshness degrades but posting continues.
- One search shard times out → return partial results only if the API marks them partial.
- Segment merge fails → keep old immutable segments and retry the merge.
Interview talking points
- Lead with: Explain realtime indexing, shard fan-out, and top-K merge.
- Async index from tweet stream; do not block write path.
- Inverted index + realtime and batch segments for freshness.
- Query fan-in across shards; merge top-K; hydrate tweets.
- Latest vs Top ranking modes.
- Candidate retrieve then re-rank.
- Tombstones for deletes; lag monitoring on indexers.
- Time sharding helps bound “recent search.”