System Design · Design Question
Design a Metrics System
Ingest, aggregate, query, and alert on time-series metrics (mini Prometheus / Datadog mental model).
In simple words:
Applications send timestamped numbers. Group them by name and labels, compress the time series, then power dashboards and alerts.
Concrete example: If http_errors{path="/checkout"} exceeds 100 per minute for 5 minutes, the alert evaluator pages on-call — the same metric also powers a Grafana graph of error rate over time.
1. Requirements
Functional
- Apps emit metrics: counters, gauges, histograms (latency).
- High ingest write rate; store efficiently with downsampling.
- Query by metric name + labels over a time range.
- Alert when threshold / anomaly rules fire.
- Retention tiers: raw short, rollups long.
Non-functional
- Handle very high write throughput with bounded query latency.
- Isolate tenants and reject unbounded label cardinality.
Back-of-envelope
10M active series sampled 4 times/minute ≈ 667k points/s.
At 16 bytes/point that is ~0.9TB/day before replication; compression and rollups are mandatory.
Cardinality warning: user_id as a label can explode series count
Ask: scrape pull vs push agents? multi-tenant?
2. API
Push:
POST /v1/write
{ "series": [{ "name": "http_requests", "labels": {"path":"/x","code":"200"},
"points": [[ts, value], ...] }] }
Query:
POST /v1/query_range
{ "query": "sum(rate(http_requests[5m])) by (path)", "start", "end", "step" }
Alerts CRUD:
POST /v1/rules { expr, threshold, for, notify }
3. Data model
metric_names + label dictionaries
time series identity = fingerprint(name + labels)
points: (series_id, ts, value) in TSDB segments / LSM / columnar blocks
rollups: 1m, 5m, 1h aggregates (sum/count/min/max for histograms sketches)
3.1 Metric types (know these)
| Type | What it tracks | Interview example |
|---|---|---|
| Counter | Only goes up (resets on restart) | http_requests_total — use rate() for req/s |
| Gauge | Up or down snapshot | queue_depth, cpu_percent |
| Histogram | Counts in latency/size buckets | http_request_duration_seconds — p99 from buckets |
Do not put high-cardinality values (user ID, request ID) in labels — each unique label set creates a new time series.
4. High-level design
Apps → Agent / SDK --push--> Ingest Gateway
|
Kafka (buffer)
|
Aggregators / writers → TSDB shards
|
Query API → fan-out to shards → merge
Alert evaluator → rules → Notification system
5. Ingest deep dive
- Batch points; compress; authenticate tenants.
- Kafka absorbs spikes; protects TSDB.
- Reject insane cardinality (too many unique label sets).
- Idempotent write optional via client batch ids (usually at-least-once OK).
6. Storage & downsampling
Hot: raw resolution 15s for 24–72h
Warm: 1m rollups for 30d
Cold: 1h rollups for 1y+
Compaction merges blocks; drop raw after retention
Histograms need careful aggregation (store buckets or use t-digest / DDSketch — mention sketches for percentiles).
7. Query path
- Parse PromQL-like expression.
- Resolve series matching label selectors.
- Fetch chunks from owning shards.
- Evaluate rate/sum/quantile; return aligned steps.
Cache recent query results carefully; prefer caching parsed series lists.
8. Alerting
Evaluator periodically runs rules against TSDB
Pending --for 5m--> Firing → notify (PagerDuty/Slack)
Resolve when expr false
Dedup and group alerts to avoid storms
9. Scaling
- Shard by series fingerprint hash.
- Separate ingest and query compute.
- Control cardinality at the gate — #1 production lesson.
- Multi-region: usually regional TSDB + global query optional.
10. Failure modes
- Kafka lag → delayed dashboards/alerts; SLO on lag.
- Hot series (one metric dominates) → isolate shard.
- Alert flapping → hysteresis /
forduration.
11. Edge cases
- Late or out-of-order points → accept within a bounded window, then compact.
- Counter resets after restart → rate calculation detects a decrease instead of showing negative traffic.
- Missing samples differ from zero; dashboards and alerts must preserve that distinction.
Interview talking points
- Lead with: Call out label cardinality before discussing storage scale.
- Pipeline: agents → ingest → Kafka → TSDB → query/alert.
- Series = name + labels; guard cardinality.
- Downsample/retain: raw short, rollups long.
- Histograms need mergeable sketches/buckets.
- Query fans out to shards and merges.
- Alert rules with pending→firing and dedup.
- At-least-once ingest; dashboards tolerate small loss better than payments.