System Design · Design Question

Design a Notification System

Multi-channel notifications: push, SMS, email — templates, preferences, fan-out, retries, and provider abstraction.

In simple words: Accept one request, queue it safely, then let channel workers deliver push, email, or SMS while respecting user preferences.

Concrete example: Order shipped → API accepts the event, queues it, and a worker sends push "Order #123 on the way" plus an email with tracking — unless the user disabled email in preferences.

1. Requirements

Functional

Send notifications on events (order shipped, new follower, OTP).
Channels: mobile push (FCM/APNs), email, SMS; in-app inbox optional.
User preferences and quiet hours; unsubscribe / opt-out.
At-least-once delivery to providers; idempotent per notification id.
High throughput bursts (campaigns) without killing transactional OTPs.

Non-functional

Absorb campaign bursts without delaying OTPs or security alerts.
Retry safely, isolate provider failures, and keep an auditable status trail.

Back-of-envelope

100M notifications/day ≈ 1.2k/s average; a 10× campaign peak is ≈ 12k/s. At ~500B of metadata per channel attempt, 100M attempts add ~50GB/day before indexes.

2. API

POST /v1/notifications { "userId": "u42", "templateId": "order_shipped", "data": { "orderId": "O9", "eta": "Tomorrow" }, "channels": ["push", "email"], // optional override "idempotencyKey": "order_shipped:O9" } → { "notificationId": "n123", "status": "accepted" } GET /v1/notifications/{id} → status per channel

3. Data model

templates(template_id, channel, body, locale...) user_devices(user_id, device_token, platform) user_prefs(user_id, channel, category, enabled) notifications(id, user_id, template_id, payload, created_at) delivery_attempts(notification_id, channel, status, provider_id, ts)

4. High-level design

5. Deep dives

Priority lanes

OTP and security alerts get a dedicated high-priority queue and provider account so a marketing blast cannot starve them.

Idempotency

Unique(idempotencyKey) on accept Workers dedupe by (notification_id, channel) Providers may also support idempotent request ids

Fan-out

One event → N devices. Expand device tokens in the push worker; batch provider APIs where supported. Soft-fail invalid tokens and remove them.

Rate limits

Respect provider quotas; token-bucket per channel. Backpressure via consumer lag metrics.

6. Scaling

Partition Kafka by userId for per-user ordering if needed.
Autoscale workers on lag; separate pools per channel.
Template + prefs cached in Redis.
Multi-region: send from region near user or near provider endpoints.

7. Observability

Acceptance rate, provider latency, bounce/complaint rates.
DLQ depth alerts; replay tooling.

8. Edge cases

User disables SMS → skip channel, still try push if allowed.
Partial success: email sent, push failed → retry push only.
PII in templates → redact logs; encrypt at rest.

9. Template rendering

Keep templates versioned. Render with strict escaping to avoid injection into HTML email. Support locale fallback: en-IN → en. Preview API for product teams reduces production mistakes.

template order_shipped@v3 push: "Order {{orderId}} shipped. ETA {{eta}}" email: subject + HTML body with same variables

10. Compliance

SMS/email marketing needs opt-in evidence; transactional may differ by region.
Honor unsubscribe links quickly; store preference changes durably.
Log delivery without logging OTP codes in plaintext.

11. Failure modes

Provider outage → retry with backoff, circuit-break, and route to a backup provider where allowed.
Poison template/data → stop after bounded retries and move only that item to DLQ.
Preference service is unavailable → use a safe cached decision; never send marketing without consent.

Interview talking points

Lead with: Protect transactional traffic with a separate priority lane.
Accept API → durable queue → channel workers → providers.
Separate transactional vs bulk lanes.
Prefs, quiet hours, templates before send.
Idempotency keys prevent duplicate OTPs/blasts.
Track per-channel attempts; DLQ poison messages.
Respect provider rate limits; clean invalid push tokens.
Priority: never let marketing block OTP.