System Design · Design Question
Design a Task Scheduler
Cron at scale: schedule one-shot and recurring jobs, dispatch reliably, and avoid double execution when workers race.
In simple words: A distributed cron system stores when jobs should run, claims due work safely across many schedulers, and hands execution to workers that must be idempotent.
Concrete example: A shop schedules "generate daily sales report at 3 AM IST." One scheduler claims that run row, enqueues work, and a worker emails the PDF. If the worker crashes, the lease expires and another worker retries without sending two emails if the handler checks run_id.
1. Requirements
Functional
- Schedule job at time T or on cron expression (e.g. every day 03:00).
- Execute handler / webhook / queue message when due.
- At-least-once delivery to workers; idempotent job handlers.
- Retry with backoff; DLQ after N failures.
- Horizontal scheduler and worker fleets; no single cron box.
Non-functional
- Do not lose due work; tolerate duplicate delivery through idempotency.
- Scale scheduler scanning and worker execution independently.
Back-of-envelope
100M active jobs with 10M runs/hour means ~2.8k dispatches/s average.
A midnight spike can be 10× higher, so partition due-time buckets and add jitter.
Ask: precision (1s vs 1min)? calendar TZ? priority queues?
max job runtime? fan-out fan-in workflows?
2. API
POST /v1/jobs
{
"name": "nightly-report",
"schedule": "0 3 * * *", // cron OR "runAt": iso8601
"timezone": "Asia/Kolkata",
"payload": { "type": "report", "id": "R1" },
"retry": { "max": 5, "backoff": "exp" }
}
→ { "jobId": "j9" }
GET /v1/jobs/{id}
DELETE /v1/jobs/{id} // cancel future runs
POST /v1/jobs/{id}/pause
3. Data model
jobs(job_id, cron_or_ts, tz, payload, state, next_run_at, version)
job_runs(run_id, job_id, scheduled_for, status, locked_by, locked_until, attempt)
UNIQUE(job_id, scheduled_for) -- prevents duplicate run rows
indexes: next_run_at WHERE state=active
4. High-level design
API → Jobs DB
^
Scheduler workers (many)
loop:
claim due jobs (SELECT ... FOR UPDATE SKIP LOCKED / lease)
enqueue execution message
compute next_run_at for cron
Execution queue → Executors
run handler / HTTP callback
ACK success → mark run succeeded
fail → retry / DLQ
5. Claiming due work (avoid double fire)
BEGIN
SELECT * FROM job_runs
WHERE status='ready' AND scheduled_for <= now()
ORDER BY scheduled_for
FOR UPDATE SKIP LOCKED
LIMIT 100
UPDATE ... SET status='leased', locked_by=worker, locked_until=now()+lease
COMMIT
-- Only one worker gets each row
Leases expire so a crashed worker does not hold a job forever. Handlers must be idempotent because lease expiry can cause a second attempt.
6. Time and cron
- Store
next_run_atin UTC; interpret cron in user TZ when computing next. - Missed runs while paused: skip vs catch-up policy — product decision.
- Do not rely on one OS crontab for distributed systems.
7. Exactly-once illusion
True exactly-once across distributed executors is hard. Practical
approach: at-least-once dispatch + idempotent business effect (store
run_id processed set).
8. Scaling
- Partition jobs by
job_idhash across scheduler shards. - Separate hot queues for high-priority / low-latency one-shots.
- Watch DB contention on claim queries; SKIP LOCKED is key.
- For massive fan-out, use Kafka time-partitioned delay or specialized delay queues.
9. Failure modes
- Poison payload → DLQ after N; alert.
- Clock skew → prefer DB
now()as authority for due checks. - Thundering herd at midnight → jitter next_run or shard schedules.
10. Edge cases
- DST skips or repeats a local time → define skip/once/twice policy per job.
- Job is cancelled while leased → executor checks state before starting when cancellation must be strict.
- Run exceeds lease → heartbeat extends it; idempotency still protects against a race.
Interview talking points
- Lead with: Promise at-least-once dispatch and require idempotent handlers.
- Persist jobs + next_run_at; many schedulers claim with leases.
- SKIP LOCKED / lease expiry prevents stuck and reduces doubles.
- Execute via queue workers; retry + DLQ.
- Idempotent handlers; unique (job_id, scheduled_for).
- Cron computed in TZ, stored next fire in UTC.
- At-least-once + idempotency beats fake exactly-once.
- Jitter midnight herds; shard claim load.