System Design / Video Resource

Let's Handle 1 Million Requests per Second

A Cododev production-scale experiment that turns throughput, bottlenecks, and infrastructure trade-offs into something you can reason about in an interview.

Open the original video on YouTube → · Cododev

Short summary: The video starts with a simple HTTP service and measures what one machine can do. It then changes the payload, concurrency, runtime, database path, cache path, load-generator topology, and server fleet until the team can reason about one million requests per second. The useful lesson is the method: estimate, measure, find the current bottleneck, change one variable, and measure again.

Why this belongs in System Design prep

Throughput is not enoughEvery result depends on payload size, latency, errors, concurrency, network bandwidth, and the cost of the machines producing the load.

Bottlenecks moveA database can be the wrong place to spend the budget at extreme write volume; CPU, memory, network, serialization, or the tester may become the limit first.

Architecture follows evidenceUse monitoring to decide whether to add threads, cluster processes, cache data, batch writes, change the hot path, or add machines.

Operations are part of designCost guardrails, resource limits, connection counts, failure handling, and result aggregation matter as much as the request handler.

Watch with these checkpoints

08:35
CPU and core utilizationBuild the mental model for per-core utilization, aggregate CPU reporting, threads, memory, disk, and network.
21:37
Benchmark the baselineUse a load generator such as autocannon and record throughput, status codes, total requests, and bytes moved.
32:01
Cluster the applicationRun multiple Node.js processes and see why more workers help only until another resource becomes the limit.
57:39
Distribute the loadSeparate the load generator from the server, introduce balancing, and watch the network card become a constraint.
80:00
Find the database limitPostgreSQL writes expose CPU pressure, connection limits, disk work, and why blindly adding database capacity is expensive.
90:00
Trade persistence for speedRedis can absorb much higher write throughput, but memory is finite and durable batch processing is still required.
120:00
Optimise the hot pathNative C++ and a faster JSON parser remove application overhead, while logging and compression are treated as measurable costs.
130:00
Scale out carefullyFleet size, CPU quotas, placement, orchestration, and result collection become part of the design problem.

Transcript-derived summary

This summary was prepared from the video's English auto-generated captions. It is an original study guide, not a verbatim transcript.

Start with the stakes: one million requests per second is an extreme environment where a small inefficiency can become a large cost. The target is a benchmark, not a universal promise for every service.
Know the machine: CPU cores, threads, memory, disk, and network are different resources. Measure them separately and be explicit about whether CPU utilization is normalized to 100% or summed across cores.
Build a trustworthy benchmark: use a representative endpoint and payload, enough connections and workers, a dedicated load generator when needed, and record throughput together with errors, latency, bytes, and resource saturation.
Expect the limit to move: a tiny response can be CPU-bound, while a 32 KB response can become network-bound. A multi-process cluster can increase throughput until the host, network card, or load tester saturates.
Be deliberate with databases: high-volume writes can make database CPU, disk, and connections dominate. An in-memory path such as Redis can increase immediate throughput, but it creates memory pressure and still needs a durable, batched write path.
Optimise only the hot path: faster serialization, fewer unnecessary logs, disabled compression where appropriate, and a lower-overhead runtime can matter, but only after measurement shows application overhead is the current limit.
Scale the whole experiment: the load generator, server fleet, placement, cloud quotas, monitoring, cost, and result aggregation all affect whether the number is meaningful.

Interview takeaway

State the target workload and payload before quoting a requests-per-second number.
Estimate bandwidth, CPU, memory, connections, and storage before choosing components.
Say what metric proves the bottleneck, then name the next change you would test.
Always include cost, failure recovery, durable writes, and observability in the design.