HLD · Traffic

Load Balancing

Distribute incoming traffic across multiple servers for scale and availability.

1. Why load balancing exists

One server has limited CPU, memory, and concurrent connections. A load balancer (LB) sits in front of a server pool and sends each request to a healthy backend server.

Example: During a sale, an e-commerce site receives 50,000 requests per second (RPS). If one VM handles 5,000 RPS, ten or more VMs behind an LB can share the load. If one VM fails, the LB stops sending requests to it.

2. Layer 4 vs Layer 7

Interview tip: Use L7 when routing depends on HTTP data, such as sending /api to API servers and /images to an image service. Use L4 when routing only needs an IP address and port, such as a raw TCP service.

3. Load balancing algorithms

• Round robin: Rotate across servers in order. Simple; ignores server load.
• Weighted round robin: Bigger machines get more requests.
• Least connections: Send to server with fewest active connections. Good for long-lived requests.
• IP hash: Same client IP → same server. Sticky sessions without cookies.
• Consistent hash: Minimal reshuffle when nodes added/removed (see Consistent Hashing note).

4. Health checks

LBs probe backends (GET /health) every few seconds. Unhealthy nodes are removed from rotation automatically.

• Active checks: LB sends probe; knows real availability.
• Passive checks: LB marks node bad after N consecutive 5xx/timeouts from real traffic.

Use a cheap health endpoint. A liveness check asks whether the process is alive. A readiness check asks whether it is ready to receive traffic.

Concrete example: A new Java server may be alive while it warms its caches. Its liveness check passes, but its readiness check fails until warm-up ends, so the LB does not send user requests too early.

5. Session stickiness (affinity)

Stateful apps may require the same user on the same server (in-memory cart). Options:

• LB cookie-based sticky sessions
• IP hash (brittle behind NAT)
• Better: Store session in Redis — any server can serve any user (stateless app tier)

6. Scaling pattern (interview walkthrough)

1. DNS → LB (single entry point)
2. LB → N stateless app servers (auto-scaling group)
3. Shared DB + cache behind apps
4. Scale horizontally until DB becomes bottleneck → read replicas, sharding

7. Failure modes

• LB itself is SPOF: Use managed LB (AWS ALB) or active-passive LB pair (keepalived + floating IP).
• Thundering herd: All traffic hits one node after others fail — use slow start / ramp-up on new nodes.
• Cold nodes: New instances accept traffic before JVM warmup — use readiness probes that fail until warm.