Load balancing

A load balancer (LB) sits between clients and a pool of servers. It decides which server handles each request, stops sending traffic to unhealthy servers, and lets you add or remove servers without clients noticing.

A redundant load balancer pair in front of a pool with health checks

What a load balancer does

Distributes load using an algorithm (below).
Health checks: pings each server (say GET /health every 5 s) and removes it after a few failures.
TLS termination: decrypts HTTPS once at the edge so servers don’t have to.
Connection management: keeps long-lived connections to backends; can buffer slow clients.
Zero-downtime deploys: drain a server, update it, add it back.

Layer 4 vs layer 7

Layer 4 (transport)

Routes by IP and port; never reads the request body
Very fast, very cheap, handles any TCP/UDP protocol
Can’t route by URL path or cookie
Examples: AWS NLB, LVS

Layer 7 (application)

Reads HTTP: routes /api/* and /images/* to different pools
Can rewrite headers, retry, cache, compress, rate-limit
More CPU per request; must terminate TLS to see content
Examples: Nginx, Envoy, HAProxy, AWS ALB

Algorithms

Algorithm	How it picks	Good for
Round robin	Next server in turn	Identical servers, similar requests
Weighted round robin	Bigger servers get more turns	Mixed server sizes
Least connections	Server with fewest open connections	Long or uneven requests (WebSockets, uploads)
Least response time	Fastest recent server	Latency-sensitive pools
IP hash	Hash of client IP	Simple stickiness
Consistent hashing	Hash of a key onto a ring	Caches and shards: the same key goes to the same node (details)

Keeping the balancer available

A single load balancer is a single point of failure. Common fixes:

Active–passive pair sharing a virtual IP; the standby takes over when heartbeats stop.
Active–active behind DNS or anycast, both serving traffic.
Managed cloud balancers, which are redundant by design.

Global load balancing

For multiple regions, GeoDNS returns the nearest region’s address, and anycast advertises one IP from many locations so the network routes users to the closest one. Global balancing also gives you regional failover: if a region goes down, DNS stops pointing users at it.

Pros

Horizontal scaling and fault tolerance in one component
Hides topology; servers can change freely
Central place for TLS, routing and rate limits

Cons

Another hop, so a little added latency
Must itself be redundant
L7 balancers need care: timeouts, retries, buffering