Load Balancing
A load balancer spreads requests across many backend instances, turning "one server" into "one address."
Load balancing is the practice of distributing incoming requests across multiple backend instances instead of sending them all to one machine, so that no single server becomes a bottleneck or a single point of failure. The load balancer itself becomes the address clients actually talk to, and it decides — per request or per connection — which backend handles it, using a strategy like round robin, least-connections, or consistent hashing.
The strategy choice matters more than it looks. Round robin is simple but ignores that requests aren't uniform cost, so a balancer that tracks active connections per backend (least-connections) handles uneven workloads better. Consistent hashing routes the same client, or the same cache key, to the same backend repeatedly — valuable when backends hold local state or a cache, since it turns "any backend can answer" into "this backend usually can," reducing cache misses without full stickiness.
Health checks are what make a load balancer more than a simple router: it polls each backend and stops routing to one that's failing, which is the mechanism that lets a fleet tolerate individual instance failures without the caller ever noticing. This pairs naturally with a Circuit Breaker on the client side — the load balancer removes a dead backend from rotation, while a circuit breaker stops a client from hammering a service that's failing entirely.
Layer 4 balancers work at the TCP/UDP level and just forward packets, cheap and protocol-agnostic; layer 7 balancers understand HTTP and can route on path, header, or cookie, which is what enables things like routing /api/ to one fleet and static assets to another, or an API Gateway doing the same job with more policy attached. A Content Delivery Network is, in a sense, load balancing generalized across geography instead of just across a server rack.
See also4
Anycast
Anycast announces the same IP address from many locations and lets normal routing send each client to the nearest one.
Networks & Distribution8 connections
Consistent Hashing
A hashing scheme where adding or removing a node remaps only a small fraction of keys, not nearly all of them.
Computation & Algorithms8 connections
Rate Limiting
Rate limiting caps how often a client can act, protecting shared capacity by rejecting or delaying the excess.
Networks & Distribution16 connections
Session Management
How a website remembers that a visitor has signed in across otherwise unrelated requests, usually with a random identifier in a cookie, and the attacks that target that identifier.
Web Platform11 connections
Linked from14
- AnycastNetworks & Distribution
Anycast announces the same IP address from many locations and lets normal routing send each client to the nearest one.
- API GatewayNetworks & Distribution
An API gateway is the single front door to a set of backend services, handling cross-cutting policy once instead of per-service.
- Blue-Green DeploymentVersion Control & Delivery
Running two identical production environments and switching traffic between them instead of upgrading one in place.
- Circuit BreakerNetworks & Distribution
A circuit breaker stops calling a failing dependency, converting slow repeated failures into a fast, cheap one.
- Content Delivery NetworkNetworks & Distribution
A CDN answers requests from servers near the requester, trading a single origin for many cached edges.
- Failure DetectionNetworks & Distribution
How the nodes of a distributed system decide that a peer has died, using heartbeats and timeouts, when a crashed node and a slow one look exactly the same from outside.
- KubernetesSystems & Tooling
A system that runs containers across a cluster of machines by continually reconciling what is running with a declared desired state, powerful and notoriously complex.
- Long PollingNetworks & Distribution
The ways a client learns that something changed on a server, from asking repeatedly, to holding a request open until there's news, to the push channels that replaced both.
- Mixture of ExpertsAgents & Language Models
A model architecture that holds many parameter blocks but runs only a few per token, decoupling total size from per-token compute.
- Rate LimitingNetworks & Distribution
Rate limiting caps how often a client can act, protecting shared capacity by rejecting or delaying the excess.
- Reverse ProxyNetworks & Distribution
A server that sits in front of other servers and receives clients' requests on their behalf, handling encryption, routing, caching, and protection in one place.
- Session ManagementWeb Platform
How a website remembers that a visitor has signed in across otherwise unrelated requests, usually with a random identifier in a cookie, and the attacks that target that identifier.
- Tail LatencyNetworks & Distribution
The slowest few percent of response times, which averages hide and which dominate the experience of large systems, because one request often waits on many others.
- Zero-Downtime DeploymentVersion Control & Delivery
Deploying a new version while the service keeps serving every request, with no maintenance window.