A load balancer is a traffic cop sitting in front of a group of identical servers, deciding which one handles each incoming request. It spreads load so no single machine gets overwhelmed, and it uses health checks to stop sending traffic to servers that have died. The underlying shift it represents: reliability comes from redundancy, not from building one perfect server.
Quick Answer: A load balancer distributes incoming requests across a pool of interchangeable servers and continuously checks their health, routing around any that stop responding. It's why products survive traffic spikes and server crashes — but it can't fix bugs in your code or a database that's the sole point of failure.
What Problem Is a Load Balancer Actually Solving?
A load balancer exists because a single server has a ceiling on how many requests it can handle before it slows down or falls over. Instead of buying one enormous machine, you run several identical, smaller ones and put a router in front that spreads the work. This turns a scaling problem into an addition problem — need more capacity, add another server.
Think about what happens without one. Every user's request goes to the same machine, competing for the same CPU, memory, and network connections. During normal traffic, that's fine. During a spike — a product launch, a viral post, a Black Friday sale — that one machine runs out of room, and every user, not just the newest ones, starts experiencing timeouts.
The load balancer's job is narrow but critical: accept every incoming connection, decide which backend server should handle it, and forward the request there. It does this thousands of times a second, invisibly, for every request your product receives.
- It sits between the internet and your application servers, often referred to as being "in front of" the pool.
- It has no idea what your product does — it doesn't understand checkout, search, or messaging. It only understands "here's a request, which server gets it."
- It's usually the first place traffic hits after DNS resolves, which is why load balancer failures are so visible — nothing works if the front door is broken.
This is the layer beneath what you already understand about how the web works as a PM mental model — a request still travels from browser to server, but now there's a dispatcher deciding which of many servers answers it.
How Does a Load Balancer Actually Choose Which Server Gets a Request?
Most load balancers pick a server using one of a handful of well-established algorithms, each trading off simplicity against precision. The most common is round robin — cycle through servers in order — followed by algorithms that account for how busy each server already is. None of them are exotic; they're closer to a deli's "take a number" system than anything resembling artificial intelligence.
The Common Algorithms, Compared
| Algorithm | How it decides | Best for | Weakness |
|---|---|---|---|
| Round robin | Cycles through servers in fixed order | Uniform request sizes, similar server capacity | Ignores current load per server |
| Least connections | Sends the request to whichever server has the fewest active connections | Long-lived or variable-length requests | Slightly more overhead to track |
| Weighted round robin | Round robin, but bigger servers get proportionally more requests | Mixed-capacity server fleets | Weights need manual tuning |
| IP hash | Routes based on a hash of the client's IP | Simple session affinity without shared state | Uneven distribution if IPs cluster (e.g., behind a corporate NAT) |
The table above compares the mechanism, not the marketing — every major cloud provider's load balancer (AWS's Application Load Balancer, Google Cloud's Load Balancing, NGINX, HAProxy) offers some combination of these. The takeaway for a PM: the algorithm choice is an engineering trade-off between simplicity and fairness, not a product decision you need to weigh in on — but knowing it exists explains why one server can occasionally look "hotter" than others in a dashboard.
- A request arrives at the load balancer.
- The load balancer consults its algorithm and its current view of server health.
- It forwards the request to a chosen, healthy server.
- That server processes the request and returns a response, usually routed straight back to the client.
Why Do Health Checks Matter More Than the Algorithm Itself?
Health checks matter more than the routing algorithm because a perfectly balanced distribution across a dead server is worthless — the algorithm only helps if every server it's choosing between is actually working. A health check is the load balancer periodically pinging each backend server to confirm it's still alive and responsive, and pulling any server that fails from rotation.
Without health checks, a crashed server keeps receiving its share of traffic anyway, and every request routed to it fails. Health checks are what convert "one server crashed" from a partial outage into a non-event — the traffic that would have gone there is quietly redistributed to the servers still standing.
What a Health Check Typically Verifies
- Is the process even running? A basic TCP check — can we open a connection at all?
- Does the application respond? An HTTP check hitting a lightweight endpoint (often literally
/healthor/status) that returns a200if the app is functioning. - Is it responding fast enough? Some checks fail a server that's technically up but too slow to be useful — a distinction covered well in any deep dive on HTTP methods and status codes for PMs.
- How many failures trigger removal? Almost every implementation requires 2-3 consecutive failures before pulling a server, to avoid overreacting to one blip.
This is the concrete mechanism behind the reliability shift mentioned earlier. Reliability comes from redundancy, not one perfect server — you're not trying to prevent any single machine from ever failing (that's unrealistic at any scale); you're building a system that notices failure fast and routes around it. Netflix's engineering team popularized this thinking publicly through their Chaos Engineering practice, deliberately killing production servers to prove the system tolerates it — a philosophy directly downstream of taking health checks and redundancy seriously.
Why Do Sticky Sessions and Shared State Complicate an Otherwise Simple Idea?
Sticky sessions complicate load balancing because they reintroduce the exact problem load balancing exists to solve: dependence on one specific server. A sticky session (also called session affinity) means the load balancer routes a given user's requests to the same backend server for the duration of their session, instead of spreading each request freely.
Why would you want that? Because many applications historically stored session data — like "what's in this user's cart" — in that one server's memory. If the next request lands on a different server that never saw the cart being built, the cart looks empty.
Sticky Sessions, Walked Through a Checkout Flow
- A shopper adds three items to their cart. The load balancer routes that request to Server A, which stores the cart in its local memory.
- The shopper proceeds to checkout. Without sticky sessions, the load balancer might route this request to Server B — which has never heard of this cart.
- Server B returns an empty cart, and the shopper sees their items vanish at the worst possible moment — mid-purchase.
- With sticky sessions enabled, every request from that shopper keeps going back to Server A, so the cart stays intact throughout checkout.
That fixes the symptom but creates a new fragility: if Server A crashes mid-checkout, that shopper's cart is gone regardless of how many other healthy servers exist. You've quietly rebuilt a single point of failure inside a system designed to eliminate them. Sticky sessions trade systemic redundancy for individual consistency — a trade most teams don't realize they're making until the affinity server is the one that goes down during a sale.
The durable fix isn't clever routing — it's moving session state out of any individual server. Store cart and session data in a shared, external store (Redis, Memcached, or a database) that every server can read regardless of which one handles the next request. This is what's called a stateless application tier, and it's the architecture that lets load balancing work as cleanly as the "traffic cop" mental model implies.
Sticky Sessions vs. Shared/Externalized State
| Approach | Where session data lives | Resilience to server failure | Complexity |
|---|---|---|---|
| Sticky sessions | In-memory on one specific server | Low — that server's crash loses the session | Low to set up, but fragile |
| Shared external store | Redis/database, external to all servers | High — any server can pick up any request | Moderate — requires a shared data layer |
Most mature systems converge on shared state because it lets every algorithm in the earlier table work as intended, without carving out exceptions for "but not this user's requests." It's also the pattern that scales cleanly with customer journey flows that span multiple steps and devices — a shopper who starts checkout on mobile and finishes on desktop needs their session available everywhere, not pinned to one machine.
What Can't a Load Balancer Save You From?
A load balancer cannot save you from a bad database, a buggy release, or a downstream dependency that all your servers share — it only protects against the failure of an individual, replaceable server. If every server in your pool queries the same single database and that database goes down, spreading traffic across ten healthy application servers accomplishes nothing.
This is the gap between "the servers are up" and "the product works," and it's a distinction worth internalizing as a PM writing reliability requirements.
- A shared single-instance database is often the real bottleneck — the load balancer distributes traffic to app servers, but if they all funnel into one unreplicated database, that database is your actual single point of failure.
- A bad deploy pushed to every server simultaneously fails everywhere at once — health checks catch a crashed process, not a logic bug that returns the wrong price to every customer.
- A third-party API your product depends on (payments, shipping rates, authentication) being down takes you down too, no matter how many of your own servers are healthy.
- Correlated failures — all your servers running the same region's data center, hitting the same regional outage — defeat the redundancy since "different server" didn't mean "different failure domain."
This is exactly the gap Amazon's own engineering culture points to in its "everything fails all the time" framing (a phrase attributed to former CTO Werner Vogels): load balancing addresses one failure mode well, but a resilient system needs redundancy at every layer that can fail — database, cache, region — not just the application tier. The Site Reliability Engineering discipline that Google formalized and published treats this as a first-class distinction: availability of a service is a property of its weakest dependency, not its best-replicated layer.
Bringing the Infrastructure Layer Back to the Interface Layer
Understanding load balancing changes how you read an incident report or a reliability requirement — you can ask "is this actually a server-capacity problem, or a shared-state problem in disguise?" instead of accepting "we added more servers" as an automatic fix. It also clarifies a subtlety in what an API is for product managers: the API contract your team designs is what actually gets load-balanced — every request to /checkout or /cart is stateless from the load balancer's point of view, hitting whichever healthy server picks it up.
That's a useful thing to have concretely in front of you rather than abstract. Prodinja's API Designing tool lets you sketch the same endpoints — /checkout, /cart, /session — as a spec with example requests and responses, which is a way to see, on paper, exactly what a load balancer is routing between servers underneath. It's a prototype tool for shaping the interface layer, not a substitute for the infrastructure decisions engineering makes about redundancy — but seeing the endpoint list side by side with "these need to be stateless" is a genuinely useful exercise before you write a reliability requirement.
Key Takeaways
- A load balancer distributes requests across identical servers, turning a single point of failure into a pool that can absorb both traffic spikes and individual server crashes.
- Health checks are what make redundancy actually work — routing around a dead server, usually after 2-3 consecutive failed checks, rather than continuing to send it traffic.
- The underlying shift is redundancy over perfection — you stop trying to make one server unbreakable and instead build a system that notices and routes around failure fast.
- Sticky sessions solve a state problem but reintroduce fragility, pinning a user to one server whose crash can lose that user's session mid-checkout.
- Externalizing session state to a shared store (Redis, a database) is the more resilient fix, letting any healthy server pick up any request.
- A load balancer only protects the layer it sits in front of — a shared database, a bad deploy, or a downstream API outage can take down a perfectly load-balanced fleet.
- The API endpoints your team designs are literally what's being load-balanced, which is a useful mental link between interface design and infrastructure reliability.
Frequently Asked Questions
What is a load balancer in simple terms?
A load balancer is software or hardware that sits in front of a group of servers and decides which one handles each incoming request, so no single server gets overwhelmed. It's the mechanism that lets a product handle far more traffic than any one machine could alone.
How does load balancing improve reliability, not just speed?
Load balancing improves reliability because it pairs with health checks — if one server crashes, the load balancer stops sending it traffic and reroutes to the servers still healthy. The speed benefit (spreading load so no server is overworked) is real, but the reliability benefit (surviving individual server failure) is often the bigger reason teams adopt it.
Do sticky sessions defeat the purpose of load balancing?
Sticky sessions don't defeat load balancing entirely, but they narrow its benefit for that specific user — if the server they're pinned to crashes, their session is lost regardless of how many other servers are healthy. Most teams resolve this by externalizing session state so no user depends on one specific server at all.
Is a load balancer the same thing as a CDN?
No — a load balancer distributes requests across your own application servers, while a CDN caches and serves content from edge locations closer to users, often before a request ever reaches your servers. They're frequently used together, but they solve different problems: one is about even distribution, the other about proximity and caching.
What happens if the load balancer itself goes down?
If the load balancer itself fails, every server behind it becomes unreachable even if all of them are healthy — which is why production setups typically run multiple load balancers behind a further layer of redundancy (like DNS failover or a cloud provider's managed, multi-instance load balancing service) rather than a single instance. This is the same redundancy-over-perfection principle applied one layer up, a theme explored further in the broader technical foundations guide for product managers.