How Load Balancers Work: Round Robin vs Least Connections

One address, many servers

It's sale day, and thousands of people open the same shop at the same moment. They all type one address, but no single computer could serve them all. So who decides which computer answers you?

Picture the servers as checkouts, and every visit as a shopper. At the door stands a host who sends each shopper to a checkout. That host is a load balancer: one front door, and many servers behind it.

Taking turns

The simplest rule is taking turns: checkout 1, then 2, then 3, then 4, and around again. It's called round robin, and it's what NGINX does when you choose nothing else.

But one shopper brings a cart for the whole month. Taking turns keeps sending people to that checkout anyway, and the line grows while others stand empty.

Counting the line

So count instead. Before each shopper, the host checks how many people each checkout is still serving, and picks the fewest. That's least connections.

Is counting always better? When every visit takes about the same time, taking turns is the smoothest and fairest, as the HAProxy manual puts it. Counting wins when some visits take far longer.

When a checkout breaks

Now a card reader dies. Counting can't see that, so the host keeps asking every checkout: are you working? Amazon's cloud load balancer asks every 30 seconds by default. After 2 failed answers in a row, no one is sent there, and after 5 good ones, it's back.

Why not one big server

Why not buy one giant checkout instead? Even the biggest has a limit, and when it breaks, the whole shop stops. Behind a load balancer, you add checkouts as the crowd grows, and one failure closes one checkout, not the shop.

In practice

In NGINX, switching from turns to counting is one line in the setup.

So a load balancer makes one small choice for every visit: which server answers, by turns or by counting, and never one that's down.

What's next

Subscribe for the next system in miniature. Before any of this, your browser had to find the shop. How does it turn a name into an address?