What Little's law says, and why it is unusually trustworthy
Little's law relates three averages: the number of items in a system, the rate at which they arrive, and how long each one stays. L = λ × W. If a service completes 250 requests per second and each takes 120 ms on average, then on average 30 requests are inside it at any instant.
What makes the law remarkable is what it does not assume. There is no requirement that arrivals be Poisson, that service times follow any distribution, that the queue be first-in-first-out, or that there be one server rather than many. It needs only that the system be stable over the period you average across — that nothing arrives and never leaves. John Little's 1961 proof established this generality, and it is why the same three-term relationship describes a thread pool, a hospital ward, a supermarket queue and a factory's work in progress.
The practical consequence is that you can never move one of the three quantities without moving another. If throughput doubles and response time holds, concurrency doubles. If concurrency is capped by a pool size and throughput rises, response time must rise to compensate. A performance goal that changes one number while promising the other two are unaffected is arithmetically impossible, and this identity is the fastest way to show it.
Where the law is silent is on why W has the value it does. It tells you the relationship between the three, not what happens to response time when you add load — that is queueing theory, and it is much less forgiving.
Rearranging the law, and the two derived quantities
All three forms are the same equation. L = λ W gives concurrency from a measured rate and latency. λ = L / W gives the throughput a fixed pool can deliver — the form to use when concurrency is capped by a connection pool or a thread limit. W = L / λ gives the response time implied by a level of concurrency, which is how you show that a large pool under heavy load must be queueing.
Keep the units consistent: λ in requests per second and W in seconds. Milliseconds are the natural unit for latency and the most common source of a factor-of-1000 error, so this calculator converts explicitly and shows the converted value in the steps.
Worker slots = ⌈L / ρ⌉, where ρ is the target utilisation. Sizing a pool to L exactly would mean every worker busy all the time, which no real system achieves, because arrivals are not evenly spaced. Dividing by a target utilisation of 70% gives the slack that absorbs the bunching. The choice of target is a judgement, and the queueing consequences of choosing it badly are covered below.
Virtual users = λ × (W + Z), where Z is think time. This is Little's law applied to a larger system that includes the user sitting idle between requests. A user occupies the loop for W seconds of service and Z seconds of thinking, so generating a given request rate takes proportionally more users as think time rises. Load tests that report “500 concurrent users” without stating think time are reporting almost nothing: at 5 s think time and 200 ms response, those 500 users generate about 96 requests per second, while with no think time they would generate 2,500.
One caution on which average to use. The law is exact for the mean response time and gives no information about percentiles. Feeding it a 95th-percentile latency produces a concurrency figure that is not the average concurrency and not the peak either — it is not a quantity that means anything.
Worked example: sizing a thread pool for 250 requests per second
An API handles a peak of 250 requests per second with a mean response time of 120 ms. You are choosing the size of its request-handling thread pool and the database connection pool behind it.
- Convert the units. 120 ms = 0.12 s.
- Apply the law. L = 250 × 0.12 = 30 requests in flight on average. That is the raw concurrency the service carries at peak.
- Add headroom. At a 70% utilisation target, the pool needs ⌈30 ÷ 0.7⌉ = ⌈42.857⌉ = 43 slots. The extra 13 exist because arrivals bunch; they are idle on average and busy exactly when they are needed.
- Check the ceiling. With 43 slots each turning over a 120 ms request, the absolute maximum is 43 ÷ 0.12 = 358.33 requests per second — and only if every slot is busy every instant, which means unbounded queueing. Treat it as a hard ceiling, not a target.
- Confirm the actual utilisation. 30 ÷ 43 = 69.8%, just under the 70% target because the slot count was rounded up.
- Size the connection pool. If 40 ms of each request is spent in the database, then database concurrency is 250 × 0.04 = 10 connections on average, so a pool of ⌈10 ÷ 0.7⌉ = 15 covers the same target. The connection pool is much smaller than the thread pool, because connections are held for a third of the request.
- Size the load test. With no think time, generating 250 req/s needs 250 × 0.12 = 30 virtual users. Add 3 s of think time and it needs 250 × 3.12 = 780. Same server load, twenty-six times the user count.
Step 6 is the one people skip, and it is where over-provisioned connection pools come from. A pool sized to the thread count rather than to the time actually spent in the database wastes database memory and, worse, allows far more concurrent queries than the database can serve well.
Choosing a utilisation target, and what the law cannot tell you
The law gives you L. It does not tell you how big to make the pool, and the difference between the two is where queueing theory takes over.
In the standard M/M/1 model — Poisson arrivals, exponential service times, one server — the average number of items in the system is ρ/(1−ρ), where ρ is utilisation. That is 1 at 50%, 4 at 80%, 9 at 90% and 19 at 95%. The queue grows without bound as utilisation approaches 1, and by Little's law the response time grows with it. This is why 100% utilisation is not an achievable operating point but a limit, and why the last few percent of a resource cost so much more than the first few.
Real systems are usually worse than M/M/1 rather than better, because request arrivals are bursty rather than Poisson and service times have long tails. That argues for targets in the 60–75% range for latency-sensitive services, with higher targets reserved for batch work where queueing delay does not matter. The table below shows how the slot count moves with the target for your own concurrency figure.
Three limits on the law itself are worth stating plainly. It describes averages over a period, so a system that is stable across an hour can still be saturated for ten seconds inside it. It requires stability: if arrivals exceed service capacity, the queue grows without limit and no steady-state L exists. And it says nothing about causation — finding L = 30 does not mean 30 threads will deliver 250 req/s, because adding threads to a CPU-bound service raises W through contention and can lower throughput. The law is an accounting identity that any correct capacity model must satisfy; it is not a capacity model on its own.
Concurrency for common throughput and latency combinations
| Response time | 100 req/s | 250 req/s | 1,000 req/s | 5,000 req/s |
|---|---|---|---|---|
| 10 ms | 1 | 2.5 | 10 | 50 |
| 50 ms | 5 | 12.5 | 50 | 250 |
| 100 ms | 10 | 25 | 100 | 500 |
| 250 ms | 25 | 62.5 | 250 | 1,250 |
| 500 ms | 50 | 125 | 500 | 2,500 |
| 1 s | 100 | 250 | 1,000 | 5,000 |
| 2 s | 200 | 500 | 2,000 | 10,000 |
Concurrency is linear in both variables, so halving response time halves the concurrency required at the same throughput — which is why latency work reduces the hardware needed as directly as it improves the user experience.
Ways this calculation is misused
- Feeding it a percentile instead of a mean. Little's law is exact for the average and undefined for the 95th percentile. A pool sized from p95 latency is oversized by whatever the tail happens to be.
- Assuming more workers means more throughput. Beyond the point where a bottleneck resource saturates, extra workers add contention and raise W, so λ stops rising and may fall. The law still holds; the numbers just move the other way.
- Mixing units. Response time in milliseconds against throughput per second is the classic factor-of-1000 error, and the result looks plausible enough to survive review.
- Averaging across a period that hides the peak. An hourly mean can be comfortable while a ten-second burst saturates the pool. Size on the peak interval that matters to users.
- Reporting virtual users without think time. 500 users means nothing on its own; 500 users at 200 ms response and 5 s think time is about 96 requests per second.
- Using it on an unstable system. If arrivals exceed capacity there is no steady state, the queue grows without limit, and any L you calculate is an artefact of the measurement window.
- Sizing a downstream pool from an upstream one. Connection pools should be sized from the time actually spent downstream, which is usually a fraction of the request, not from the thread count.
Where the law fits in performance and capacity work
Little's law is the connective tissue between measurements you already have and decisions you have to make. Monitoring gives you throughput and response time; the law converts them into concurrency, which is what pool sizes, worker counts and licence tiers are actually expressed in. Running the conversion in the other direction turns a configured limit into the throughput ceiling it implies, which is often the fastest explanation of why a service plateaus at a particular rate.
It also underpins the standard load-testing models. A closed model with N virtual users and think time Z produces λ = N / (W + Z), which is the same identity rearranged, and it explains the characteristic shape of a closed-loop test: as the system slows, throughput falls rather than the queue growing, because users cannot issue their next request until the previous one returns. Open-model tests, which inject a fixed arrival rate regardless of response, behave like the real world instead and expose saturation that closed tests hide.
For CPU capacity rather than concurrency, the companion calculation is service demand — CPU seconds consumed per request — which the server vCPU sizing calculator turns into a core and instance count. For the parallel-speedup limits that determine how much extra hardware can help at all, see the Amdahl's law speedup calculator. On the network side, the bandwidth-delay product calculator is Little's law in different clothing: bits in flight equals bit rate multiplied by round-trip time, exactly L = λ W with bits for requests.
Finally, the law is a good habit as well as a formula. Whenever someone claims a change will raise throughput without affecting latency or concurrency, write down the three numbers before and after. One of them has to move.