Service demand is the number that makes sizing possible
The quantity that turns a traffic figure into a hardware figure is service demand: the CPU time one request consumes, summed across every core it touches. It is not the same as response time. A request that takes 200 ms of wall clock but spends 160 ms waiting on a database consumes 40 ms of CPU, and it is the 40 ms that determines how many cores you buy.
Get service demand and the rest is arithmetic. Multiply it by the request rate and you have the number of cores that must be busy continuously — 1,200 requests per second at 35 ms each is 42 vCPU of pure work. Divide by a utilisation target because no fleet can run flat out, add headroom for growth, round up into whole instances, and add spares for failure. Every step after the first is a policy decision; only the first is a measurement.
Measure it rather than estimate it. Total CPU-seconds consumed over an interval divided by requests served in that interval gives the average directly, and every monitoring system can produce both numbers. A profiler gives the same figure with a breakdown you can act on. What you must not do is derive it from response time, which includes all the waiting that costs you nothing in cores.
This calculation covers CPU. If memory, database connections, disk throughput or an external API quota binds first, the fleet size is set by that instead — sizing on CPU alone is right only when CPU is the constraint, and confirming that is part of the work.
From request rate to instance count
Busy cores = X × D. Throughput multiplied by service demand. This is the utilisation law from queueing network analysis, and in these units it is the number of vCPU that would be occupied every instant if the work arrived perfectly smoothly. At 1,200 req/s and 35 ms of CPU each, that is 1,200 × 0.035 = 42 vCPU.
Required cores = busy cores ÷ target utilisation × (1 + growth). Traffic does not arrive smoothly, and a resource driven close to saturation queues. Dividing 42 by a 65% target gives 64.62 vCPU, and a 20% growth allowance brings it to 77.54. The utilisation target is where queueing theory enters an otherwise linear calculation: as utilisation approaches 100%, waiting time rises without bound, so the last few percent of a core is unusable in practice.
Instances = ⌈required ÷ vCPU per instance⌉ + spares. Rounding up is where instance size starts to matter. 77.54 vCPU across 4-vCPU instances is 20 nodes and 80 vCPU, wasting 2.46 vCPU to rounding; across 32-vCPU instances it is 3 nodes and 96 vCPU, wasting 18.46. Smaller instances round more finely; larger ones amortise per-node overhead better and are simpler to operate. The table below shows both effects for your own numbers.
Utilisation with one instance lost = busy cores ÷ ((instances − 1) × vCPU each). This is the figure that decides whether the redundancy is real. A fleet of 21 four-vCPU instances has 84 vCPU; lose one and 80 remain, so 42 busy cores become 52.5% utilisation — comfortably inside target. A fleet of 3 thirty-two-vCPU instances has 96 vCPU; lose one and 64 remain, giving 65.6%, which is at the edge. Same total capacity, very different failure behaviour, and the reason large instances need a higher spare count rather than the same one.
Maximum sustainable rate = total vCPU × target ÷ D. Reading the calculation backwards gives the headroom in the units the business speaks: the fleet above supports 84 × 0.65 ÷ 0.035 = 1,560 requests per second before it exceeds its own utilisation target.
Worked example: 1,200 requests per second on 4-vCPU nodes
A service peaks at 1,200 requests per second. Monitoring shows 42 CPU-seconds consumed per wall-clock second at peak across the current fleet, and 1,200 requests served each second, so service demand is 42 ÷ 1,200 = 35 ms per request. The team targets 65% utilisation, deploys 4-vCPU containers, wants N+1, and allows 20% for growth.
- Convert service demand. 35 ms = 0.035 CPU-seconds per request.
- Find the busy cores. 1,200 × 0.035 = 42 vCPU continuously busy at peak. This matches the measured 42 CPU-seconds per second, which is the consistency check worth doing every time.
- Apply the utilisation target. 42 ÷ 0.65 = 64.62 vCPU.
- Add growth headroom. 64.62 × 1.20 = 77.54 vCPU.
- Round into instances. 77.54 ÷ 4 = 19.38, so ⌈19.38⌉ = 20 instances for capacity.
- Add redundancy. N+1 gives 21 instances, which is 21 × 4 = 84 vCPU deployed.
- Check utilisation at peak. 42 ÷ 84 = 50%. That is below the 65% target because rounding up, the growth allowance and the spare instance all add capacity above the minimum — the target applies to the capacity you sized for, and the fleet you deploy is larger.
- Check the failure case. Lose one instance and 20 remain, giving 80 vCPU. 42 ÷ 80 = 52.5%, still well inside target, so a single failure at peak costs nothing in latency.
- Find the ceiling. 84 × 0.65 ÷ 0.035 = 1,560 requests per second before the fleet exceeds its own 65% target — 30% above today's peak, which is the growth allowance showing up in traffic terms.
Step 7 catches a common misreading. Sizing to 65% does not mean running at 65%; it means the capacity was chosen so that 65% would be the utilisation if there were no rounding, no growth allowance and no spare. Compare the deployed utilisation against the target and the difference is exactly the margin you deliberately bought.
Choosing the target, the instance size and the spare count
Utilisation target. For latency-sensitive request handling, 60–75% is the usual band. The reason is queueing: as a resource approaches saturation, waiting time rises steeply rather than linearly, so the same one-second burst that is absorbed at 60% causes visible latency at 90%. Batch and asynchronous work, where queueing delay is not user-visible, is routinely run far higher, and running batch capacity at 85–95% is normal practice rather than a risk.
Instance size. Small instances round finely, spread failure impact thinly and scale in small increments, at the cost of more per-node overhead — agents, sidecars, operating system memory and a share of every fixed cost. Large instances are the reverse: less overhead per core, coarser rounding, and each failure removing a larger share of the fleet. The practical guide is the failure column in the table: if losing one instance pushes utilisation past your target, the instances are too large for the spare count you have chosen.
Spare count. N+1 protects against one concurrent failure, which covers a host failure or a rolling deployment taking one node out at a time. N+2 is worth it when a whole failure domain can go at once, when deployments replace more than one instance at a time, or when the recovery time for a failed node is long relative to how often nodes fail. On a fleet spread across three availability zones, losing a zone removes a third of the capacity — which is a different and larger question than N+1, and this calculator models single-instance loss rather than zone loss.
Autoscaling changes what you are sizing. With autoscaling you are choosing the minimum fleet, the scaling target and the maximum, not a fixed count. The figures here map onto that directly: the target utilisation becomes the scaling metric target, the redundant instances become the minimum, and the maximum comes from the peak you are willing to serve. What autoscaling does not remove is the need for headroom, because scaling takes time — a node that needs three minutes to become useful cannot answer a burst that arrives in thirty seconds.
vCPU required for common service demands
| CPU per request | 500 req/s | 1,000 req/s | 5,000 req/s | 1,000 req/s sized at 65% |
|---|---|---|---|---|
| 1 ms | 0.5 vCPU | 1 vCPU | 5 vCPU | 1.54 vCPU |
| 5 ms | 2.5 vCPU | 5 vCPU | 25 vCPU | 7.69 vCPU |
| 10 ms | 5 vCPU | 10 vCPU | 50 vCPU | 15.38 vCPU |
| 25 ms | 12.5 vCPU | 25 vCPU | 125 vCPU | 38.46 vCPU |
| 35 ms | 17.5 vCPU | 35 vCPU | 175 vCPU | 53.85 vCPU |
| 50 ms | 25 vCPU | 50 vCPU | 250 vCPU | 76.92 vCPU |
| 100 ms | 50 vCPU | 100 vCPU | 500 vCPU | 153.85 vCPU |
| 250 ms | 125 vCPU | 250 vCPU | 1,250 vCPU | 384.62 vCPU |
The relationship is linear in both directions, so halving service demand halves the fleet at the same traffic — usually the cheapest capacity available to a team.
Assumptions and where they break
- CPU is assumed to be the binding resource. If memory, connection limits, disk or an external quota saturates first, the fleet is set by that instead and this figure is a lower bound.
- Service demand is assumed constant. Real demand varies by endpoint and by payload size, so a mixed workload needs a weighted average — and a traffic mix that shifts changes the average without any code changing.
- vCPU are not all equal. A vCPU is usually a hardware thread rather than a core, so two vCPU on one physical core deliver less than two independent cores. Measure service demand on the instance family you will actually deploy.
- Burstable instance families are excluded. Credit-based instances deliver full performance only until credits run out, which makes them unsuitable for sustained peak load however good the price looks.
- Peak means the peak that matters. A one-minute peak is what the fleet has to absorb; sizing on a five-minute or hourly average understates it, sometimes badly.
- Deployment overlap is not modelled. A rolling deployment that runs old and new instances together temporarily needs capacity for both, which can exceed the N+1 spare.
- Scaling is not instantaneous. Autoscaling reduces steady-state cost but does not remove the need for headroom, because instances take time to start, warm up and pass health checks.
How this fits with the rest of capacity planning
Two laws do most of the work in capacity planning, and they answer different questions. The utilisation law used here — throughput times service demand — tells you how much of a resource a load consumes. Little's law tells you how many requests are in flight, which is what thread pools, connection pools and load-test virtual user counts are sized from. A complete plan needs both: the core count from this page and the concurrency from that one, because a fleet with enough CPU and too few worker threads still queues.
Where parallelism is involved, Amdahl's law sets the ceiling on what extra cores can achieve inside a single request. It is the reason a service can be CPU-bound and still fail to get faster on a bigger instance: only the parallel fraction benefits, and the serial fraction sets the floor on latency no matter how many cores you add.
On the cost side, the instance count from this page multiplied by an hourly rate is the compute line of the bill, which the cloud VM monthly cost calculator works through including reserved and spot pricing. Bandwidth is billed separately and often exceeds compute for content-heavy services — see the cloud egress cost calculator. And the availability commitment those instances support is a separate calculation again, covered by the uptime and SLA downtime calculator.
The single most valuable habit here is to re-measure service demand after every significant release. It is the only input that is a property of your code rather than a policy choice, and it moves silently: a new dependency, an extra serialisation pass or a lost cache can change it by tens of percent without changing any traffic figure at all. A capacity plan built on last quarter's service demand is a plan for last quarter's software.