What AMAT measures and why a single ratio is not enough
Average memory access time is the expected latency of one access, averaging over hits and misses. It exists because a hit ratio on its own cannot be compared between systems: 90% is excellent when a miss costs 20 ns and disastrous when a miss costs 10 ms. AMAT folds the ratio and the penalty into one number in units of time, which is the unit decisions are actually made in.
The structure is worth reading carefully. The hit time is paid on every access, hits and misses alike, because the cache has to be checked before anything else can be consulted. The miss penalty is paid only on the fraction that miss. So AMAT = thit + m × P, and the two terms behave completely differently: the first is a floor you cannot get below by improving the hit ratio, and the second is the part that responds to it.
That structure has an immediate consequence people find surprising. A cache with a zero hit ratio is worse than no cache, by exactly its hit time — you pay the lookup and then pay the backing store anyway. There is a break-even hit ratio below which caching is a net loss, and it is not always negligible: with a 4 ns hit time and a 100 ns backing store the break-even is 4%, but with a 20 ms edge lookup and a 30 ms origin it is 67%.
The model is the one Hennessy and Patterson use for processor caches, and it applies unchanged to every other layer that has the same shape. A browser cache in front of a CDN in front of an origin is a two-level hierarchy. A Redis instance in front of a database is a single-level cache with a hit time near 0.5 ms and a miss penalty in the tens of milliseconds. An operating system page cache in front of an SSD is another. The arithmetic does not care what the layers are made of.
Local miss rate, global miss rate, and why the distinction matters
For a single cache, AMAT = thit + m × tmem. Nothing subtle.
For two levels, the cost of a first-level miss is itself an AMAT: it is the second level's hit time plus the second level's miss rate times the backing store. So AMAT = t₁ + m₁(t₂ + m₂·tmem). Each level is a cache in front of everything below it, and the recursion continues for as many levels as you have.
The term that causes trouble is m₂. It is the local miss rate — misses at the second level divided by the accesses that reached the second level, which is only the first level's misses. It is not the fraction of all accesses that miss at level two. Local rates always look bad: an L2 serving 60% of the requests that L1 could not is doing well, even though 60% would be an alarming figure for a first-level hit ratio.
The global miss rate is the product of the local miss rates, m₁ × m₂, and it is the number that matters operationally, because it determines how many requests reach the backing store. With a 95% L1 hit ratio and a 60% L2 local hit ratio, the global miss rate is 0.05 × 0.40 = 2%. Ninety-eight per cent of traffic never touches the layer behind, and the request rate to that layer falls by a factor of fifty.
Two multi-level effects the formula deliberately shows. First, adding a level always adds its hit time to every miss from the level above, so a second level with a poor local hit rate can make AMAT worse. Second, the levels are not independent in practice: an inclusive L2 holds everything L1 holds, so its effective capacity for new data is smaller than its size suggests. The formula treats the local rates as given, which is the right approach when you have measured them and the wrong approach when you are predicting them from cache sizes.
Worked example: a two-level hierarchy, then the value of one more nine
A processor has a first-level cache with a 4 ns hit time. Over a measurement window it records 9,500,000 hits and 500,000 misses. The second-level cache serves 60% of those misses with a 15 ns hit time, and main memory takes 100 ns. Traffic is 106 accesses per second.
- First-level ratios. 9,500,000 ÷ 10,000,000 = 95% hit, so m₁ = 0.05.
- Second-level local miss rate. 1 − 0.60 = 0.40.
- Cost of a first-level miss. 15 + 0.40 × 100 = 55 ns.
- AMAT. 4 + 0.05 × 55 = 4 + 2.75 = 6.75 ns.
- Global miss rate. 0.05 × 0.40 = 2%.
- Load on memory. 106 × 0.02 = 20,000 accesses per second, down from 106.
- Speedup against no cache. 100 ÷ 6.75 = 14.8×.
Remove the second level and the same first-level ratios give AMAT = 4 + 0.05 × 100 = 9 ns, a global miss rate of 5%, and 50,000 accesses per second on memory. The second level is therefore worth 2.25 ns per access and removes 30,000 memory accesses a second — the sort of trade that is easy to argue about and easy to settle with arithmetic.
Now ask what improving the first level buys. Going from 95% to 99% takes the miss term from 0.05 × 55 = 2.75 ns to 0.01 × 55 = 0.55 ns, so AMAT falls from 6.75 to 4.55 ns — a saving of 2.20 ns. Going further, from 99% to 99.8%, takes the miss term to 0.002 × 55 = 0.11 ns and AMAT to 4.11 ns, a saving of just 0.44 ns. Both steps cut the miss rate by the same factor of five, and the second delivers exactly one fifth as much time — 0.44 against 2.20 — because the 4 ns hit time is a floor that neither can touch.
Reading the result: the floor, the break-even, and the offload
Watch the floor. AMAT can never fall below the hit time, so the useful question is not “how close to 100% can we get” but “how much of AMAT is still the miss term”. When the miss term is already small relative to the hit time, further hit-ratio work is spent, and the remaining lever is a faster cache rather than a bigger one.
Check the break-even. Caching is a net loss below a hit ratio of 1 − (tmem − thit)/P. Where the cache is fast relative to the backing store this is close to zero and you can cache almost anything. Where the cache is slow relative to the backing store — a remote cache in front of a local database, or an edge cache in front of a nearby origin — the break-even can be most of the way to 100%, and caching data with poor reuse actively hurts.
Use the offload figure for capacity, not the latency figure. For a CDN or a database cache, the operationally important output is usually the request rate reaching the backing store, because that is what determines how many origin servers or database replicas you need. Moving from a 90% to a 95% hit ratio halves that load, which is a much larger effect on cost than the latency change. It also halves egress from the origin, which you can price with the cloud egress cost calculator.
Do not average across workloads. A cache serving 99% on static assets and 40% on API responses has a blended figure that describes neither, and the two need different fixes.
Remember that latency is not throughput. AMAT tells you what one access costs when nothing is queueing. Under load, a backing store near saturation adds queueing delay that grows far faster than linearly, so the real penalty at 90% utilisation is much larger than its unloaded latency. Relate arrival rate, service time and concurrency with Little's law before assuming the miss penalty is constant.
AMAT and speedup by hit ratio
| Hit ratio | Miss term (ns) | AMAT (ns) | Speedup | Saved vs previous row (ns) |
|---|---|---|---|---|
| 50% | 50.0 | 54.0 | 1.85× | — |
| 80% | 20.0 | 24.0 | 4.17× | 30.0 |
| 90% | 10.0 | 14.0 | 7.14× | 10.0 |
| 95% | 5.0 | 9.0 | 11.11× | 5.0 |
| 99% | 1.0 | 5.0 | 20.00× | 4.0 |
| 99.9% | 0.1 | 4.1 | 24.39× | 0.9 |
| 100% | 0.0 | 4.0 | 25.00× | 0.1 |
The last column is the point of the table. Each halving of the miss rate halves the miss term, so equal effort at higher ratios buys progressively less time — and the total available below 99% is only 1 ns, against 50 ns available below 50%.
The same formula at every layer of the stack
Substitute the latencies and the model works unchanged wherever a fast small store sits in front of a slow large one. Approximate figures worth carrying: an L1 cache hit is a handful of nanoseconds and main memory is around 100 ns; a local Redis lookup is a few hundred microseconds and a database query a few milliseconds; a CDN edge hit is tens of milliseconds and an origin fetch across an ocean is hundreds. In each pair the ratio between hit and miss is one to two orders of magnitude, which is exactly the condition that makes caching worthwhile.
What changes between layers is the break-even hit ratio, because it depends on the ratio of the hit time to the miss penalty. Processor caches are so much faster than memory that almost any reuse pays. A remote cache whose network round trip is comparable to the query it replaces needs a genuinely high hit ratio before it earns its place, and it is worth computing that threshold before deploying one.
Pitfalls when measuring and modelling caches
- Confusing local and global miss rates. A second-level cache's hit ratio is quoted against the requests that reached it, not against all requests. Multiplying the miss rates gives the global figure.
- Ignoring the hit time on misses. The cache is checked first every time, so a miss costs hit time plus penalty. Leaving out the hit time understates AMAT and hides the fact that a low-hit cache is a net loss.
- Measuring over a window that includes a cold start. Compulsory misses on an empty cache depress the ratio. Measure in steady state, and report warm-up separately.
- Treating the miss penalty as constant under load. A saturated backing store queues, and its effective latency rises sharply. The unloaded figure is a floor.
- Blending unlike workloads. A single ratio over static assets and dynamic responses describes neither and misleads about both.
- Forgetting write traffic. Write-through, write-back and write-around policies have entirely different miss costs, and a read-only AMAT model can be badly wrong on a write-heavy workload.
- Assuming a bigger cache raises the hit ratio proportionally. Hit ratio against size is a strongly diminishing curve set by the workload's reuse distances; doubling the cache often buys a fraction of a percentage point.
- Counting revalidations as hits. A conditional request returning 304 Not Modified still paid a full round trip, so it is a hit for bandwidth and a miss for latency.
Key terms
- AMAT
- Average memory access time — the expected latency of one access, equal to hit time plus miss rate times miss penalty. Also called effective access time.
- Local miss rate
- Misses at a cache level divided by the accesses that reached that level. Always higher than the global rate for any level below the first.
- Global miss rate
- Misses that reach the backing store divided by all accesses — the product of the local miss rates. The figure that determines load on the layer behind the cache.
- Compulsory, capacity and conflict misses
- The three classic causes: data never seen before, a cache too small to hold the working set, and entries evicting each other despite free space elsewhere. Each responds to a different fix.
- Break-even hit ratio
- The hit ratio at which AMAT equals the backing-store latency. Below it, the cache adds more latency in lookups than it saves in avoided fetches.
What to change when the number is disappointing
The three classic miss causes point at three different fixes, and knowing which you have is worth more than any amount of tuning. Compulsory misses are unavoidable first touches; prefetching and larger blocks help, warming does not. Capacity misses mean the working set exceeds the cache; a bigger cache helps, and so does making the working set smaller by compressing entries or storing only what is reused. Conflict misses mean entries are evicting each other despite space being free elsewhere; higher associativity, a different hash, or changing the access pattern helps, and more capacity does not.
Beyond the cache itself, two levers sit in the formula and are often cheaper than raising the hit ratio. Reducing the hit time lowers the floor that no ratio improvement can reach — moving a cache closer to the caller, or from a network hop to a process-local store, does this. Reducing the miss penalty shrinks the whole miss term: a faster backing store, a read replica nearer the caller, or a smaller object to fetch all count.
When you are choosing which of those to spend on, the arithmetic resembles Amdahl's law, and for the same reason: you are improving one component of a total and the rest of the total bounds the benefit. If the hit time is 60% of AMAT, no achievable hit-ratio improvement can more than shave the other 40%, exactly as the Amdahl's law speedup calculator shows for the serial fraction of a program. Recognising which term dominates is the whole of the analysis, and it is the same discipline that keeps an algorithmic estimate honest in the big-O operations estimator.