AI, LLM & Machine Learning Engineering

Model Memory & GPU Sizing Calculators

Whether a model fits, and on what: weight memory at each precision, KV-cache growth with context length and batch size, the memory a quantisation step actually saves, and what the GPU hours cost.

4 calculators in this category

LLM GPU VRAM Requirement CalculatorThis calculator adds up the four things that occupy GPU memory when you run a language model: the weights, the KV cache, the…
KV Cache Size CalculatorTransformer key-value cache memory modelEvery token a transformer has already seen leaves a key and a value vector in memory for each attention layer, and that cache…
Model Quantisation Memory Savings CalculatorWeight-precision memory accountingA model's weight memory is just parameters times bytes per parameter, so quantisation buys memory in exact proportion to the…
GPU Hours Cost CalculatorGPU-hour cost accountingA GPU-hour is the unit every cloud accelerator bill is written in: one GPU held for one hour. This calculator turns an hourly…

More ai, llm & machine learning engineering categories

LLM Inference Cost & Serving Training & Fine-Tuning Compute RAG, Embeddings & Vector Search Model Evaluation & Metrics

Related categories in other subjects

Where this work overlaps other trades and disciplines.

Bandwidth, Throughput & LatencyComputing Computer Science, Data & Application MetricsComputing