LLM VRAM calculator: weights and KV cache, mapped to real GPU configurations

An LLM fits a GPU when its weights plus its KV cache stay inside the memory the serving engine claims, with room to spare. This calculator does that sum for QuantaCloud's GPU configurations. Choose a model, the format of its weights and cache, the context length per request and how many requests you serve at once, and each configuration gets a verdict, the longest context each request can have in the memory left after the weights, and its live price with a Launch link.

How the calculator works#

The method is the one in the KV cache guide: weights plus cache, against the memory each card reports in nvidia-smi. You set six inputs. The model is a preset for one of the models the site's guides cover, or a custom entry with its parameter count, layers, KV heads, head dimension and maximum context. The weight format is BF16 or FP16, FP8 or INT8, or a 4-bit format, and the KV cache is BF16 or FP8. The context length per request starts at 32,768 tokens, concurrent requests at 1, and memory utilization at 0.92, which is vLLM's default.

From those it works out each part of the sum:

Tight is a warning, not a pass. Activations and CUDA graphs need memory the calculator does not count, and they may not fit in what is left, so test a Tight configuration before you rely on it. The sentence under the result tiles works through the lowest-priced configuration that fits, per GPU, and gives its whole pool: the tokens of KV cache the memory left after the weights holds, shared by every running request. Each price is the live hourly rate, billed as pricing explains.

For a custom model, the numbers are in its config.json: num_hidden_layers, num_key_value_heads (not num_attention_heads), head_dim, or hidden_size divided by num_attention_heads when there is no head_dim, and max_position_embeddings for the maximum context. The parameter count is on its Hugging Face page. Count only the layers whose cache grows with context. gpt-oss-120b keeps a growing cache on 18 of its 36 layers, so entering all 36 would double its cache.

The four 48 GB cards, the RTX A6000, RTX 6000 Ada, L40 and L40S, count at the 46,068 MiB they report with ECC on. ECC on GDDR6 memory keeps back a sixteenth of it (49,140 MiB x 15/16), and NVIDIA's product briefs list ECC as on by default for the L40 and L40S. NVIDIA publishes no default for the two workstation cards, so the calculator takes the lower figure for all four. With ECC off, a card reports 49,140 MiB, about 3 GiB more, which adds 2.8 GiB to its budget at 0.92.

On a VM with several GPUs, the weights are split across all of them, and the KV cache across up to as many GPUs as the model has KV heads. That limit comes from vLLM's tensor parallelism, which gives each GPU a share of the KV heads and stores a head on more than one GPU when there are more GPUs than heads. Llama 3.3 70B has 8 KV heads, so its cache spreads over all eight GPUs of an 8-GPU VM. Qwen3-235B-A22B has 4 KV heads, so its cache spreads over four GPUs, and on eight each head is stored twice. The vLLM multi-GPU guide covers the GPU counts tensor parallelism accepts, and when pipeline parallelism, which splits by layers instead, is the better choice.

Worked example: Llama 3.3 70B in FP8 on one H100 PCIe

Llama 3.3 70B, the model the calculator opens on, shows every step with FP8 weights on one H100 PCIe. This is our calculation, from the memory the card reports and the size of RedHatAI's FP8 checkpoint. Checkpoint sizes are in GB (10^9 bytes) and nvidia-smi reports MiB, so both are converted to GiB first:

At the default context the verdict is Does not fit: one request needs 10 GiB and 5.6 GiB is left, so the table gives 18,338 tokens as the longest context per request. At 16,384 tokens the cache needs 5 GiB and leaves 0.6 GiB, under 1 percent of the budget, so the verdict is Tight. The weights alone take 92.4 percent of the budget, so this model never shows Fits on one 80 GB card, and the A100, with 5.9 GiB left after the weights, is in the same position.

Two changes move it. An FP8 cache halves the figure to 160 KiB per token, so the same 5.6 GiB holds about 36,700 tokens. In vLLM that is --kv-cache-dtype fp8, which uses scales of 1.0 unless the checkpoint was calibrated, so check output quality on your own prompts. And one H200 NVL leaves 61.5 GiB after the same weights, room for about 201,000 tokens in all. A single request cannot use more than the model's maximum, so its row shows 131,072 (max). Set six concurrent requests and each can have 33,581 tokens, enough for 32,768 each, though with 1.5 GiB left the verdict is Tight.

What it leaves out#

The calculator counts weights and KV cache only, so its verdicts and token counts are ceilings. A real server also needs memory for these:

Two parts of the sum lean in opposite directions. For a custom model, parameters x bytes is a lower bound on the weights, because real checkpoints keep the embeddings and the output layer in 16-bit: Llama 3.3 70B comes to 70.6 GB in FP8 by parameter count, and RedHatAI's file is 72.67 GB. The cache, on the other side, assumes every request runs at its full context at the same time. vLLM allocates the cache in blocks as each request grows, so traffic with shorter requests fits more of them than the calculator shows. vLLM needs room for one request at the full context to start, and the vLLM out-of-memory guide reads the real figures from its startup log.

FAQ#

How much VRAM does a 70B model need?

Llama 3.3 70B needs 72.67 GB (67.7 GiB) for its FP8 checkpoint before any cache, 141.1 GB (131.4 GiB) in BF16 and 39.8 GB (37.0 GiB) as a 4-bit AWQ file. In FP8 it does not fit a 48 GB card, leaves room for about 18,300 to 19,400 tokens of BF16 cache on one 80 GB card, and about 201,000 on an H200 NVL. In BF16 it is more than one H200 NVL's 129.2 GiB budget, so it needs two GPUs of 80 GB or more, or four 48 GB cards. The AWQ file loads on a 48 GB card with 4.35 GiB, about 14,300 tokens, left for the cache with ECC on. How much VRAM you need covers other model sizes and workloads.

Why does vLLM report fewer KV cache tokens than the calculator?

Because vLLM subtracts what the calculator leaves out. It takes 0.92 of the memory CUDA reports, loads the weights, measures the peak activations, sets aside an estimate for CUDA graphs and turns the rest into cache. The GPU KV cache size line in its startup log is the real figure for your configuration, and the calculator's number is the ceiling above it. For gpt-oss the gap is wider, because vLLM also reserves sliding-window space for each request: a public vLLM 0.30.0 log for gpt-oss-120b on one H200 at a memory utilization of 0.90 shows 1,439,479 tokens, against 1,910,548 by the formula.

Does it work for fine-tuning?

No. Training holds a gradient and an optimizer state for every trainable parameter, plus the activations saved for the backward pass, and serving needs none of them. Full fine-tuning with mixed-precision Adam costs 16 to 18 bytes per parameter before activations, and the published QLoRA tables put a 70B model at 40 to 48 GB. The LoRA, QLoRA and full fine-tuning guide has the figures by model size, and estimating fine-tuning cost turns a run into hours.

What if no configuration fits?

Change the inputs in order of cost: a shorter context, then an FP8 KV cache, then a smaller weight format, then more GPUs in one VM or a GPU with more memory. On 2026-09-27 the largest on-demand VM was 8x A100 SXM4 with 640 GB. A model that needs more, such as DeepSeek-R1 with 688.59 GB of FP8 weights, needs a reserved build, and we build those to order. The DeepSeek GPU requirements guide works through that case, and the reserved capacity page has the brief form.

My rule: size for the context and concurrency you will really serve, pick the smallest configuration that shows Fits, and treat Tight as a reason to test before you rely on it. The first run settles it, because vLLM prints the cache it really got in its startup log, and the vLLM Docker guide shows where to read it. To price the hours on the configuration you pick, use the GPU cost calculator. Both calculators are on the GPU planning tools page, and the GPU catalog lists every configuration with its live price.