vLLM quantization: which format to run on which GPU

The rule I follow is to pick the quantization format by GPU generation, because vLLM runs some formats at full speed only on newer GPUs and falls back to a weight-only path on older ones. On the RTX A6000 and A100 (Ampere), 4-bit AWQ or GPTQ is the format to use, and FP8 only saves memory. On the L40S, L40 and RTX 6000 Ada, use per-channel FP8 checkpoints, which get FP8 math. On the H100 PCIe and H200 NVL, any FP8 checkpoint gets FP8 math. In the 2026-09-27 catalog, the RTX PRO 6000 was the only on-demand GPU that computes in NVFP4. In every case vLLM reads the format from the checkpoint, so a pre-quantized model needs no --quantization flag.

The formats vLLM 0.30 loads#

vLLM reads a checkpoint's quantization_config and picks the matching method, and the table lists the ones you will meet on Hugging Face. The --quantization value is only needed to override that detection or to quantize a 16-bit model while it loads.

Two formats moved out of vLLM. bitsandbytes now needs vllm-bnb-plugin and GGUF needs vllm-gguf-plugin, and vLLM's docs call its GGUF support highly experimental and under-optimized. If a model only exists as GGUF, Ollama or llama.cpp is the natural server for it, and vLLM vs Ollama covers what changes when you switch.

What each format needs from the GPU#

The dividing line is compute capability, as vLLM 0.30.0's kernel code draws it. FP8 math needs 8.9 or newer, block-scaled FP8 math needs 9.0 or newer, and FP4 math needs Blackwell. When a GPU cannot run a format's math, vLLM keeps the weights compressed and runs them through its Marlin kernel as 16-bit math: you keep the memory saving and lose the speed of the low-precision Tensor Cores.

AWQ and GPTQ never compute in 4 bits. vLLM unpacks the weights inside its kernels, Marlin on every GPU here and Machete on Hopper when it supports the layout, so their benefit is memory and the bandwidth saved reading smaller weights. MXFP4 loads on every QuantaCloud GPU, but vLLM's gpt-oss recipe still lists Ampere and Ada as work in progress, which the gpt-oss GPU guide covers.

The weight-only fallback announces itself. vLLM logs a warning that begins Your GPU does not have native support for FP8 computation but FP8 quantization is being used, or the same sentence for FP4, and names Marlin. If you see it on a card you chose for its FP8 or FP4 Tensor Cores, you loaded the wrong checkpoint for that card. NVFP4 explained and FP8 vs FP16 vs BF16 go deeper into the number formats.

Memory saved, from real checkpoint sizes#

Quantization roughly halves the weights at 8 bits and cuts them by about 70% at 4 bits. The sizes below are the checkpoint files on Hugging Face, which run larger than the parameter count suggests because the embedding and output layers stay 16-bit and 4-bit formats carry scales.

NVFP4 files come out a little larger than AWQ and GPTQ, because every 16 values carry an FP8 scale. The Llama 3.3 checkpoints carry Meta's Llama 3.3 licence, and the base repo is gated. The Qwen3 checkpoints are Apache-2.0.

What fits on 48, 80, 96 and 141 GB#

The saving matters because of what it leaves for the KV cache. vLLM takes 92% of the GPU by default, and after the weights the rest holds the context of every running request: 256 KiB per token for Qwen3-32B and 320 KiB for Llama 3.3 70B at BF16. This table is our calculation from the memory each card reports in nvidia-smi, with ECC on for the 48 GB cards, before activations and CUDA graphs take their share:

Read each cell as the pool all running requests share. A 32B model at 4 bits on a 48 GB card has more room for context than the same model at FP8 on it, and a 70B model at FP8 is effectively a single-conversation setup on an 80 GB card. The KV cache guide has the formula for other models, and fixing vLLM out-of-memory errors covers what to do when the cache does not fit.

Two notes apply to the table. The 48 GB column uses the 46,068 MiB, about 45 GiB, these cards report with ECC on, and an RTX A6000 with ECC off reports 49,140 MiB, which adds about 2.8 GiB to each of its cells (our calculation: 0.92 x 3 GiB). And NVIDIA's NVFP4 checkpoint of Llama 3.3 70B sets an FP8 KV cache in its quantization config, which vLLM 0.30.0 applies by default, so that row holds about twice the tokens shown.

Serve a quantized checkpoint#

A pre-quantized checkpoint runs with the same command as any other model. The setup, including the driver check that sets $VLLM_TAG and the API key, is in deploying vLLM with Docker. On a 48 GB Ada card such as the L40S, the per-channel FP8 build of Qwen3-32B:

docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  "vllm/vllm-openai:$VLLM_TAG" \
  RedHatAI/Qwen3-32B-FP8-dynamic \
  --max-model-len 16384 \
  --reasoning-parser qwen3 \
  --api-key "$VLLM_API_KEY"

docker logs -f vllm

The limit is 16,384 tokens because this model is tight on the L40S. One 32,768-token request needs 8 GiB of KV cache, and 92% of the 45 GiB the card reports leaves about 9.4 GiB after the 32 GiB of weights, before activations and CUDA graphs take their share (our calculation). --max-model-len auto finds the real ceiling if you need a longer context.

When the log shows Application startup complete, press Ctrl+C and pull out the two lines that matter: the weight-only warning, which should be absent on this card, and the KV cache size.

docker logs vllm 2>&1 | grep -E "native support|GPU KV cache size"

Swap in Qwen/Qwen3-32B-AWQ or RedHatAI/Qwen3-32B-NVFP4 and the command stays the same. On the L40S the NVFP4 build logs the FP4 version of the warning, because it runs weight-only below Blackwell.

To quantize a 16-bit checkpoint as it loads instead, pass --quantization fp8 with the BF16 model, Qwen/Qwen3-32B here. vLLM 0.30.0 materializes each layer just in time and converts its weights to per-tensor FP8. I prefer a pre-quantized checkpoint on QuantaCloud anyway, because stopping an instance deletes its disk and the next launch downloads the weights again: 34.3 GB for the FP8 checkpoint against 65.5 GB for the BF16 one.

The KV cache has its own setting. --kv-cache-dtype fp8 halves the cache on any of these GPUs, but without calibration it uses scales of 1.0, so check output quality on your own prompts. vLLM's docs recommend calibrating with llm-compressor, and --kv-cache-dtype-skip-layers sliding_window keeps sensitive sliding-window layers at full precision.

Two kinds of checkpoint fail to load in 0.30.0 for reasons that have nothing to do with the GPU. GPTQ files made with group activation ordering stop with GPTQ group activation ordering (desc_act=True) is no longer supported, so pick one made with desc_act=False or static ordering. bitsandbytes and GGUF files need their plugin installed into the image, for example with a two-line Dockerfile that starts FROM vllm/vllm-openai:v0.30.0 and runs uv pip install --system vllm-gguf-plugin.

How to choose on QuantaCloud GPUs#

The GPU you rent narrows the choice to one or two formats. Prices per GPU-hour:

Prices checked 6 Oct 2026, 02:18 UTC

On the RTX A6000 and the A100, use 4-bit AWQ or GPTQ for anything that does not fit in 16 bits. FP8 checkpoints load on these cards, but only as a memory saving. On the L40S, L40 and RTX 6000 Ada, choose per-channel FP8 checkpoints, the FP8-dynamic builds from llm-compressor, over block-scaled ones such as Qwen's own FP8 releases, which run weight-only below Hopper. On the H100 PCIe and H200 NVL, either FP8 flavour gets FP8 math, and NVFP4 saves memory but no more than a 4-bit AWQ checkpoint does. On the RTX PRO 6000, NVFP4 is the format that uses the card's FP4 Tensor Cores, and FP8 runs natively too.

vLLM quantization FAQ#

Do I need the --quantization flag?

No, not for a pre-quantized checkpoint. vLLM reads the method from the checkpoint's quantization_config. You need the flag only to quantize a 16-bit model while it loads, for example --quantization fp8, or to override detection.

Does FP8 work on the A100 or RTX A6000?

It loads, weight-only. Both are Ampere cards without FP8 Tensor Cores, so vLLM runs FP8 weights through its Marlin kernel with 16-bit math and logs a warning that says so. You get the memory saving and not the FP8 speed.

Is NVFP4 worth it on an H100?

Only for the memory. Hopper has FP8 Tensor Cores but no FP4 ones, so vLLM runs NVFP4 weight-only there, and a 4-bit AWQ or GPTQ checkpoint of the same model is slightly smaller. On the RTX PRO 6000, NVFP4 runs in FP4.

How do I quantize my own model?

vLLM's docs point to llm-compressor, which produces FP8, INT8, INT4 and NVFP4 checkpoints that vLLM loads directly, and they mark the older AutoAWQ library as deprecated. GPTQModel is the other tool the docs cover for GPTQ files.

Can I quantize the KV cache as well?

Yes. --kv-cache-dtype fp8 stores keys and values in 8 bits and halves the cache, independent of the weight format. Without a calibrated checkpoint it uses scales of 1.0, so compare outputs before you rely on it.

My rule is to match the format to the card before anything else: 4-bit AWQ or GPTQ on Ampere, per-channel FP8 on Ada, any FP8 on Hopper and NVFP4 on the RTX PRO 6000. Then size the GPU from the checkpoint's real file size plus the KV cache you need, and check the startup log for the weight-only warning on the first run. If a 32B model at FP8 is your target, a 48 GB L40S is the place to start for moderate contexts, and the GPU catalog has the larger cards for long ones.