LLM quantization formats: GGUF, AWQ, GPTQ, FP8 and bitsandbytes

Quantization stores a model's weights in fewer bits, 8 or 4 instead of 16, with a scale for each small group of weights so the values can be stretched back to size. Memory falls almost in proportion: Qwen3-32B goes from 65.5 GB in BF16 to about 19 GB at 4 bits, and Llama 3.3 70B from 141 GB to about 40 GB. Quality falls far less, close to nothing at 8 bits and a little at 4, by the publishers' own measurements below. The format you choose depends on the engine you serve with: GGUF for llama.cpp and Ollama, AWQ, GPTQ, FP8 and NVFP4 for vLLM and SGLang, bitsandbytes for loading a model to fine-tune, and EXL3 for ExLlama.

What quantization does to memory#

The weights take parameters times bits divided by 8 bytes, and the files on Hugging Face come out a little larger than that, because most formats keep the embedding and output layers in 16 bits and store their scales. These are real checkpoints of a 32B and a 70B model, with the plain arithmetic for comparison:

The 4-bit files run 5 to 26% above the plain arithmetic (our calculation). For Qwen3-32B the BF16, AWQ and GGUF files are Qwen's own, and for Llama 3.3 70B the BF16 file is Meta's, the AWQ build casperhansen's and the GGUF bartowski's. Red Hat published the FP8 and GPTQ builds of both, Red Hat and NVIDIA the NVFP4 ones, Unsloth the NF4 ones, and turboderp and kaitchup the EXL3 ones.

The weights are only part of what a server needs. The KV cache holds the context of every running request and has to fit next to them, so a 4-bit 32B model fits a 48 GB card with room for long contexts, while a 4-bit 70B model on the same card has room for about 14,000 tokens. The vLLM quantization guide works out what is left on each QuantaCloud GPU, the KV cache guide has the formula, and the LLM VRAM calculator adds it all up for your model.

What it does to quality#

The honest answer comes from the publishers' own evaluations, and they agree that 8 bits is close to free and 4 bits costs a little:

Read each Red Hat figure against its own card. The NVFP4 card scored its BF16 baseline differently from the other three, so the percentages compare a format with the original model, not the formats with each other. The GPTQ card's recovery cell for that average reads 98.3%, which does not match its own numbers: 77.49 against 77.94 is 99.4% (our calculation). The llama.cpp column shows where quality really goes: Q2_K raises perplexity about 20 times as much as Q4_K_M (our calculation). Check your own prompts anyway, especially code and math.

The formats, by where they run#

Every GPU QuantaCloud rents is Ampere or newer, which clears the minimum each engine lists for loading these formats. The column that changes the decision is FP8 and FP4 math: the L40S, L40, RTX 6000 Ada, the Hopper cards and the RTX PRO 6000 compute in FP8, and of the on-demand GPUs only the RTX PRO 6000 Blackwell computes in FP4.

GGUF for llama.cpp and Ollama

GGUF is llama.cpp's single-file format, and Ollama runs the same files: a Modelfile with FROM /path/to/file.gguf imports one. Its types are named by size, from Q8_0 down to Q2_K, and Q4_K_M is the usual default. It is the format to pick when a model has to run partly on the CPU, or on one machine for a few users. vLLM loads it only through the separate vllm-gguf-plugin, which its docs call highly experimental. The GGUF guide explains the types and which file to download, and llama.cpp on a GPU runs them.

AWQ and GPTQ for vLLM and transformers

AWQ and GPTQ both store 4-bit integer weights with one scale per group of weights, 128 in every build in the tables above, and both learn their rounding from a small calibration set. They differ in how. GPTQ chooses its rounding from approximate second-order information about each layer, and its paper quantized 175-billion-parameter models in about four GPU hours with negligible loss at 3 or 4 bits. AWQ starts from the activations: its paper finds that protecting only 1% of the weights, the salient channels it identifies from activation statistics, greatly reduces the error, and it scales those channels up instead of keeping them in higher precision.

At serving time the difference mostly disappears. The files are the same size, 19.3 against 19.2 GB for Qwen3-32B, and vLLM 0.33.0 picks the 4-bit kernel for both through the same chooser. So I pick by who published the checkpoint and by tooling. AutoAWQ is deprecated, and vLLM's docs point to llm-compressor, which took over its functionality. GPTQ files come from GPTQModel or llm-compressor, and the older AutoGPTQ has not had a release since March 2024. One vLLM catch: GPTQ files made with group activation ordering (desc_act=True) no longer load, which the vLLM quantization guide covers with the other format details.

FP8 and NVFP4 for GPUs that compute in them

FP8 and NVFP4 quantize the activations as well as the weights, so a GPU with the matching Tensor Cores computes in them directly instead of unpacking to 16 bits first. FP8 math starts at compute capability 8.9 (Ada, Hopper and Blackwell), and FP4 math needs Blackwell. On older GPUs vLLM still loads both and runs them weight-only. FP8 is the format I reach for first on any GPU that computes in it, because the 70B card above puts it within a fraction of a percent of BF16. FP8 vs FP16 vs BF16 explains the number formats, and NVFP4 covers the 4-bit floats, including the MXFP4 format gpt-oss ships in.

bitsandbytes for loading and QLoRA

bitsandbytes quantizes a model while it loads, with no calibration data and no pre-quantized file. transformers hands it a BitsAndBytesConfig and gets 8-bit weights through LLM.int8() or 4-bit weights in NF4, the data type the QLoRA paper introduced for weights that follow a normal distribution. Nested quantization quantizes the scales a second time and saves about 0.4 bits per parameter. With transformers==5.22.2, bitsandbytes==0.50.2 and accelerate==1.15.0:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map="auto")
print(f"{model.get_memory_footprint() / 1e9:.1f} GB")

This is the loader for fine-tuning: QLoRA keeps the frozen base in NF4 and trains a small adapter on top, and the fine-tuning VRAM guide gives the memory by model size. Pre-quantized NF4 checkpoints, such as Unsloth's -bnb-4bit repos, save the download of the 16-bit weights. For serving, vLLM needs a separate plugin, vllm-bnb-plugin, so I serve AWQ, GPTQ or FP8 files instead.

EXL2 and EXL3 for ExLlama

ExLlama is a separate inference engine with its own formats, built for running local models on consumer GPUs, and its server is TabbyAPI, with an OpenAI-compatible API. EXL2, from ExLlamaV2, is based on GPTQ's method and mixes 2- to 8-bit levels inside one model to hit any average bitrate between 2 and 8. EXL3, from ExLlamaV3, is a simplified version of QTIP, a trellis-coded method from Cornell, and needs only the original model and a target bitrate. By the project's account, converting takes a couple of minutes for small models and up to a few hours for 70B and larger on one high-end consumer GPU.

EXL3 is where the work happens now: ExLlamaV3 1.6.6 came out on 2026-10-31, and ExLlamaV2's last release, 0.3.2, is from July 2025. Its files are the smallest at 4 bits in the memory table, and it goes lower: turboderp's 3.0-bit build of Qwen3-32B is 13.9 GB.

Quantize a model yourself#

Most popular models already have quantized builds on Hugging Face, so check before you make one. When you do need your own, each format has its tool:

The FP8 recipe from vLLM's documentation, here for Qwen3-8B. vLLM's docs recommend keeping llm-compressor and vLLM in separate environments:

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "Qwen/Qwen3-8B"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)
oneshot(model=model, recipe=QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]))
model.save_pretrained("Qwen3-8B-FP8-Dynamic")
tokenizer.save_pretrained("Qwen3-8B-FP8-Dynamic")

Stopping a QuantaCloud instance terminates it and deletes its disk, so push the result to the Hugging Face Hub or copy it off before you stop.

Choose by serving engine#

The engine narrows the choice to one or two formats:

Prices per GPU-hour for the GPUs these formats are matched to:

Prices checked 14 Nov 2026, 09:11 UTC

LLM quantization FAQ#

What is quantization in an LLM?

It is storing the model's weights in fewer bits than they were trained in, usually 8 or 4 instead of 16, plus a scale for each group of weights. The model needs half or a quarter of the memory, usually generates faster because each token reads fewer bytes, and loses a little accuracy, mostly at 4 bits and below.

AWQ or GPTQ: which is better?

Neither wins in general. Both are 4-bit weight-only formats with group scales and near-identical file sizes, and vLLM runs both through the same kernel chooser. Take the one with a checkpoint from a publisher you trust, and avoid GPTQ files with desc_act=True on vLLM.

Does quantization make a model faster?

Usually, when generating. Producing tokens one at a time is limited by memory bandwidth, so smaller weights mean less to read per token. vLLM's documentation puts FP8 at up to 1.6 times the throughput. 4-bit weight-only formats compute in 16 bits after unpacking, so their gain is mostly the memory and bandwidth.

Does quantizing the weights shrink the KV cache?

No. The cache has its own setting: --kv-cache-dtype fp8 in vLLM, and -ctk q8_0 -ctv q8_0 in llama.cpp. The KV cache guide shows how much each saves.

What is EXL3?

It is the quantization format of ExLlamaV3, based on the QTIP method, that can hit any average bitrate. You run it with ExLlamaV3 or its server, TabbyAPI, not with vLLM or llama.cpp.

My rule: choose the engine first and the format second. On vLLM, start with FP8 on any GPU that computes in it and drop to AWQ or GPTQ only when the 8-bit file leaves too little room for context. On llama.cpp or Ollama, start with Q4_K_M and move to Q8_0 when it fits. Use bitsandbytes to fine-tune, not to serve. A 48 GB card such as the L40S holds a 32B model at FP8 for moderate contexts, and self-hosted LLM inference puts the engines side by side.