GGUF explained: the format, the quantization types and which file to download

GGUF is the model file format of llama.cpp: one binary file that carries a model's weights together with its architecture settings, its tokenizer, its chat template and the quantization type of every tensor. That is why a single .gguf download runs in llama.cpp, in Ollama and in the tools built on them with no config files beside it. The file name tells you what you are getting. Qwen3-32B-Q4_K_M.gguf is Qwen3-32B at about 4.9 bits per weight: 19.76 GB, against 65.52 GB for the 16-bit original. On a GPU server two questions decide the download: which quantization to take, and whether the file plus its context fits the card.

What is inside a GGUF file#

A GGUF file has four parts in a fixed order: a header, the metadata, a table of tensors and the tensor data. The header starts with the four bytes GGUF, followed by the format version (3 today), the number of tensors and the number of metadata entries. The metadata is a list of typed key-value pairs. general.architecture names the model family, general.file_type records the quantization most tensors use, the tokenizer.ggml.* keys hold the vocabulary, and tokenizer.chat_template holds the Jinja template that turns a list of chat messages into a prompt. Then comes one entry per tensor with its name, shape, type and offset, and last the weights themselves, aligned to 32 bytes by default so a loader can memory-map the file instead of copying it.

Two things follow from that layout. The file describes itself, so the loader needs nothing else. And every tensor carries its own type, so one file can mix precisions, which is how a Q4_K_M file keeps some tensors at 5 or 6 bits.

The spec also sets a naming convention: base name, size, fine-tune, version, encoding and an optional shard number, joined by hyphens. Large models are split into shards named like -00001-of-00003.gguf, and a file whose name starts with mmproj- is the vision or audio projector that loads beside a base model.

You can read the metadata without loading the model. On Hugging Face, the Hub shows a GGUF file's metadata and tensor list in the browser. On a server, gguf-dump from the gguf Python package, which is maintained in the llama.cpp repository, prints the same thing. With uv installed as in the Hugging Face download guide:

uv tool install gguf
gguf-dump --no-tensors ~/models/Qwen3-8B-Q4_K_M.gguf

What GGUF stands for, and GGUF vs GGML#

The specification never spells the name out. It defines GGUF as the successor to three older file formats, GGML, GGMF and GGJT, and GGML is also the name of the tensor library underneath llama.cpp. Expansions of the acronym circulate, but none of them is in the spec. GGUF was merged into llama.cpp on 2023-08-21.

The difference from GGML is the metadata. The old formats stored a model's hyperparameters as an untyped list, so a file did not say which architecture it held, and adding one field broke every existing file. GGUF replaced the list with named, typed keys, so new information can be added without breaking old readers. The format has changed twice since: version 2 widened the counts to 64 bits, and version 3 added big-endian files.

An old GGML .bin file does not load in current llama.cpp. The repository still carries convert_llama_ggml_to_gguf.py, but I would download a current GGUF of the same model instead of converting a file from 2023.

What reads GGUF, and what does not#

GGUF is native to llama.cpp and to everything built on it, and a guest elsewhere. This is the state on 2026-11-12:

The practical reading: if the server is llama.cpp or Ollama, take GGUF. If it is vLLM, take a safetensors checkpoint, as vLLM vs llama.cpp explains, and see llama.cpp vs Ollama for how those two relate.

The quantization types#

The quantization type is the part of the file name after the model: Q4_K_M, Q8_0, F16. llama.cpp publishes two sets of figures for them. Its quantize README gives the bits per weight and the file size for Llama 3.1 8B, and its llama-quantize tool lists how much each type raises perplexity on Llama-3-8B, where a smaller increase means closer to the 16-bit model:

The names decode like this. Q is for quantized and the digit is the nominal number of bits. K marks the k-quant family, which improved on the older Q4_0 and Q5_0 types: at almost the same size, Q4_0 costs +0.4685 where Q4_K_S costs +0.2689. S, M and L are the small, medium and large mixes, where the larger ones keep more tensors at a higher-precision type. That is why Q4_K_M works out at 4.89 bits and not 4. The IQ types are a newer family for low bit rates, from 2.0 bits per weight for IQ1_S to 4.68 for IQ4_NL, and F16 and BF16 are unquantized 16-bit files.

The curve has a knee. Q8_0 is almost indistinguishable from 16-bit on this measure, Q4_K_M gives up a little, and Q2_K gives up about 20 times as much as Q4_K_M (our calculation from the tool's figures). The figures come from 8B Llama models, so read them as an ordering, not as a promise for every model.

GGUF also holds formats that are not llama.cpp's own. gpt-oss ships in MXFP4, which is why its GGUF needs no choice: gpt-oss-20b-MXFP4.gguf is 12.11 GB. llama.cpp has a file type for NVFP4, and v0.9.0 added one for FP8, so the converter can now keep a checkpoint's FP8 weights instead of expanding them to 16 bits.

Which GGUF file to download for your GPU#

The rule I follow is to take the largest quantization whose file plus KV cache fits the GPU with room for llama.cpp's compute buffer, starting from Q4_K_M and going below it only when nothing else fits. The file size is the weight memory, near enough. The KV cache is 2 x layers x KV heads x head size x 2 bytes for every token of context: 256 KiB per token for Qwen3-32B and 320 KiB for Llama 3.3 70B. When llama.cpp fits a model automatically, it also keeps 1,024 MiB of each GPU free.

These are the largest files that fit, worked against the memory each card reports in nvidia-smi minus that margin (our calculation, with ECC on for the 48 GB cards):

The left-over column still has to hold the compute buffer, so the rows with 2 GiB to spare are the tight ones. The 48 GB rows show the real trade for a 70B model: Q4_K_M with a short context, or Q3_K_M with a usable one, at a perplexity cost almost four times higher on llama.cpp's 8B figures (+0.6569 against +0.1754). The 80 GB card removes the trade, and the 96 GB card is the first that runs the 70B model at Q8_0 with a long context. An H100 PCIe reports 0.35 GiB less than the A100, which changes none of these rows.

The q8_0 cache in the last row is -ctk q8_0 -ctv q8_0, which stores the KV cache at about 53% of its 16-bit size. Running llama.cpp on a GPU has the commands and the memory table for more models, and the KV cache guide explains the formula.

GGUF vs safetensors, AWQ and FP8#

These four are not the same kind of thing. GGUF and safetensors are file formats. AWQ and FP8 are ways of quantizing weights, and both are stored as safetensors.

The sizes line up: a 4-bit GGUF and a 4-bit AWQ checkpoint of the same model are within a few gigabytes of each other, and so are Q8_0 and FP8. What differs is the server each one feeds. GGUF goes with llama.cpp, which runs one user or a few and can keep part of a model in system RAM. AWQ and FP8 go with vLLM, which batches many requests. On QuantaCloud the oldest cards, the A100 and RTX A6000, are compute capability 8.0 and 8.6, so FP8 math is for the Ada, Hopper and Blackwell cards. LLM quantization formats compares the methods themselves, and vLLM quantization covers the vLLM side.

Convert safetensors to GGUF and quantize it#

Converting is two steps: turn the Hugging Face checkpoint into a 16-bit GGUF with convert_hf_to_gguf.py, then quantize that file with llama-quantize. Both tools are in llama.cpp's full Docker image, so on a QuantaCloud Bare Metal VM, Ubuntu 22.04 with Docker, nothing has to be built. Neither step uses the GPU. The quantizer loads the whole model into memory, so the VM needs more RAM than the 16-bit file is large.

1. Download the checkpoint. Qwen/Qwen3-8B is 16.38 GB of safetensors, Apache-2.0 and not gated. The Hugging Face download guide covers installing hf.

   TerminalCopy

   hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B
   

2. Convert it to a 16-bit GGUF. The image's entrypoint passes everything after --convert to convert_hf_to_gguf.py.

   TerminalCopy

   docker run --rm -v ~/models:/models ghcr.io/ggml-org/llama.cpp:full-b12110 \
     --convert /models/Qwen3-8B --outfile /models/Qwen3-8B-F16.gguf --outtype f16
   

3. Quantize it. The arguments are the input file, the output file and the type.

   TerminalCopy

   docker run --rm -v ~/models:/models ghcr.io/ggml-org/llama.cpp:full-b12110 \
     --quantize /models/Qwen3-8B-F16.gguf /models/Qwen3-8B-Q4_K_M.gguf Q4_K_M
   

full-b12110 is the image of llama.cpp v0.9.0, released on 2026-11-04. llama.cpp publishes official releases as version tags and several nightly builds a day as b numbers, and the Docker images carry the build number. --outtype also takes bf16, f32, q8_0 and auto, which is the default. To cut the quality loss of the low-bit types, llama-quantize accepts an importance matrix with --imatrix.

If you only want the file, Hugging Face's GGUF-my-repo space runs the same two steps for you, and it syncs with llama.cpp every 6 hours. Stopping a QuantaCloud instance deletes its disk, so copy the finished GGUF off the VM or push it to a Hugging Face repo before you stop.

Where trustworthy GGUF files come from#

My order of preference is the model vendor's own GGUF repository, then a converter whose name you know, then your own conversion. Qwen publishes Qwen/Qwen3-32B-GGUF itself under Apache-2.0, and the llama.cpp maintainers publish files such as gpt-oss under the ggml-org organization. Community converters such as bartowski publish files for many other models, and Hugging Face lists all of them under the library=gguf filter.

Four checks take a minute. Read the licence on the repository, because a GGUF inherits the licence of the model it was made from: the Llama 3.3 files carry Meta's licence whoever converted them. Look at the date, since a file converted before a tokenizer or template fix keeps the old behaviour. Qwen's 32B GGUF repository was last changed on 2025-05-09. Open the file in the Hub's viewer and read tokenizer.chat_template, because the template is part of the file and decides how your messages reach the model. And compare the SHA-256 of what you downloaded with the one the Hub shows:

cd ~/models
curl -L -O https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf
echo "d98cdcbd03e17ce47681435b5150e34c1417f50b5c0019dd560e4882c5745785  Qwen3-8B-Q4_K_M.gguf" | sha256sum -c -

llama.cpp's own security policy asks for the same two habits: confirm the hash of what you download, and run models from unknown sources in an isolated environment. A rented VM that you delete afterwards is a reasonable place for that.

GGUF FAQ#

Is GGUF only for llama.cpp?

It is llama.cpp's format, and the tools that read it best are built on llama.cpp: Ollama, llama-cpp-python, LM Studio and GPT4All. Transformers can open a GGUF file, but for most architectures it converts the weights back to full precision, which gives up the memory saving.

Can vLLM run GGUF files?

Only through vllm-gguf-plugin, and vLLM 0.33.0's docs call that support highly experimental and under-optimized. For vLLM, download an AWQ or FP8 safetensors checkpoint of the same model instead.

Which GGUF quantization should I download?

Q4_K_M first. It is what llama-server -hf downloads by default and what Ollama's default tags such as qwen3:32b contain, and it costs little on llama.cpp's own perplexity figures. Move up to Q5_K_M, Q6_K or Q8_0 when the GPU has room after the KV cache, and go below Q4 only when nothing else fits.

Can I quantize a GGUF file that is already quantized?

llama-quantize refuses unless you pass --allow-requantize, and its README warns that this can severely reduce quality compared with quantizing from 16 or 32 bits. Start from the F16 file.

My rule for GGUF: take the vendor's file when there is one, take Q4_K_M unless the card has room for more, and check the fit as file size plus KV cache before you download 40 GB. A 48 GB RTX A6000 at $0.48/GPU-hr runs a 32B model at Q8_0, and a 70B model at Q4_K_M with a short context.