GGUF models in ComfyUI: when to use them, and when not to

The rule this page exists for is short: on a 48 GB or 96 GB card, run the FP8, INT8 or 16-bit safetensors file of a model, and reach for GGUF only when the model does not fit. ComfyUI-GGUF, city96's custom node, loads GGUF files into ComfyUI's native workflows, and it was built for low-end GPUs: its README says transformer models such as FLUX lose less to quantization than older UNet models, so they can run at low bits per weight on small cards. The cost is that it expands every weight to 16-bit on the fly before computing with it. Every QuantaCloud GPU has at least 48 GB, and the 8-bit diffusion model of every current model in our ComfyUI guides fits one, so the case for GGUF here is narrow: FLUX.2 [dev] on a 48 GB card, where a 4-bit file lets the model and its text encoder stay on the card together.

What ComfyUI-GGUF is, and who maintains it#

ComfyUI has no GGUF loader of its own, so GGUF files need this custom node, and it is one person's project.

The README still describes the project as "very much WIP", and the ten months without a commit are worth knowing before you build a production workflow on it. GGUF itself is the file format from llama.cpp: what GGUF is covers the format and its quant types, such as Q8_0 and Q4_K_M.

Install it#

There are three routes, and each ends with a restart of ComfyUI, because nodes load at startup. On QuantaCloud, start from the ComfyUI template on an RTX 6000 Ada, the 48 GB card with FP8 compute that the comparisons below run on.

The manual route, on your own install with its virtual environment active:

cd ~/comfy/ComfyUI
git clone https://github.com/city96/ComfyUI-GGUF custom_nodes/ComfyUI-GGUF
pip install -r custom_nodes/ComfyUI-GGUF/requirements.txt

On QuantaCloud's ComfyUI template, ComfyUI runs in a Docker container, so installs happen inside it and disappear when you stop the instance. A registry install from the Manager dialog is a middle-risk action that the Manager allows even when ComfyUI listens beyond localhost, as it does in a container. Installing ComfyUI-Manager and custom nodes covers the security levels and how to rebuild a node set on each launch.

The loader nodes and where the files go#

The node adds six loaders under the bootleg category. Use them in place of the stock loaders and leave the rest of the workflow alone.

The README says to put model files in models/unet. The node's code registers .gguf files in both models/unet and models/diffusion_models, so either folder works. The CLIP loaders take GGUF text encoders such as city96's T5-XXL builds, whose Q8_0 file is 5.06 GB against 9.79 GB for the 16-bit one. One warning from the README: the "Force/Set CLIP Device" node that appears in some shared GGUF workflows is not part of this pack, and it is not needed on a one-GPU machine.

GGUF against FP8 and 16-bit, in memory#

The comparison people want is file size, so here it is for three current models, from Hugging Face's byte counts.

The first thing the table shows is that Q8_0 saves nothing over the native 8-bit files: it is within a gigabyte or two of fp8mixed and INT8, and larger for two of the three models. The saving only starts at Q6_K and below, and those quants change the weights more. On a card that holds the 8-bit file, GGUF buys no memory you can use.

What the node says about speed#

The honest answer is that ComfyUI-GGUF's README makes no speed claim at all. What it documents is memory: running models "in much lower bits per weight" on low-end GPUs. Its code shows where the speed goes: the operations it adds to ComfyUI "Dequantize weights on the fly before doing the compute", so every layer is expanded to 16-bit each time it runs.

Native files are handled differently. ComfyUI computes fp8 directly on GPUs with compute capability 8.9 or higher, the Ada, Hopper and Blackwell cards, which on QuantaCloud include the RTX 6000 Ada, L40, L40S and RTX PRO 6000 Blackwell, and it computes INT8 natively on every NVIDIA GPU. On those cards the native 8-bit file skips the expansion that GGUF adds. On the Ampere RTX A6000, which has no FP8, ComfyUI also expands fp8 layers to 16-bit as it computes, so the INT8 file is the native one to pick there. FP8 vs FP16 vs BF16 explains the formats.

When GGUF earns its place on a 48 GB card#

The rule I follow is to add up the files a workflow loads and compare the total with the card's budget, 41.39 GiB on a 48 GB card with ECC on and 87.95 GiB on the 96 GB RTX PRO 6000 Blackwell (our calculation, 0.92 of what each card reports). If the 8-bit or 16-bit set fits, use it.

That FLUX.2 [dev] row is the narrow case for GGUF on QuantaCloud. The fp8mixed model fits a 48 GB card on its own, so ComfyUI runs it by swapping the text encoder out before each sampling run, and back in when the prompt changes. A Q4_K_S model lets both stay on the card, at the price of 4-bit weights and the dequantization on every step. The alternatives need no quant: the 96 GB RTX PRO 6000 Blackwell holds the whole fp8mixed set with room to spare, and on two 48 GB cards ComfyUI's Select CLIP Device node keeps the text encoder on the second card, as ComfyUI with more than one GPU shows. Which is better depends on whether the Q4 quality is good enough for your images, so compare a few seeds before you commit. FLUX.2 [dev] is under the FLUX Non-Commercial License whichever file you load, and city96's GGUF repository carries the same non-commercial licence tag.

Download a GGUF model#

The download is the same as for any other model file. On the template, find the container and the models folder first:

C=$(sudo docker ps -q --filter publish=8188)
MODELS=$(curl -s http://127.0.0.1:8188/internal/folder_paths | python3 -c 'import json, os, sys; print(os.path.dirname(json.load(sys.stdin)["checkpoints"][0]))')
echo "$MODELS"

Then save the list as models.txt. These lines set up FLUX.2 [dev] for a 48 GB card:

# FLUX.2 [dev], non-commercial: Q4_K_S GGUF model, fp8 text encoder and VAE, 37.67 GB
unet             https://huggingface.co/city96/FLUX.2-dev-gguf/resolve/main/flux2-dev-Q4_K_S.gguf
text_encoders    https://huggingface.co/Comfy-Org/Flux-2-Dev/resolve/main/split_files/text_encoders/mistral_3_small_flux2_fp8.safetensors
vae              https://huggingface.co/Comfy-Org/Flux-2-Dev/resolve/main/split_files/vae/flux2-vae.safetensors

The get-models.sh script in loading models on a new ComfyUI instance and the FLUX.2 guide reads this format. Run it inside the container with sudo docker exec -e MODELS="$MODELS" "$C" bash /tmp/get-models.sh /tmp/models.txt after copying both files in with sudo docker cp. Then press r in ComfyUI, open the FLUX.2 [dev] template, and swap its Load Diffusion Model node for Unet Loader (GGUF) pointed at the file. Stopping a QuantaCloud instance deletes its disk, so this download, and the node install, happen on every launch.

LoRAs on GGUF models#

LoRAs work through the stock Load LoRA nodes, and the README calls LoRA loading "experimental but it should work". The code explains the trade: the node applies each LoRA to the weights after it dequantizes them, on every forward pass, so a LoRA adds work to every step instead of being merged once. The Advanced loader's patch_dtype sets the precision of that merge, and the code itself warns that changing it "may degrade image quality", so leave it at default. If you train or stack several LoRAs, that is one more reason to stay on the native 8-bit or 16-bit files whenever they fit.

GGUF in ComfyUI questions#

Does ComfyUI support GGUF natively?

No. ComfyUI v0.45 has no GGUF loader, and GGUF files need the ComfyUI-GGUF custom node. ComfyUI's native quantized formats include fp8, INT8, NVFP4 and MXFP8 safetensors files.

Is GGUF slower than FP8 in ComfyUI?

The node dequantizes every weight to 16-bit before computing, while ComfyUI computes native fp8 directly on Ada, Hopper and Blackwell cards and INT8 on every NVIDIA GPU. The README makes no speed claim, so time both on your own card, as the measured lines above do.

Where do GGUF files go in ComfyUI?

Diffusion models go in models/unet or models/diffusion_models, and GGUF text encoders in models/text_encoders or models/clip. Load them with Unet Loader (GGUF) and the CLIPLoader (GGUF) nodes.

Which GGUF quant should I pick?

On a 48 GB or 96 GB card, usually none: the native 8-bit file is about the size of Q8_0. When a workflow's files do not fit the card at once, pick the largest quant that makes them fit, such as Q4_K_S for FLUX.2 [dev] with its text encoder on a 48 GB card.

My decision rule for GGUF in ComfyUI: native fp8 on an Ada or Blackwell card, INT8 on an RTX A6000, 16-bit when it fits, and GGUF only when a workflow's 8-bit files will not fit the card at once, which on QuantaCloud comes down to FLUX.2 [dev] with its text encoder on a 48 GB card. When the GGUF quality is not good enough, the 96 GB RTX PRO 6000 Blackwell is the answer. ComfyUI GPU requirements has the file totals for the other image and video models, and the ComfyUI page lists every GPU with the template and its live price.