llama-swap: serve several models from one GPU behind one endpoint

llama-swap is a small proxy that lets one GPU serve several models behind one OpenAI-compatible endpoint. It reads the model field of each request, starts the server configured for that model, stops the one that was running, and forwards the request. The server is usually llama.cpp's llama-server, but anything that speaks the OpenAI API works. On a 48 GB card that is the difference between choosing one model per VM and offering three by name. llama.cpp now has a router mode of its own, covered near the end of this page, and for a plain set of GGUF files it may be all you need.

What you will build#

The setup on this page is llama-swap v272 as a single binary on the VM, starting llama.cpp's official CUDA image once for each model. Two models that cannot share a 48 GB card take turns, and each unloads after 10 idle minutes. These are the memory figures (our calculation, weights plus a 16-bit KV cache, before llama.cpp's compute buffer):

A 48 GB card with ECC on reports 46,068 MiB, about 45 GiB. The first two models together need 54.7 GiB, so they have to swap. The two small ones need 26.9 GiB together, so they can stay loaded side by side, which is what groups are for. GGUF explained covers how to size a file against a card.

Launch an Ubuntu GPU VM#

Start with the Bare Metal template on a 48 GB RTX A6000: Ubuntu 22.04 with the NVIDIA driver and Docker.

Most single-GPU VMs are running in about 3 minutes (median). Connect as ubuntu (connecting over SSH), check the GPU, and pull the llama.cpp image once so the first model load does not wait for it:

ssh ubuntu@YOUR_INSTANCE_IP
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
docker pull ghcr.io/ggml-org/llama.cpp:server-cuda-b12110
docker run --rm --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda-b12110 --list-devices

server-cuda-b12110 is the image of llama.cpp v0.9.0, released on 2026-11-04, a CUDA 12.8 build of 2.6 GB. The last command should list your GPU. If Docker answers with permission denied or cannot find a GPU device driver, running Docker with NVIDIA GPUs has both fixes, and llama.cpp on a GPU explains the image and its flags.

Download the model files#

llama-swap does not download anything. It runs the command you give it, so fetch the GGUF files first. Both repositories are Apache-2.0 and need no token:

mkdir -p ~/models && cd ~/models
curl -L -O https://huggingface.co/Qwen/Qwen3-32B-GGUF/resolve/main/Qwen3-32B-Q8_0.gguf
curl -L -O https://huggingface.co/ggml-org/gpt-oss-20b-GGUF/resolve/main/gpt-oss-20b-MXFP4.gguf
ls -l ~/models

The files should be 34,817,718,912 and 12,109,566,624 bytes. The Hugging Face download guide shows a faster client for large files.

Install llama-swap#

The release binary is the simplest install: one statically linked Go program, no dependencies, 24 MB to download. Check it against the checksum the release publishes:

cd ~
curl -L -O https://github.com/mostlygeek/llama-swap/releases/download/v272/llama-swap_272_linux_amd64.tar.gz
echo "3f21b4c13f55ea5576badcd6cc1d114ccdcf74a5d56346d3ac1e777d28e0fdf1  llama-swap_272_linux_amd64.tar.gz" | sha256sum -c -
tar -xzf llama-swap_272_linux_amd64.tar.gz llama-swap
./llama-swap -version

v272 came out on 2026-11-10. The project also publishes Docker images, and these are the tags pinned to that release:

Both bundle llama-swap with llama-server, stable-diffusion.cpp, whisper.cpp and audio.cpp, built from source. llama-swap reads /etc/llama-swap/config/config.yaml there, listens on port 8080 inside the container, and each model's cmd starts the bundled llama-server directly. A CUDA 13 image needs an R580 or newer driver on the VM. I use the binary here for two reasons. It lets every model run the llama.cpp image I pinned, where a unified image carries whatever llama.cpp build it was made with. And llama.cpp's own Linux CUDA archives do not run on Ubuntu 22.04, because they need glibc 2.38 and this OS ships 2.35, so the official llama.cpp container is the quickest way to a current llama-server on this VM.

Write the config#

The config is one YAML file. Each entry under models is an ID that clients put in the model field, and a cmd that starts a server for it. ${PORT} is a free port llama-swap assigns, and ${MODEL_ID} is the entry's name. Save this as ~/llama-swap.yaml:

healthCheckTimeout: 600

apiKeys:
  - "${env.LLAMA_SWAP_KEY}"

models:
  "qwen3-32b":
    proxy: "http://127.0.0.1:${PORT}"
    cmd: |
      docker run --name ${MODEL_ID} --init --rm --gpus all
      -p 127.0.0.1:${PORT}:8080 -v /home/ubuntu/models:/models
      ghcr.io/ggml-org/llama.cpp:server-cuda-b12110
      -m /models/Qwen3-32B-Q8_0.gguf -ngl all -c 32768
    cmdStop: docker stop ${MODEL_ID}
    unloadTimeout: 30
    ttl: 600

  "gpt-oss-20b":
    proxy: "http://127.0.0.1:${PORT}"
    cmd: |
      docker run --name ${MODEL_ID} --init --rm --gpus all
      -p 127.0.0.1:${PORT}:8080 -v /home/ubuntu/models:/models
      ghcr.io/ggml-org/llama.cpp:server-cuda-b12110
      -m /models/gpt-oss-20b-MXFP4.gguf -ngl all -c 131072
    cmdStop: docker stop ${MODEL_ID}
    unloadTimeout: 30
    ttl: 600

Each setting does one job:

-ngl all and a fixed -c are the two llama-server flags I never leave out: together they make a model that does not fit fail with an error, instead of quietly moving layers to the CPU. Inside the container llama-server listens on port 8080 on every interface, and the 127.0.0.1: in -p keeps it off the VM's public address.

Start llama-swap and call it#

Create a key, validate the config and start the proxy on loopback. llama-swap listens on every interface by default (:8080), so on a VM with a public IP always pass -listen:

export LLAMA_SWAP_KEY=$(openssl rand -hex 32)
echo "$LLAMA_SWAP_KEY"

./llama-swap -config ~/llama-swap.yaml -validate
./llama-swap -config ~/llama-swap.yaml -listen 127.0.0.1:8080

It stays in the foreground and prints its proxy log, so run it inside tmux if you want it to survive your SSH session. From a second session, set the same key and list the models, then send a request to each:

export LLAMA_SWAP_KEY=PASTE_THE_KEY_YOU_PRINTED

curl -s http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LLAMA_SWAP_KEY"

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_SWAP_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen3-32b", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 1000}'

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_SWAP_KEY" -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss-20b", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 1000}'

The first request waits while llama-swap starts the qwen3-32b container and the model loads. The second names a different model, so llama-swap stops the first container, starts the other and then answers. Both models think before they reply, which is why max_tokens leaves room. To see what is loaded at any moment:

curl -s http://127.0.0.1:8080/running -H "Authorization: Bearer $LLAMA_SWAP_KEY"
docker ps --format '{{.Names}}'
nvidia-smi --query-gpu=memory.used --format=csv

curl -Ns http://127.0.0.1:8080/logs/stream with the same header follows the logs of llama-swap and the model servers, which is where a failed load explains itself. POST /api/models/unload unloads everything at once. There is also a web UI at /ui with the model list, logs and a playground, behind the same key. Reach it from your laptop through a tunnel, ssh -N -L 8080:127.0.0.1:8080 ubuntu@YOUR_INSTANCE_IP, as the SSH and VS Code guide describes.

Keep models loaded together with groups#

Out of the box llama-swap runs one model at a time. Groups change that: a group is a named set of models with three flags.

To keep the two small models loaded together and give the 32B model the card to itself, download the third file, add a third model entry, and add a routing block:

curl -L -o ~/models/Qwen3-8B-Q8_0.gguf https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q8_0.gguf

  "qwen3-8b":
    proxy: "http://127.0.0.1:${PORT}"
    cmd: |
      docker run --name ${MODEL_ID} --init --rm --gpus all
      -p 127.0.0.1:${PORT}:8080 -v /home/ubuntu/models:/models
      ghcr.io/ggml-org/llama.cpp:server-cuda-b12110
      -m /models/Qwen3-8B-Q8_0.gguf -ngl all -c 32768
    cmdStop: docker stop ${MODEL_ID}
    unloadTimeout: 30
    ttl: 600

routing:
  router:
    use: group
    settings:
      groups:
        big:
          swap: true
          exclusive: true
          members: [qwen3-32b]
        small:
          swap: false
          exclusive: true
          members: [gpt-oss-20b, qwen3-8b]

The model entry goes under models, indented like the other two, and routing goes at the top level. Restart llama-swap after the edit. A request for either small model now loads it without touching the other, 26.9 GiB for the pair. A request for qwen3-32b unloads both, and a request for a small model unloads the 32B one. A model can belong to only one group, and a model in no group lands in a default group that behaves like big.

Two more routers exist when groups cannot express what you need. use: matrix takes a list of model combinations that may run together, and use: capacity takes resource pools, such as megabytes per GPU, and lets any combination run that fits them. Older guides put groups at the top level of the file. v272 still reads that form, but not in the same file as a routing block.

Use it behind Open WebUI#

Open WebUI treats llama-swap as one OpenAI connection and shows every model ID in its model list. On the Bare Metal VM, run Open WebUI in host networking mode so that it can reach llama-swap on 127.0.0.1, and tell it to listen on loopback too:

docker run -d --name open-webui --network host \
  -e HOST=127.0.0.1 -e PORT=3000 \
  -e ENABLE_OLLAMA_API=False \
  -e OPENAI_API_BASE_URL=http://127.0.0.1:8080/v1 \
  -e OPENAI_API_KEY="$LLAMA_SWAP_KEY" \
  -e WEBUI_SECRET_KEY="$(openssl rand -hex 32)" \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:v0.13.3

Open a tunnel with ssh -N -L 3000:127.0.0.1:3000 ubuntu@YOUR_INSTANCE_IP and browse to http://127.0.0.1:3000. The first account you create becomes the admin. Open WebUI reads those connection variables only on its first start, and after that the connection lives under Settings, Admin, Connections, OpenAI. Leave the connection's provider type at its default: the llama.cpp type adds an eject button that calls llama-server's own unload endpoint, which is not llama-swap's.

Picking another model in the chat window triggers a swap, and the first reply waits for the load. Setting sendLoadingState: true at the top of the llama-swap config sends loading messages into the reply's reasoning field, so the chat shows that something is happening. Open WebUI with Ollama covers Open WebUI itself.

llama-server's own router mode#

llama.cpp now does a simpler version of this itself. Start llama-server without a model and it becomes a router: it lists the GGUF files it knows, loads the one a request names, and unloads the least recently used model when more than --models-max are loaded (4 by default). With the same files on the VM:

docker run -d --name llama-router --gpus all \
  -p 127.0.0.1:8081:8080 \
  -v ~/models:/models \
  -e LLAMA_API_KEY="$LLAMA_SWAP_KEY" \
  ghcr.io/ggml-org/llama.cpp:server-cuda-b12110 \
  --models-dir /models --models-max 1 -ngl all -c 32768

curl -s http://127.0.0.1:8081/models -H "Authorization: Bearer $LLAMA_SWAP_KEY"

A file in --models-dir is named after itself without the suffix, so the 32B model is Qwen3-32B-Q8_0 in the model field. Flags on the router's command line, here -ngl all -c 32768, are passed to every model. Per-model flags go in an INI file given with --models-preset, one section per model. Stop this container before you go back to llama-swap, since both would load models onto the same GPU.

My rule: use the router when everything is a GGUF file and one llama.cpp build suits every model, because it is one process less to run. Use llama-swap when you need to say exactly which models may share the GPU, when models need different builds or images, or when one of them is not llama.cpp at all. One caution from llama.cpp's own security policy: it lists the router among the server features it does not recommend for untrusted environments, so keep it on loopback like everything else here.

Before you stop the instance#

Stopping a QuantaCloud instance deletes its disk: the config, the binary, the pulled images and 55.6 GB of model files. Keep llama-swap.yaml and the download commands on your laptop, so the next launch is a copy and a wait. The first hour is charged at launch, and the unused seconds are refunded when you stop.

llama-swap FAQ#

Where is llama-swap on GitHub?

At github.com/mostlygeek/llama-swap, under the MIT licence. v272 was the latest release on 2026-11-12, published two days earlier, and the Linux binary is on its releases page.

Does llama-swap work with vLLM?

It works with any server that speaks the OpenAI API, since it only starts a command and proxies requests to it. Its README recommends running Python servers such as vLLM through Docker or Podman, so that they shut down cleanly when llama-swap stops them. vLLM vs llama.cpp explains when vLLM is worth the extra memory.

Can llama-swap run two models at the same time?

Yes, with a group whose swap is false, as in the groups section above. Without any routing config it runs exactly one model at a time.

Is it safe to expose llama-swap to the internet?

No. An API key protects the endpoints, but llama-swap starts processes on your machine, and its own docs say apiKeys is not a substitute for a firewall. Keep it on 127.0.0.1 and use an SSH tunnel.

Do I still need llama-swap now that llama-server has a router?

Not for the simple case. The router covers GGUF files served by one llama.cpp build with a limit on how many are loaded. llama-swap is still the tool for mixed servers, per-model images and explicit rules about which models share the card.

My rule for several models on one rented GPU: download the files first, pin the llama.cpp image, give every model -ngl all and a fixed -c, and let llama-swap decide what is loaded. Work out weights plus KV cache for each model before you group them, so that the models you keep together really fit. A 48 GB RTX A6000 at $0.48/GPU-hr holds one 32B model at Q8_0, or a 20B and an 8B side by side.