Self-hosting an LLM: the model, the GPU, the server and the real cost

Self-hosting an LLM takes five decisions, in this order: a model that does the job, a GPU with the memory to hold it, a server that matches how it will be called, a lock on its port, and a front end if people rather than programs will use it. On QuantaCloud all five run on a GPU VM you rent by the hour, the Bare Metal template with Ubuntu 22.04, the NVIDIA driver and Docker, and the software on it is yours to run. The cost answer is shorter than most people expect: per token, a hosted API is cheaper until your GPU is busy for most of the hours you pay for, so the case for a self-hosted AI server rests on control as much as on price. Self-hosted LLM inference is the reference for sizing and serving stacks, and each step below links to the guide that covers it.

The path in one table#

These are my defaults for each step, and the guide that covers each one in full:

Pick the model by its job, then by its memory#

The rule I follow is to choose the model for the job first and check the fit second, because a model that fits and cannot do the work is the expensive mistake. Read the licence before the benchmark. gpt-oss, Qwen3.8-27B and Gemma 4 are Apache-2.0. Some 2026 releases are not: Qwen3.8-Flash-Next needs a separate licence from Qwen for commercial use by any company that runs a model-as-a-service business, and Kimi K3 needs a separate agreement once such a business passes 20 million US dollars of revenue in any 12 months. Open-weight LLMs in late 2026 sorts the current models by the memory they need, with their licences.

Memory is two numbers added together. The weights take about the download size of the build you run. The KV cache holds every token of every conversation in progress, so it grows with the context and with the number of requests at once, and its size per token comes from the model's attention layout: 24 KiB for gpt-oss-20b, 64 KiB for Qwen3.8-27B and 320 KiB for Llama 3.3 70B at 16-bit precision (our calculation from each model's config). One full 131,072-token conversation costs Llama 3.3 70B 40 GiB of cache, as much as its 4-bit weights. The KV cache guide has the formula, and the LLM VRAM calculator adds weights and cache for your context and your number of users.

Pick the GPU that holds it#

The GPU to rent is the smallest one where the weights and the cache for your context fit with room to spare. With the 4-bit builds in Ollama's library it maps like this (our calculation from the download sizes and the memory each card reports):

Prices checked 14 Nov 2026, 06:56 UTC

The catalog moves, and four- and eight-GPU configurations come and go. When a model needs one of those, take it when the catalog lists it, or use the built-to-order path at the end of this page. On a 48 GB card I set the context to 32,768 tokens rather than trusting the default, which can be larger than the card: Ollama context length explains why.

Pick the server by how it will be called#

The rule I follow: Ollama when people chat with the model, vLLM when programs send requests that overlap, and llama.cpp when the build that fits is a GGUF file you chose yourself.

Ollama is the shortest path from an empty VM to a working model: one install command, a library of ready builds and a context picked for you. It answers one request at a time per model unless you raise OLLAMA_NUM_PARALLEL, and each extra slot costs a full context of KV cache, which is fine for one person or a small team in a chat window. Ollama in Docker and Ollama environment variables cover running it on a GPU server.

vLLM batches requests that arrive at the same time and hands out KV cache in blocks as each request grows, so it keeps up when an application or a room full of users sends requests at once. vLLM with Docker is the launch, and vLLM vs Ollama puts the two side by side.

llama.cpp runs GGUF files from 1.5-bit to 8-bit, and it can keep part of a model on the CPU when the model is larger than GPU memory. Ollama 0.40.16 already serves its GGUF models through llama.cpp's own server, so the difference between the two is packaging: Ollama adds a model library, a scheduler and its defaults, and llama.cpp gives you every flag. llama.cpp on a GPU covers it.

Lock the port before anything else#

The safest setup has the server listening on 127.0.0.1 and you reaching it through an SSH tunnel, so the only open door is SSH on port 22. Ollama listens on 127.0.0.1:11434 by default, and its local API has no authentication at all. llama.cpp's server listens on 127.0.0.1 and checks a key only when you start it with --api-key. Its default port has been 9931 since October 2026, while its official Docker image still listens on 8080, the port older guides show. vLLM is the one to watch. In release 0.33.0 it listens on every interface unless you pass --host, and its --api-key covers only the /v1, /v2, /inference and /cohere routes, so /invocations answers anyone. Docker adds its own trap: a port published with -p 8000:8000 is open to the world and skips ufw, so publish it as 127.0.0.1:8000:8000.

One command on your laptop opens the tunnel, here to Ollama's port:

ssh -N -L 11434:127.0.0.1:11434 -i ~/.ssh/your_key ubuntu@<instance-ip>

When an application on another machine needs the API, put a reverse proxy with TLS and a token in front of the server. The Ollama API guide and the vLLM Docker guide each have a Caddy config for it, and a LiteLLM proxy gives every app or person a key and a budget of its own. Connecting over SSH covers the keys for the tunnel.

Add a front end if people will use it#

Open WebUI is the chat interface I would put in front of either server. It talks to Ollama directly and to vLLM through an OpenAI-compatible connection, and it keeps its own accounts: the first account on a fresh install becomes the administrator, sign-up then switches off, and anyone you let in later waits as pending until you approve them. QuantaCloud's Open WebUI + Ollama template starts both at launch, behind your QuantaCloud login, and suits one person, because the app address opens only for the account that launched it. For a team, run Open WebUI yourself with Docker Compose on Bare Metal, behind your own HTTPS proxy. Open WebUI with Ollama has both paths, and Open WebUI on QuantaCloud describes the template.

What it costs, against a per-token API#

The honest comparison is hours against tokens. A QuantaCloud GPU bills by the hour from launch to stop: the first hour is charged at launch, each further hour when the previous one is used up, and the unused seconds of the current hour are refunded when you stop. An API bills by the token, whether you send ten requests a day or a million. Today's prices:

A worked example at the prices on 2026-11-12, when one RTX A6000 cost $0.48 and one RTX PRO 6000 $2.39 per GPU-hour, for three ways of running the GPU (our calculation: hours x price):

On the same day, OpenAI's API pricing page listed two of the four flagship models at the top of the page, gpt-6-luna at $0.10 per million input tokens and $0.50 per million output tokens, and gpt-6.1-terra at $1.00 and $5.00, at standard speed for prompts of up to 272K input tokens. Take a chat request of 2,000 input tokens and 500 output tokens: it costs $0.00045 on gpt-6-luna and $0.0045 on gpt-6.1-terra. Divide the GPU's month by that and you get the number of requests a month at which the API costs the same (our calculation):

Below those numbers the API is cheaper, and above them the GPU is, as long as it can serve that many requests in the hours it runs. The hours cancel out, which gives a simpler test: the A6000 pays for itself against gpt-6.1-terra when it averages more than 14.8 output tokens a second over the hours it runs, and against gpt-6-luna above 148 (our calculation: $0.48 an hour divided by 3,600 seconds and by the API's cost per output token, with each request's input counted in). For the RTX PRO 6000 the same lines sit at 74 and 738.

Two things the comparison leaves out cut both ways. It compares price, not quality: whether gpt-oss-20b on an A6000 answers your prompts as well as gpt-6.1-terra is a test only you can run. And the API's Batch tier halves both prices for work that can wait, while a GPU that idles between requests still bills for the hour. So the GPU wins on cost when it is busy, and on control at any volume: the model and its version are yours, the prompts are processed on your VM, and no one else's rate limit applies. The GPU cost calculator works out the month for your own hours at live prices.

Run it on a VM whose disk is deleted on stop#

Stopping a QuantaCloud instance terminates it and deletes its disk: the server, the models and everything you configured go with it, and there are no volumes or snapshots. So treat the VM as disposable and keep the setup in a script on your laptop. This one installs a pinned Ollama with the official script, sets the server defaults and pulls two models:

#!/usr/bin/env bash
# setup.sh: run once on a fresh Bare Metal instance, as ubuntu
set -euo pipefail
sudo apt-get update && sudo apt-get install -y zstd
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.16 sh
sudo mkdir -p /etc/systemd/system/ollama.service.d
printf '[Service]\nEnvironment="OLLAMA_CONTEXT_LENGTH=32768"\nEnvironment="OLLAMA_KEEP_ALIVE=-1"\nEnvironment="OLLAMA_NO_CLOUD=1"\n' \
  | sudo tee /etc/systemd/system/ollama.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart ollama
until curl -sf http://127.0.0.1:11434/api/version > /dev/null; do sleep 2; done
for m in gpt-oss:20b qwen3.8:27b; do ollama pull "$m"; done

Copy it to a new instance and run it from your laptop:

scp -i ~/.ssh/your_key setup.sh ubuntu@<instance-ip>:
ssh -i ~/.ssh/your_key ubuntu@<instance-ip> bash setup.sh

The install script needs zstd because Ollama's Linux packages are zstd archives, and it sets Ollama up as a systemd service on 127.0.0.1:11434. The override file is where systemd keeps the server's variables, and Ollama environment variables explains the three set here. For vLLM, the same kind of script holds the docker run line from the vLLM Docker guide. Most single-GPU VMs are running in about 3 minutes (median), and the downloads come on top, every time: 14 GB for gpt-oss:20b and 18 GB for qwen3.8:27b. Downloading Hugging Face models fast shortens the wait for vLLM's weights.

Two billing rules matter for a server people depend on. If your balance cannot cover the next hour, QuantaCloud terminates the instance and deletes its disk, so turn on auto top-up. And an idle GPU still bills: stopping idle GPUs with the API shows how to stop one automatically, and the API reference covers launching Bare Metal instances from a script.

When dedicated capacity makes sense#

Reserve when the server has to run around the clock for months, when the model needs more memory than one VM holds, or when you need hardware nobody else shares. Reserved capacity is built to order: we order and build the hardware to your spec, from one dedicated GPU server to a GPU cluster, and the configuration, lead time and terms come back in writing before you commit. Reserved vs on-demand has the break-even, and on-premise LLM covers what owning the hardware takes and who sees your data in each case.

Self-hosting an LLM FAQ#

Is it cheaper to self-host an LLM than to use an API?

Only when the GPU is busy. At the 2026-11-12 prices, and for chat requests of 2,000 input and 500 output tokens, an RTX A6000 costs less than gpt-6.1-terra once it averages about 15 output tokens a second over the hours it runs, and less than gpt-6-luna above about 148 (our calculation, worked through above). Light or bursty use is cheaper on the API.

What GPU do I need to host my own LLM?

A 48 GB card such as the RTX A6000 for models up to about 35B parameters at 4-bit, and 80 to 96 GB for a 70B model or gpt-oss-120b. The KV cache for your context comes on top of the weights, so size for both.

Can I run my own LLM without a GPU?

Yes. Ollama and llama.cpp both run on the CPU, much more slowly, and ollama ps then reads 100% CPU. On a GPU server it should read 100% GPU, and Ollama not using the GPU explains a CPU share.

Do my prompts leave the server?

Not to a model provider. Prompts go from your client to your VM and are processed there, and the weights come from Hugging Face or the Ollama library. The exception is Ollama's cloud models, the tags that end in cloud, which run on Ollama's servers: OLLAMA_NO_CLOUD=1 turns those features off.

How long does setup take?

Most single-GPU VMs are running in about 3 minutes (median). The setup script then installs Ollama and pulls the models, so the rest is as long as the downloads take: 14 GB for gpt-oss:20b.

My rule for self-hosting an LLM: start with an Apache-2.0 model of up to about 35B parameters on a 48 GB RTX A6000, serve it with Ollama behind an SSH tunnel, and keep the setup in a script. Move to vLLM when requests start to overlap, to a bigger card when the cache for your context does not fit, and to a dedicated server only when the GPU is busy around the clock. Before you call it cheaper than an API, measure your tokens a second against the break-even line above.