LiteLLM proxy in front of vLLM and Ollama on your own GPU

The LiteLLM proxy gives every model you host one OpenAI-compatible endpoint, on port 4000, with its own keys, budgets, routing and logs. In its config file a vLLM server is a hosted_vllm/ model and an Ollama server is an ollama_chat/ model. The version matters more than usual: LiteLLM releases shipped malware twice this year, so I pin ghcr.io/berriai/litellm:v1.108.6, published on 2026-11-12, check its digest, and never pull latest.

Pin a version, and know which ones to avoid#

The rule I follow with LiteLLM is an exact version, a checked digest and a look at the project's advisories before every upgrade. This is its record for 2026, from LiteLLM's own incident reports and its GitHub security advisories:

The poisoned packages were removed from PyPI, and the 1.107.0 images are gone from the registries. LiteLLM replaced its image signing key on 2026-10-23, because it believes the old one was exposed and the 1.107.0 images carry a valid signature from it. In all, the repository lists 18 published advisories, 5 of them critical.

I pinned 1.108.6 for three reasons. It is the newest patch on the 1.108 line, which LiteLLM has run as a long-term support pilot since the week of 2026-11-02. It is newer than every fix in the table. And it was signed with the new key. Pull it and check that the digest is the one this page was written against:

docker pull ghcr.io/berriai/litellm:v1.108.6
docker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/berriai/litellm:v1.108.6
# ghcr.io/berriai/litellm@sha256:c64fe10f82a1f77906f88574f954317c50f67c617e0c97b70ac8fc61b74d3ec3

If you have cosign installed, verify the signature against the key LiteLLM pins by commit:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
  ghcr.io/berriai/litellm:v1.108.6

That release came out the day this page was written, so read the LiteLLM advisories again on the day you deploy.

What the proxy needs on the VM#

LiteLLM needs no GPU of its own. It is a small web service, a 420 MB image, that forwards requests to the servers that do the work. Start those first, each on 127.0.0.1 as its guide shows: vLLM with Docker on port 8000 with its VLLM_API_KEY, and Ollama on port 11434 with a model pulled, here qwen3:8b. Both run on the Bare Metal template.

Write the config file#

The config file maps the names your clients use to the servers behind them. Save this as ~/litellm/config.yaml:

model_list:
  - model_name: qwen3-8b
    litellm_params:
      model: hosted_vllm/Qwen/Qwen3-8B
      api_base: http://127.0.0.1:8000/v1
      api_key: os.environ/VLLM_API_KEY
  - model_name: qwen3-8b-ollama
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
  json_logs: true

Three details decide whether it works. The text after hosted_vllm/ is the model name vLLM serves, and the api_base ends in /v1, because LiteLLM appends /chat/completions to it. The older vllm/ prefix is deprecated. For Ollama I use ollama_chat/, which calls Ollama's /api/chat and is the prefix LiteLLM recommends, and its api_base has no path. os.environ/NAME reads a value from the environment, so no key sits in the file.

Start the proxy and its database#

Virtual keys, budgets and spend logs need a Postgres database. Without one the proxy still forwards requests, but the master key is the only credential and no budget is enforced. Create the secrets once, in the shell where VLLM_API_KEY is still set from the vLLM guide:

mkdir -p ~/litellm && cd ~/litellm
POSTGRES_PASSWORD=$(openssl rand -hex 16)
cat > .env <<EOF
LITELLM_MASTER_KEY=sk-$(openssl rand -hex 32)
LITELLM_SALT_KEY=sk-$(openssl rand -hex 32)
POSTGRES_PASSWORD=$POSTGRES_PASSWORD
DATABASE_URL=postgresql://litellm:$POSTGRES_PASSWORD@127.0.0.1:5432/litellm
VLLM_API_KEY=$VLLM_API_KEY
EOF
chmod 600 .env

The master key is the root credential, and the proxy refuses to start without one. Its sk- prefix is LiteLLM's convention, not a requirement. The salt key encrypts credentials stored in the database and cannot be rotated later, so keep the file. Then save this as compose.yaml next to it:

services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.108.6@sha256:c64fe10f82a1f77906f88574f954317c50f67c617e0c97b70ac8fc61b74d3ec3
    network_mode: host
    command: ["--config", "/app/config.yaml", "--host", "127.0.0.1", "--port", "4000"]
    env_file: .env
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: litellm
    ports:
      - "127.0.0.1:5432:5432"
    volumes:
      - litellm-db:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    restart: unless-stopped

volumes:
  litellm-db:

The proxy uses the host's network so it can reach vLLM and Ollama on 127.0.0.1, where ports published to the loopback address are out of reach of an ordinary container. The price is that nothing isolates its port, so --host 127.0.0.1 is not optional: LiteLLM's default is every interface. Start it and wait for the health check:

docker compose up -d
curl -s http://127.0.0.1:4000/health/liveliness

Send a first request#

The clients change one thing: the base URL. The model name is the one from your config, not the upstream one:

export LITELLM_MASTER_KEY=$(grep ^LITELLM_MASTER_KEY ~/litellm/.env | cut -d= -f2)

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8b", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 1000}'

Change the model to qwen3-8b-ollama and the same request goes to Ollama. With the OpenAI SDK, set base_url="http://127.0.0.1:4000/v1" and pass a LiteLLM key as api_key.

Virtual keys, budgets and rate limits#

The master key is for administration. Applications and people get virtual keys, which you create with it:

curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key_alias": "batch-job", "models": ["qwen3-8b"], "rpm_limit": 60, "tpm_limit": 200000, "max_budget": 25, "budget_duration": "30d"}'

The response carries the new key in its key field, shown once. This one may call only qwen3-8b, at 60 requests and 200,000 tokens a minute, and stops at $25 of spend in each 30 days. A key over its budget gets an error that starts Budget has been exceeded!.

Dollar budgets have a catch on your own GPUs. LiteLLM's price list has no entry for a model you host, so its spend is recorded as $0 and the budget never fills. Give the deployment a price and the budget starts to mean something:

  - model_name: qwen3-8b
    litellm_params:
      model: hosted_vllm/Qwen/Qwen3-8B
      api_base: http://127.0.0.1:8000/v1
      api_key: os.environ/VLLM_API_KEY
    model_info:
      input_cost_per_token: 0.0000002
      output_cost_per_token: 0.0000006

Those two rates are examples, $0.20 and $0.60 per million tokens, not a recommendation. Set them from what the GPU costs you, with the GPU cost calculator. If you only need to stop one client from starving the others, the request and token limits do that without any price.

Balance and fall back between two vLLM servers#

Two deployments with the same model_name are one model to the client. LiteLLM spreads requests across them, at random by default, and moves on when one fails. On a 2-GPU VM that means a second vLLM container on port 8001, each pinned with --gpus device=0 and --gpus device=1 as in running Docker with a GPU:

model_list:
  - model_name: qwen3-8b
    litellm_params:
      model: hosted_vllm/Qwen/Qwen3-8B
      api_base: http://127.0.0.1:8000/v1
      api_key: os.environ/VLLM_API_KEY
  - model_name: qwen3-8b
    litellm_params:
      model: hosted_vllm/Qwen/Qwen3-8B
      api_base: http://127.0.0.1:8001/v1
      api_key: os.environ/VLLM_API_KEY

router_settings:
  num_retries: 2
  fallbacks: [{"qwen3-8b": ["qwen3-8b-ollama"]}]

A failing deployment is taken out of rotation for a cooldown, 5 seconds by default, while the other keeps serving. fallbacks is the second line of defence: when qwen3-8b still fails after its retries, the request goes to the Ollama model. Restart the proxy after a config change with docker compose restart litellm.

Log requests#

With the database connected, LiteLLM records every request by default: the key, the model, the tokens, the spend and whether it failed. The Admin UI shows them at http://127.0.0.1:4000/ui through an SSH tunnel, with admin and the master key as the login. It does not store prompts or answers unless you ask it to:

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  store_prompts_in_spend_logs: true

Think before you turn that on, because it puts every prompt in the Postgres volume. For the server's own log, json_logs: true in the config above makes docker compose logs litellm print one JSON object per line.

Keep the gateway private#

The proxy now holds the keys to every model behind it, so treat it as you treat vLLM: on 127.0.0.1, reached through an SSH tunnel.

ssh -N -L 4000:127.0.0.1:4000 ubuntu@YOUR_INSTANCE_IP

When another machine has to reach it, put a reverse proxy with TLS in front, as in the vLLM Docker guide, and set NO_DOCS, NO_REDOC and NO_OPENAPI to True, because LiteLLM serves its API documentation without a key by default. LiteLLM's own security guide says to run the gateway on a private network when possible.

When a gateway is worth running#

A gateway earns its place when more than one client or more than one server is involved. With several users it gives each a key you can limit and revoke. With several servers it gives clients one address and you the freedom to move models behind it. With both, it is where spend per key is counted.

For one application calling one vLLM server it is one more thing to break. It adds a process, a database and a project that has had two supply-chain incidents this year, and vLLM's own --api-key already covers that case. I would add LiteLLM on the day the second user or the second server arrives, not before.

Before you stop the instance#

Stopping a QuantaCloud instance terminates it and deletes its disk, and the Postgres volume goes with it: virtual keys, budgets and spend logs. Keep config.yaml, compose.yaml and the .env file on your laptop, and export what you need from the spend logs first. A new instance with the same .env starts with the same master key and no virtual keys.

LiteLLM proxy FAQ#

Is it safe to use the latest tag?

No. On 2026-10-21 the latest and main-stable image tags pointed at a compromised build for up to two and a half hours. Pin an exact version, check its digest, and upgrade on your own schedule.

Does LiteLLM need a database?

Not to forward requests. It needs Postgres for virtual keys, budgets, rate limits per key and spend logs. Without one, a request that carries a virtual key fails with No connected db.

Which prefix does vLLM use in LiteLLM?

hosted_vllm/, followed by the model name vLLM serves, with an api_base that ends in /v1. The plain vllm/ prefix is deprecated.

Can clients keep using the OpenAI SDK?

Yes. The proxy speaks the OpenAI API, so a client changes its base URL to the proxy and its key to a virtual key, and asks for the model name from your config.

My rule: run vLLM or Ollama on their own until a second user or a second server shows up, then put LiteLLM in front, pinned to an exact version on its long-term support line and bound to 127.0.0.1. Give every client its own key with a request limit, and price your models before you rely on a dollar budget. Self-hosted LLM inference shows where the gateway sits among the other pieces.