Run Claude Code, OpenCode or Qwen Code against a model on your own GPU

All three agents can work against a model you serve yourself, and one vLLM server covers them: it speaks the OpenAI Chat Completions API that OpenCode and Qwen Code use and the Anthropic Messages API that Claude Code uses, on the same port. On QuantaCloud that means launching a Bare Metal VM, starting one container with tool calling turned on, and reaching it through an SSH tunnel. QuantaCloud has no managed endpoint, so the VM, the server and their security are yours to run. One caveat before you start: Anthropic does not support Claude Code with non-Claude models, so that path is the least certain of the three.

What the model and the server have to do#

A coding agent sends tool definitions with every request, so the server has to turn the model's output back into tool calls. In vLLM that takes two flags: --enable-auto-tool-choice, and --tool-call-parser with the parser that matches the model. The parser differs by family:

The second requirement is context. An agent keeps files, tool output and its own reasoning in the conversation, and Ollama's docs ask for a window of 64k or more for both Claude Code and OpenCode. I use 131,072 tokens here. Open-weight coding models compares the models and their benchmark scores. This walkthrough uses Qwen3.8-27B, which is Apache-2.0, on one 96 GB RTX PRO 6000, the GPU that vLLM's recipe for the model lists as verified.

Launch the GPU and check the driver#

Launch the Bare Metal template on an RTX PRO 6000 Blackwell (the deploy flow), then connect as ubuntu with your SSH key (connecting over SSH).

ssh ubuntu@YOUR_INSTANCE_IP
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

The one thing I always check is the driver, because vLLM 0.33.0 publishes CUDA 13.0 images only and they need an R580 or newer driver. On an R535 or R570 driver, add -e VLLM_ENABLE_CUDA_COMPATIBILITY=1 to the docker run command below, which vLLM supports on select professional and datacenter GPUs. The vLLM Docker guide covers the GPU check for Docker and the NVIDIA Container Toolkit.

Serve the model with tool calling on#

Generate a key, then start the server with the port published on 127.0.0.1 only:

export VLLM_API_KEY=$(openssl rand -hex 32)
echo "$VLLM_API_KEY"

docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:v0.33.0 \
  Qwen/Qwen3.8-27B \
  --served-model-name qwen38 \
  --max-model-len 131072 \
  --max-num-seqs 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --api-key "$VLLM_API_KEY"

docker logs -f vllm

Everything after the image name is a vllm serve argument. --served-model-name qwen38 gives the model a short name without a slash, which Claude Code needs, and a name that starts with qwen, which Qwen Code uses to pick its thinking controls. --reasoning-parser qwen3 matters as much as the tool parser: Qwen3.8 thinks before it answers, and without the flag the thinking lands in the reply. --max-num-seqs 8 is the limit vLLM's recipe sets for this model on an RTX PRO 6000. If you never send screenshots, --language-model-only skips the vision encoder. The first start downloads 9.74 GB of image and 55.56 GB of weights, and the repository is not gated, so you need no Hugging Face token. vLLM's recipe ran this model on its own vllm/vllm-openai:qwen38 image, and I pin the release instead.

The server is ready when the health route answers 200. Then ask for a tool call:

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/health

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen38", "max_tokens": 2000, "messages": [{"role": "user", "content": "List the files in the current directory."}], "tools": [{"type": "function", "function": {"name": "run_shell", "description": "Run a shell command and return its output", "parameters": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}}}]}'

The reply should carry a tool_calls array with run_shell in it and a finish_reason of tool_calls. If the call shows up as text inside content, the parser flag is wrong for the model.

What the context window costs#

The context you allow is paid for in KV cache. Qwen3.8-27B keeps 64 KiB per token (2 x 16 attention layers x 4 KV heads x 256 x 2 bytes), so one full 131,072-token session holds 8 GiB and a 262,144-token one holds 16 GiB. The RTX PRO 6000 reports 97,887 MiB, vLLM takes 92% of it, and the BF16 weights are 51.75 GiB, which leaves 36.2 GiB: room for four full sessions at 131,072 tokens, or two at the model's native 262,144 (our calculation, before activations). vLLM prints what it actually got when it starts:

GPU KV cache size: N tokens, Maximum concurrency for 131,072 tokens per request: X.XXx

An agent resends the whole conversation on every turn, and vLLM's prefix caching, which is on by default, means only the new tokens are computed. The KV cache guide has the formula for other models, and --kv-cache-dtype fp8 halves the cost per token if you need more room.

Keep the endpoint private with a key and a tunnel#

The default I recommend is to leave the port on 127.0.0.1 and reach it through SSH. Run this on your laptop and leave it open:

ssh -N -L 8000:127.0.0.1:8000 ubuntu@YOUR_INSTANCE_IP

Your laptop's http://127.0.0.1:8000 now reaches the server, so the agent runs next to your repository and the model runs on the GPU. Keep the key on as well. vLLM's --api-key guards the /v1 routes, which include /v1/chat/completions, /v1/messages and /v1/responses, but /invocations on the same port needs no key at all. That is why I do not publish port 8000 to the internet. The vLLM Docker guide shows a reverse proxy that lets only /v1 through if a team needs to share the endpoint.

On your laptop, put the key where the agents can read it:

export VLLM_API_KEY=paste-the-key-you-printed

OpenCode with a custom provider#

OpenCode 1.19.5 talks to any OpenAI-compatible server through a custom provider. Install it with curl -fsSL https://opencode.ai/install | bash or npm install -g opencode-ai, then save this as opencode.json in your project, or as ~/.config/opencode/opencode.json for every project:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "mygpu": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM on my GPU",
      "options": {
        "baseURL": "http://127.0.0.1:8000/v1",
        "apiKey": "{env:VLLM_API_KEY}"
      },
      "models": {
        "qwen38": {
          "name": "Qwen3.8-27B",
          "limit": {
            "context": 131072,
            "output": 32768
          }
        }
      }
    }
  },
  "model": "mygpu/qwen38"
}

@ai-sdk/openai-compatible is the package OpenCode uses for servers that expose /v1/chat/completions. The model key, qwen38, has to match --served-model-name, and limit.context tells OpenCode how much context it has left, so keep it equal to --max-model-len. Run opencode, then /models to confirm the model is selected.

Qwen Code with an OpenAI-compatible endpoint#

Qwen Code 0.27.2 is Apache-2.0 and reads its providers from ~/.qwen/settings.json. Install it with npm install -g @qwen-code/qwen-code@latest (Node.js 22 or newer), then write:

{
  "modelProviders": {
    "openai": [
      {
        "id": "qwen38",
        "name": "Qwen3.8-27B on my GPU",
        "envKey": "VLLM_API_KEY",
        "baseUrl": "http://127.0.0.1:8000/v1",
        "generationConfig": {
          "contextWindowSize": 131072
        }
      }
    ]
  },
  "security": {
    "auth": {
      "selectedType": "openai"
    }
  },
  "model": {
    "name": "qwen38"
  }
}

envKey is the name of the environment variable that holds the key, not the key itself. selectedType and model.name make Qwen Code start on this model without the /auth dialog. Qwen Code infers thinking control from a model id that starts with qwen. If you serve the model under another name, its docs say to add "capabilities": {"reasoning": {"profile": "qwen-chat-template"}} to the entry. Run qwen in your project, and /model to switch.

Claude Code with an Anthropic-compatible endpoint#

Claude Code 2.1.322 does not speak the OpenAI API. It posts to /v1/messages in Anthropic's format, and it finds the server through two variables: ANTHROPIC_BASE_URL, without /v1 at the end, and ANTHROPIC_AUTH_TOKEN, which it sends as a Bearer token, the form vLLM's key check expects.

export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
export ANTHROPIC_AUTH_TOKEN="$VLLM_API_KEY"
export ANTHROPIC_MODEL=qwen38
export ANTHROPIC_DEFAULT_FABLE_MODEL=qwen38
export ANTHROPIC_DEFAULT_OPUS_MODEL=qwen38
export ANTHROPIC_DEFAULT_SONNET_MODEL=qwen38
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen38
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1

The model variables point every alias Claude Code might ask for, including the one it uses for background work, at the single model you serve. CLAUDE_CODE_MAX_CONTEXT_TOKENS is needed because Claude Code assumes a 200K window for a model name it does not recognize, which is more than this server allows. The last variable turns off update checks and telemetry that would otherwise go to Anthropic. Test the endpoint the way Claude Code's docs do, then start it:

curl -sS -X POST "$ANTHROPIC_BASE_URL/v1/messages" \
  -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "qwen38", "max_tokens": 500, "messages": [{"role": "user", "content": "Say hello."}]}'

claude

A reply that starts with {"id":"msg_ means the route and the key work. Inside Claude Code, /status should show your address on the Anthropic base URL line. If requests fail with an error about unexpected fields, set CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1, which stops Claude Code sending its pre-release request fields.

These are the servers that expose an Anthropic Messages endpoint today, each checked in its own docs at the version shown:

SGLang and the LiteLLM proxy have their own guides.

What you lose against the hosted models is worth spelling out. Anthropic's docs say it does not support routing Claude Code to non-Claude models through any gateway, so when something breaks you are on your own. Web search depends on a server-side capability that a vLLM server does not have. Cloud sessions, Remote Control, the Chrome extension, computer use and voice dictation need a claude.ai account, and Remote Control is disabled whenever ANTHROPIC_BASE_URL points at another host. Claude Code also sends the request fields it would send to a current Claude model, such as adaptive thinking and effort, and the server has to accept or ignore them. And you lose the model: on the official Terminal-Bench 4.1 leaderboard, checked on 2026-11-12, the best open-weight entry scores 41.2% with Claude Code as the harness, against 65.8% for the top hosted model, and that entry is GLM-5.3, which needs eight GPUs.

Other agents#

Most other coding agents take an OpenAI-compatible endpoint, so the same server and tunnel work.

Using Ollama instead of vLLM#

Ollama 0.40.16 does the wiring for you: ollama launch claude and ollama launch opencode start each agent against a model you pick, through Ollama's own /v1/messages and /v1 endpoints. By hand, Claude Code needs ANTHROPIC_BASE_URL=http://localhost:11434, ANTHROPIC_AUTH_TOKEN=ollama and an empty ANTHROPIC_API_KEY, and OpenCode needs the provider block above with http://localhost:11434/v1 as its baseURL. Ollama's library has Qwen3.8-27B as qwen3.8:27b, an 18 GB download that fits a 48 GB card. Set the context to 64k or more first, because Ollama's default depends on the GPU: Ollama context length shows how, and the Ollama API on a remote GPU covers reaching it from your laptop. I would still use vLLM when more than one person shares the server, for the reasons in vLLM vs Ollama.

What a working day costs#

The cost is the GPU's hourly price times the hours it runs, whether the agent is busy or idle. The RTX PRO 6000 is $2.17/GPU-hr now. On 2026-09-27 it was $2.39 an hour, which made an eight-hour day $19.12 (our calculation).

Prices checked 14 Nov 2026, 04:33 UTC

The first hour is charged at launch and unused seconds are refunded when you stop (how billing works). Stopping terminates the instance and deletes its disk, so every morning starts with the same commands and the same 55.56 GB download. Keep them in a script on your laptop, never keep the only copy of your code on the VM, and stop the instance when you finish: auto-stopping an idle GPU has a script for that, and the GPU cost calculator turns hours into a monthly figure.

A 48 GB card costs less. An RTX A6000 is $0.48/GPU-hr, and Qwen's FP8 build of the same model is 30.87 GB, which leaves 12.6 GiB for the cache on that card (our calculation). vLLM's recipe does not list the FP8 build for Ampere or Ada GPUs, so I treat that combination as untested.

How fast it feels#

What an agent feels like depends on output tokens per second for a single request, because an agent turn is one long generation after another. That number comes from a test on the actual GPU, not from a model card.

My rule for a first setup: serve one model under a short name with the tool and reasoning parsers on, keep the port on 127.0.0.1, and connect OpenCode or Qwen Code first, because they are built for an OpenAI-compatible endpoint. Add Claude Code once the server is proven, and expect to give up web search and Anthropic's support. If the model is the limit, the open-weight LLM roundup lists what the next tier of GPUs can hold.