Deploy SGLang with Docker on a GPU server

The fastest way to run SGLang is its official Docker image, lmsysorg/sglang, pinned to a release tag and started with sglang serve, with port 30000 published on 127.0.0.1 only and an API key set. On QuantaCloud that means launching the Bare Metal template (Ubuntu 22.04 with the NVIDIA driver and Docker), checking the driver, and running one container. QuantaCloud has no SGLang template and no managed endpoint, so this is a do-it-yourself deployment: the VM, the container and their security are yours to run.

What you end up with is an OpenAI-compatible server: /v1/chat/completions, /v1/completions and /v1/models, which any OpenAI SDK can call by changing base_url. The current release is SGLang 0.5.23: its image and PyPI package came out on 2026-10-30, and its GitHub release notes on 2026-11-02. The code lives in the sgl-project/sglang repository and the reference is at docs.sglang.io.

Launch an Ubuntu GPU VM#

Launch the Bare Metal template on a GPU that holds your model (the deploy flow). This walkthrough uses one RTX A6000 and Qwen3-8B, whose BF16 weights are 16.4 GB. The LLM VRAM calculator sizes other models.

Most single-GPU VMs are running in about 3 minutes (median). Connect as ubuntu with the SSH key you picked at deploy time (connecting over SSH):

ssh ubuntu@YOUR_INSTANCE_IP

Check the driver first#

The one thing I always check before pulling the image is the driver version, because SGLang 0.5.23 ships only as a CUDA 13 build and CUDA 13 needs an R580 or newer driver. The last image for older drivers is v0.5.19-cu129, and SGLang no longer publishes CUDA 12 images.

nvidia-smi --query-gpu=name,driver_version,memory.total,compute_cap --format=csv

Set the tag once, so every later command uses the right one:

DRIVER_MAJOR=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -n1 | cut -d. -f1)
if [ "$DRIVER_MAJOR" -ge 580 ]; then SGLANG_TAG=v0.5.23; else SGLANG_TAG=v0.5.19-cu129; fi
echo "lmsysorg/sglang:$SGLANG_TAG"

The commands on this page are written for 0.5.23. A v0.5.23-runtime tag is the same release with fewer development tools, 14.5 GB instead of 16.1 GB. Checking your GPU, driver and CUDA version explains the numbers nvidia-smi prints, and running Docker with a GPU covers the NVIDIA Container Toolkit if docker run --rm --gpus all ubuntu nvidia-smi fails.

Run SGLang#

Pin the tag, publish the port on 127.0.0.1 only, and set an API key. Generate the key first and keep it, because every client sends it as a Bearer token.

export SGLANG_API_KEY=$(openssl rand -hex 32)
echo "$SGLANG_API_KEY"

docker run -d --name sglang --gpus all --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN="${HF_TOKEN:-}" \
  "lmsysorg/sglang:$SGLANG_TAG" \
  sglang serve Qwen/Qwen3-8B \
  --host 0.0.0.0 --port 30000 \
  --context-length 32768 \
  --reasoning-parser qwen3 \
  --api-key "$SGLANG_API_KEY"

docker logs -f sglang

Three details differ from vLLM's image. The SGLang image has no entrypoint, so the command starts with sglang serve and the model comes right after it. SGLang listens on 127.0.0.1 by default, so inside the container it needs --host 0.0.0.0 for Docker to forward to it, while the -p 127.0.0.1: prefix keeps the port off the VM's public interface. And the port is 30000, not 8000.

--ipc=host gives PyTorch the shared memory it needs, and the Hugging Face mount keeps the weights on the VM's disk when you recreate the container. HF_TOKEN only matters for gated repos. --reasoning-parser qwen3 returns Qwen3's thinking separately from its answer. Wait for The server is fired up and ready to roll! in the log. Older guides start the server with python3 -m sglang.launch_server --model-path, which still works.

The same deployment with Docker Compose#

Compose keeps the flags in a file and restarts the server if it crashes. Write the settings to ~/sglang/.env and save the Compose file below next to it as compose.yaml:

mkdir -p ~/sglang && cd ~/sglang
cat > .env <<EOF
SGLANG_TAG=$SGLANG_TAG
SGLANG_API_KEY=$SGLANG_API_KEY
HF_TOKEN=${HF_TOKEN:-}
EOF
chmod 600 .env

services:
  sglang:
    image: lmsysorg/sglang:${SGLANG_TAG}
    command: ["sglang", "serve", "Qwen/Qwen3-8B", "--host", "0.0.0.0", "--port", "30000", "--context-length", "32768", "--reasoning-parser", "qwen3", "--api-key", "${SGLANG_API_KEY}"]
    environment:
      - HF_TOKEN=${HF_TOKEN}
    ports:
      - "127.0.0.1:30000:30000"
    ipc: host
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:30000/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 600s

The health check can call /health without the key, and start_period gives the first start 10 minutes to download the weights before failures count. Remove the container from the previous section, then start the Compose version:

docker rm -f sglang
docker compose up -d
docker compose logs -f

Send a first request#

Both calls need the key. From the VM:

curl -s http://127.0.0.1:30000/v1/models -H "Authorization: Bearer $SGLANG_API_KEY"

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Authorization: Bearer $SGLANG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Explain the KV cache in two sentences."}], "max_tokens": 1000}'

Qwen3 thinks before it answers, and the thinking counts against max_tokens, so leave it room. From your laptop, open an SSH tunnel and point the OpenAI Python SDK at it:

ssh -N -L 30000:127.0.0.1:30000 ubuntu@YOUR_INSTANCE_IP

import os
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key=os.environ["SGLANG_API_KEY"])
reply = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],
    max_tokens=1000,
)
print(reply.choices[0].message.content)

The flags that decide whether it fits#

SGLang budgets GPU memory with one number, --mem-fraction-static: the share of the card for the weights plus the KV cache pool. Activations and CUDA graph buffers live outside it. When you leave it unset, SGLang computes it from the GPU's memory minus a reserve, and its tuning guide says leaving 5 to 8 GB for activations is usually enough.

The line to read is the one SGLang prints just before it is ready:

max_total_num_tokens=N, chunked_prefill_size=N, max_prefill_tokens=N, max_running_requests=N, context_len=32768, available_gpu_mem=X.XX GB

max_total_num_tokens is the KV pool that all running requests share, and available_gpu_mem is what is left for activations. Qwen3-8B stores 144 KiB of KV cache per token, so one 32,768-token request takes 4.5 GiB of that pool (our calculation, from the KV cache formula).

Unlike vLLM, SGLang starts even when one full-length request does not fit. It logs a warning that begins KV pool holds and says long requests will be refused or truncated at serving time. If you see it, lower --context-length or raise --mem-fraction-static.

When it runs out of memory#

The fix depends on when the error appears. SGLang's startup error names the first fix, and its tuning guide gives the others:

A different symptom is the warning KV cache pool is full. Retract requests. It means the pool, not the GPU, ran out. SGLang's guide calls about once a minute fine. If it comes more often, raise --schedule-conservativeness to 1.3 so SGLang admits fewer requests. Fixing CUDA out of memory covers the generic PyTorch errors.

Use more than one GPU#

The flag for one model across the GPUs of a VM is --tp-size: add --tp-size 2 on a 2x VM. If SGLang reports peer access is not supported between these two devices, add --enable-p2p-check. When a full copy of the model fits on each GPU, SGLang's tuning guide favours data parallelism for throughput: a second copy instead of a split. --dp-size 2 does that inside one server, and the docs recommend SGLang's Model Gateway in front of separate servers as the better way to run the copies. On QuantaCloud, multi-GPU configurations appear in the GPU catalog when they have capacity, and larger sets are built to order.

Serve a quantized model#

A pre-quantized checkpoint needs no flag. SGLang reads the method from the checkpoint's config, and its docs warn against adding --quantization on top of a model that is already quantized. That covers AWQ, FP8 and NVFP4 checkpoints from Hugging Face. In the docker run command, only the model name changes:

sglang serve Qwen/Qwen3-32B-AWQ --host 0.0.0.0 --port 30000 \
  --context-length 32768 --reasoning-parser qwen3 \
  --api-key "$SGLANG_API_KEY"

To quantize a 16-bit model as it loads, pass --quantization fp8. SGLang's docs recommend pre-quantized checkpoints over this. Two details are worth knowing. The plain gptq method was removed on NVIDIA GPUs in favour of gptq_marlin, the Marlin kernel. And the KV cache has its own flag, --kv-cache-dtype fp8_e4m3. The quantization formats guide explains which format suits which GPU.

Metrics and logs#

Add --enable-metrics and SGLang serves Prometheus metrics at /metrics:

curl -s http://127.0.0.1:30000/metrics | grep -E "sglang:(token_usage|num_running_reqs|num_queue_reqs|gen_throughput)"

sglang:token_usage is the share of the KV pool in use, and sglang:time_to_first_token_seconds is the latency histogram to alert on. The server log carries the same picture in lines that start Decode batch. SGLang does not log request contents unless you add --log-requests.

Expose it safely#

The default I recommend is the SSH tunnel from the first request, because then the API is reachable only through port 22. SGLang's key protects every route except the ones that start with /health, /ready and /metrics, which is stricter than vLLM's. Those three stay open, so anyone who can reach the port can read your metrics. Docker also routes published ports before ufw sees them, so -p 30000:30000 on all interfaces ignores your firewall. When an app on another machine needs the API, put a reverse proxy with TLS in front, as in the vLLM Docker guide, or put both servers behind a LiteLLM proxy.

Install with pip or uv instead#

SGLang also installs into a Python environment, which suits a VM where it is one part of a larger project. It needs Python 3.11 or newer and the same R580 driver. With uv installed as in the vLLM install guide:

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --prerelease=allow sglang==0.5.23
sglang serve Qwen/Qwen3-8B --port 30000 --api-key "$SGLANG_API_KEY"

Without --host, the server binds to 127.0.0.1, which is what you want outside a container. Give it its own environment and do not upgrade torch inside it, because SGLang pins torch==2.14.1.

SGLang or vLLM#

SGLang and vLLM share most of their design, and both serve the same OpenAI API. SGLang's case is its stricter API key and its scheduler, which reorders waiting requests to reuse cached prefixes when you pass --schedule-policy lpm. The default policy is first come, first served. vLLM's case is the smaller download, 9.7 GB for 0.33.0 against 16.1 GB, which you pay again on every fresh instance. vLLM vs SGLang compares them feature by feature.

Before you stop#

Stopping a QuantaCloud instance terminates it and deletes its disk, including the pulled image and the Hugging Face cache. The next launch downloads both again: 16.1 GB of image plus 16.4 GB for Qwen3-8B. Keep the commands from this page in a script on your laptop and run it over SSH on each new instance. The first hour is charged at launch and the unused seconds are refunded when you stop (how billing works).

My rule for the first deployment is to start on an RTX A6000 with an 8B model and exactly these commands, then read the startup line. Raise --mem-fraction-static only when available_gpu_mem shows more than 8 GB to spare, and move to a larger GPU when the KV pool is too small for your context.