Serve embedding models for RAG with Text Embeddings Inference

Text Embeddings Inference (TEI) is Hugging Face's server for embedding and reranker models, and the fastest way to run it is the Docker image built for your GPU's architecture. The current release is 1.9.4, from 2026-09-15. On QuantaCloud that means the Bare Metal template, one container published on 127.0.0.1 with an API key, and a RAG app that calls /v1/embeddings the way it would call OpenAI. QuantaCloud has no managed embedding endpoint, so this is self-hosted inference: the container and its security are yours to run.

Pick the image for your GPU#

TEI ships one image per GPU architecture, and the tag has to match the GPU's compute capability. These are the 1.9.4 tags for the GPUs QuantaCloud rents:

A B200 in a reserved build takes 100-1.9.4, also marked experimental. TEI's quick tour uses cuda-1.9, an image that holds several builds and picks one when it starts. It runs the 8.0 build on 8.6 and 8.9 cards, though, so on an RTX A6000 or an L40S I use the tag built for that architecture. Every image is about a 3.1 GB download, built on CUDA 12.9.1 with NVIDIA's compatibility package, and TEI's docs ask for a driver that supports CUDA 12.2 or newer.

Launch the Bare Metal template on your GPU (deploying a GPU), connect as ubuntu (connecting over SSH), and let the GPU pick the tag:

CAP=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -n1)
case "$CAP" in
  8.0) TEI_TAG=1.9.4 ;;
  8.6) TEI_TAG=86-1.9.4 ;;
  8.9) TEI_TAG=89-1.9.4 ;;
  9.0) TEI_TAG=hopper-1.9.4 ;;
  12.0) TEI_TAG=120-1.9.4 ;;
  *) echo "No TEI GPU image for compute capability $CAP" ;;
esac
echo "ghcr.io/huggingface/text-embeddings-inference:$TEI_TAG"

A mismatch fails at start-up rather than running slowly: the 8.6 image on an H100 stops with Runtime compute cap 90 is not compatible with compile time compute cap 86.

Pick a model#

The model decides retrieval quality, and every model here fits a 48 GB card with room to spare. TEI loads weights in FP16 on a GPU by default, so bge-m3 takes about 1.1 GB of memory before activations and Qwen3-Embedding-8B about 15.1 GB (our calculation: parameters x 2 bytes).

My default for RAG is bge-m3. It covers more than 100 languages, reads up to 8,192 tokens and needs no instruction on queries, and it is the model the Open WebUI RAG guide runs through Ollama. bge-m3's repo has no safetensors file, so TEI logs that it is using pytorch_model.bin instead and loads it more slowly.

The Qwen3 embedding models are the ones TEI's README ranks highest: it lists the 8B, 4B and 0.6B at MTEB ranks 2, 3 and 4. They expect an instruction on queries, and TEI adds the one stored with the model when you send "prompt_name": "query" to /embed. Documents go in as they are. Their 32,768-token context is longer than TEI's default --max-batch-tokens of 16,384, so TEI logs a warning and cuts longer inputs to 16,384 tokens unless you raise that flag. nomic-embed-text-v1.5 is the smallest, and it needs a search_query: or search_document: prefix on every text.

Check that Docker can see the GPU#

Docker hands the GPU to TEI through the NVIDIA Container Toolkit, so check that before the first pull:

docker run --rm --gpus all ubuntu nvidia-smi

If it prints the GPU table, Docker and the GPU work together. If Docker answers could not select device driver "" with capabilities: [[gpu]], or failed to discover GPU vendor from CDI from Docker 29.3 on, install the toolkit and run the check again. Put sudo in front of docker while your user is not in the docker group.

Run TEI with an API key#

Publish the port on 127.0.0.1, set an API key and keep the model cache on the VM's disk. Generate the key first and keep it, because every client sends it as a Bearer token:

export TEI_API_KEY=$(openssl rand -hex 32)
echo "$TEI_API_KEY"

docker run -d --name tei --gpus all \
  -p 127.0.0.1:8080:80 \
  -v ~/tei-data:/data \
  -e API_KEY="$TEI_API_KEY" \
  "ghcr.io/huggingface/text-embeddings-inference:$TEI_TAG" \
  --model-id BAAI/bge-m3

docker logs -f tei

Everything after the image name is a TEI argument. The image listens on port 80 and caches models in /data, so the mount keeps bge-m3 on the VM when you recreate the container. The API_KEY variable does the same job as --api-key. Wait for Ready in the log, then press Ctrl+C to leave the log.

Two more flags are worth knowing. --revision with a commit hash pins the model files, so every launch serves the same weights. -e HF_TOKEN passes the Hugging Face token from your shell for gated models, such as google/embeddinggemma-300m. Load it first with read -rs HF_TOKEN; export HF_TOKEN, which keeps it out of your shell history. TEI uses the first GPU it sees, so on a VM with several GPUs run one container per GPU, each with --gpus device=N and its own port.

Call the API#

TEI answers on its own routes and on an OpenAI-compatible one, and every inference call needs the key. /embed takes one text or a list and returns one vector per text, normalized by default:

curl -s http://127.0.0.1:8080/embed \
  -H "Authorization: Bearer $TEI_API_KEY" -H "Content-Type: application/json" \
  -d '{"inputs": ["Stopping an instance deletes its disk.", "Billing runs from launch to stop."]}'

curl -s http://127.0.0.1:8080/info -H "Authorization: Bearer $TEI_API_KEY"

/info reports the model, its precision, the longest input it accepts and the batching limits, which is the quickest way to confirm your flags took effect. /v1/embeddings takes the OpenAI request format, so the OpenAI Python SDK works with a changed base_url:

import os
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=os.environ["TEI_API_KEY"])
result = client.embeddings.create(model="BAAI/bge-m3", input=["How is GPU time billed?"])
print(len(result.data[0].embedding))

Save it as embed_test.py and run it with uv run --with openai embed_test.py in the shell that holds TEI_API_KEY. It prints 1024, bge-m3's vector size. The benchmark section below installs uv if the VM does not have it yet.

A reranker runs in a container of its own, because one TEI process serves one model. Start it on port 8081 with the same key, wait for Ready in docker logs tei-rerank, then send a query with the passages your vector search returned:

docker run -d --name tei-rerank --gpus all \
  -p 127.0.0.1:8081:80 \
  -v ~/tei-data:/data \
  -e API_KEY="$TEI_API_KEY" \
  "ghcr.io/huggingface/text-embeddings-inference:$TEI_TAG" \
  --model-id BAAI/bge-reranker-v2-m3

curl -s http://127.0.0.1:8081/rerank \
  -H "Authorization: Bearer $TEI_API_KEY" -H "Content-Type: application/json" \
  -d '{"query": "What happens to my data when I stop?", "texts": ["Stopping deletes the disk.", "Prices are per GPU-hour."]}'

It returns an index and a score for every text, and "return_text": true adds the text back.

Keep it on 127.0.0.1#

The safest setup is the one above: published on 127.0.0.1 and reached through an SSH tunnel, so TEI is reachable only through SSH on port 22. TEI listens on every interface inside the container, and Docker publishes a port on every interface of the VM unless you name an address. Docker also routes published ports before ufw sees them, so the firewall does not help. On Docker releases older than 28.0.0, hosts on the same network segment can even reach ports published to 127.0.0.1, so check docker version and keep the key on regardless.

From your laptop, a tunnel puts TEI on your own port 8080 (the SSH and VS Code guide covers keys and port forwarding):

ssh -N -L 8080:127.0.0.1:8080 ubuntu@YOUR_INSTANCE_IP

When an app on another machine has to call TEI directly, put a reverse proxy with TLS in front and keep the key on, as the vLLM Docker guide does with Caddy. The key guards the inference routes and /info, but not /health, /ping, /metrics or the /docs page, so let only the routes you use through the proxy.

Tune batching#

The flag that matters most is --max-batch-tokens, the number of tokens TEI packs into one batch on the GPU. TEI's help text says to set it as high as possible until the model is compute-bound, because TEI cannot work out the memory overhead for you. Raise it in steps, rerun the benchmark below with nvidia-smi open, and stop when tokens per second stop rising.

A request over the client limit fails with HTTP 422 and batch size 64 > maximum allowed batch size 32. TEI's OpenAPI page still lists 413 for that error, but the code has returned 422 since 1.9.0. Texts beyond the concurrency limit get HTTP 429 with Model is overloaded.

Pair it with Open WebUI and vLLM#

TEI fits a RAG stack as the embedding service, next to a chat model served by vLLM or Ollama. In the Docker Compose setup from the Open WebUI and Ollama guide, add TEI as a third service on the same Compose network and leave it unpublished, so only Open WebUI can reach it. First add the key and the tag to the .env file next to compose.yaml, after running the tag snippet from the first section in the same shell:

cd ~/open-webui
echo "TEI_API_KEY=$(openssl rand -hex 32)" >> .env
echo "TEI_TAG=$TEI_TAG" >> .env

Then add this service under services:, and tei: under volumes:, in compose.yaml:

  tei:
    image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
    container_name: tei
    restart: unless-stopped
    command: ["--model-id", "BAAI/bge-m3"]
    environment:
      - API_KEY=${TEI_API_KEY}
    volumes:
      - tei:/data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

After sudo docker compose up -d, point Open WebUI at it in Settings > Admin > Documents:

1. Set Embedding Model Engine to OpenAI.
2. Set API Base URL to http://tei:80/v1, and API Key to the TEI_API_KEY value from .env.
3. Replace the text-embedding-3-small that Open WebUI fills in with BAAI/bge-m3.
4. Set Embedding Batch Size to 32, TEI's limit per request, and Embedding Concurrent Requests to 8, then click Save.
5. If knowledge bases already exist, click Reindex, because vectors from two models do not compare.

The concurrency setting matters because its default, 0, means no limit. Open WebUI then sends every batch of a large file at once, and TEI answers 429 as soon as more than 512 texts are in flight. Eight requests of 32 texts stay at 256.

Leave reranking out of this link. Open WebUI's External reranking engine sends Cohere-style requests with documents and top_n and expects a relevance_score back, which TEI's /rerank does not speak, so call a TEI reranker from your own code. On the Open WebUI + Ollama template, the simpler route is bge-m3 through the template's own Ollama, as the RAG guide shows.

On the same GPU as vLLM, start TEI first and leave it room. vLLM claims 92% of the GPU's memory by default, and refuses to start when that is more than is free, with an error that says the free memory is less than desired GPU memory utilization. On an RTX A6000 with ECC on, --gpu-memory-utilization 0.80 leaves 9.0 GiB of the card's 45 GiB for TEI and the CUDA contexts, and still gives Qwen3-8B about 151,000 tokens of KV cache, against about 190,000 at the default (our calculation: 46,068 MiB / 1,024 x 0.80 = 35.99 GiB, minus 15.26 GiB of weights, at 144 KiB per token). The KV cache guide has the formula for other models, and LLM inference on QuantaCloud covers the vLLM side.

Throughput and cost per million tokens#

Throughput is what you pay for, because every model above fits every GPU. The cost of embedding a million tokens is the hourly price divided by (tokens per second x 3,600), times 1,000,000. Today's prices for the RTX A6000, L40S and H100 PCIe:

The benchmark script

The script sends 640 requests of 32 passages each, 16 at a time after one warm-up request, and prints passages and tokens per second. Each passage starts with its own number, so no two requests are the same. Sixteen requests of 32 passages keep 512 texts in flight, exactly the default --max-concurrent-requests, so raise that flag before you raise either number in the script. Save it as bench_tei.py on the VM and run it against a running TEI with uv run --with httpx bench_tei.py:

# bench_tei.py: embed 20,480 passages, 32 per request, 16 requests in flight
import asyncio
import os
import time

import httpx

URL = "http://127.0.0.1:8080"
HEADERS = {"Authorization": f"Bearer {os.environ['TEI_API_KEY']}"}
PASSAGE = ("Stopping a GPU instance terminates it and deletes its disk, so copy results "
           "off first and keep the vector database somewhere else. ") * 12
REQUESTS, PER_REQUEST, IN_FLIGHT = 640, 32, 16


async def main():
    async with httpx.AsyncClient(base_url=URL, headers=HEADERS, timeout=300) as client:
        r = await client.post("/tokenize", json={"inputs": f"0.0 {PASSAGE}"})
        tokens = len(r.json()[0])
        gate = asyncio.Semaphore(IN_FLIGHT)

        async def send(i):
            async with gate:
                texts = [f"{i}.{j} {PASSAGE}" for j in range(PER_REQUEST)]
                r = await client.post("/embed", json={"inputs": texts})
                r.raise_for_status()

        await send(-1)  # warm-up
        start = time.perf_counter()
        await asyncio.gather(*(send(i) for i in range(REQUESTS)))
        took = time.perf_counter() - start

    passages = REQUESTS * PER_REQUEST
    print(f"{passages} passages of about {tokens} tokens in {took:.1f} s: "
          f"{passages / took:.0f} passages/s, {passages * tokens / took:.0f} tokens/s")


asyncio.run(main())

The token count comes from TEI's own /tokenize, so it is correct for whichever model is loaded. If uv is not on the VM yet, curl -LsSf https://astral.sh/uv/install.sh | sh followed by source $HOME/.local/bin/env installs it.

Before you stop#

Stopping the instance terminates it and deletes its disk: the image, the model cache and anything stored next to them. Keep the vector database your app writes to off the VM, or copy it off before you stop, because there are no volumes or snapshots. The next launch pulls the 3.1 GB image and the model again, 2.27 GB for bge-m3, so keep the docker run line in a script, and see downloading Hugging Face models fast to shorten the wait. The first hour is charged at launch and the unused seconds are refunded when you stop (pricing).

TEI questions#

Which GPU should I rent for TEI?

The lowest-priced one that keeps up with your queue. On 2026-09-27 that was the RTX A6000, from $0.48/GPU-hr today, and the largest model above, Qwen3-Embedding-8B, needs about 15.1 GB of its 48 GB. Move to an L40S or an H100 PCIe when your own benchmark shows more tokens per dollar there.

Can one TEI container serve two models?

No. One TEI process serves one model, so an embedding model and a reranker run as two containers, each on its own port.

Does TEI return bge-m3's sparse vectors?

No. TEI returns bge-m3's dense vectors. Its sparse route, /embed_sparse, needs SPLADE pooling, which TEI offers only for masked-language-model checkpoints, and bge-m3 is not one.

Do I need a Hugging Face token?

Only for gated or private models. bge-m3, the Qwen3 embedding models, nomic-embed-text-v1.5 and bge-reranker-v2-m3 are not gated, while google/embeddinggemma-300m is.

My rule for RAG embeddings is one RTX A6000, the 86-1.9.4 image, bge-m3 behind an API key on 127.0.0.1, and Open WebUI or your own code calling /v1/embeddings. Raise --max-batch-tokens only after the benchmark shows the GPU has room, and keep the vector database off the instance, because stopping deletes the disk. When a chat model shares the card, start TEI first and lower --gpu-memory-utilization in the command from the vLLM Docker guide.