Ollama context length: the default, how to change it, and what it costs in VRAM

Ollama's default context length depends on how much GPU memory the server finds: 4,096 tokens below 23 GiB, 32,768 from 23 GiB and 262,144 from 47 GiB, capped at the context each model was trained with. Ollama's documentation rounds the two thresholds to 24 and 48 GiB. That is the rule in Ollama 0.40.16, released on 2026-11-09, and it has been the rule since 0.15.5. On a GPU with 48 GB or more it means a model can start at its full context, the KV cache for all of it is allocated when the model loads, and if model and cache do not fit together, part of the model moves to the CPU without an error. So set the context on purpose, and check what the loaded model really got.

The default, release by release#

The default has changed more than once, which is why answers about it disagree. These are the changes in Ollama's own release notes:

Ollama's own documentation has not caught up with itself. At the 0.40.16 tag the context-length page gives the memory rule, the FAQ still says 4,096 tokens, and the Modelfile reference still lists num_ctx with a default of 2,048. The code is what runs. At startup the server adds up the total memory of every GPU it found and picks 262,144 tokens at 47 GiB or more, 32,768 at 23 GiB or more, and 4,096 below that. The thresholds sit 1 GiB under the documented 48 and 24, and a comment in the source gives the reason: small differences in the memory a card reports.

Two details of that rule matter on a GPU server. It uses total memory and adds the GPUs together, so two 48 GB cards count as about 90 GiB and land in the top tier. And a single 48 GB card can fall on either side of the line. With ECC on, as the L40 and L40S ship, it reports 46,068 MiB, which is 44.99 GiB and gets 32,768 tokens. With ECC off it reports 49,140 MiB and gets 262,144.

See what a loaded model is really using#

The CONTEXT column of ollama ps is the number to trust, because it shows the context after every rule has been applied. Send the model a request first, since ollama ps lists only loaded models, then run it on the server (connecting over SSH covers getting there):

ollama ps
curl -s http://127.0.0.1:11434/api/ps

Ollama's documentation shows the output like this:

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest    c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

Release 0.40.16 prints one more column, RUNNER, between CONTEXT and UNTIL. The API call returns the same value as context_length. Three other readings fill in the rest:

The server log is in the journal for the systemd service and in docker logs for a container:

journalctl -u ollama --no-pager | grep -E "vram-based default context|KV buffer size"
sudo docker logs ollama 2>&1 | grep -E "vram-based default context|KV buffer size"

Three ways to set it#

The context can be set on the server, on a model or on a single request, and the more specific setting wins: a request beats a model, and a model beats the server.

On a server I set it once, as an environment variable. With the systemd service that is an override, followed by a restart:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Ollama environment variables covers the Docker form and how to check that the server read the value. Setting the variable also switches off the automatic choice by GPU memory.

To bake a context into a model, write a two-line Modelfile and create a variant under a new name:

FROM qwen3.8:27b
PARAMETER num_ctx 32768

ollama create qwen3.8-32k -f Modelfile

Inside ollama run, /set parameter num_ctx 32768 changes it for the session, and /save qwen3.8-32k keeps the result as a model. Keep the override and the Modelfile in your setup script, because stopping a QuantaCloud instance deletes its disk, with every model and variant on it. For one API call, send it in options:

curl http://127.0.0.1:11434/api/chat -d '{
  "model": "qwen3.8:27b",
  "messages": [{"role": "user", "content": "Why is the sky blue?"}],
  "options": {"num_ctx": 32768},
  "stream": false
}'

Two rules apply to all three. A value above the model's trained context is lowered to it, with the warning requested context size too large for model in the log. And a request whose num_ctx differs from the one the model was loaded with makes Ollama reload the model, so two clients that send different values reload it back and forth. That is the reason I keep num_ctx out of requests and set one value on the server.

The OpenAI-compatible endpoint has no context field#

Requests to /v1/chat/completions cannot set the context. Ollama's docs say that the OpenAI API has no way of setting the context size for a model, and in 0.40.16 the only request fields the compatibility layer turns into model options are the stop sequences, max_tokens, the temperature, the seed, the two penalties and top_p. A client that speaks the OpenAI API, which includes most coding agents and anything built on the OpenAI SDK, gets whatever the model or the server is set to: the num_ctx in the model's parameters if there is one, then OLLAMA_CONTEXT_LENGTH, then the automatic default.

So for these clients either set the server variable, or create a variant with PARAMETER num_ctx as above and call it by its new name:

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8-32k", "messages": [{"role": "user", "content": "Say this is a test"}]}'

The Ollama API guide covers reaching that endpoint from another machine, and coding agents on your own GPU covers the context an agent needs. Ollama's docs put the floor for agents, web search and coding tools at 64,000 tokens.

What a longer context costs in VRAM#

Every token of context reserves KV cache on the GPU, and the cache for the whole context is allocated when the model loads, not as the conversation grows. The size per token comes from the model's architecture: 2 x the layers whose cache grows with context x KV heads x head size x 2 bytes for a 16-bit cache. The KV cache guide explains the formula. For three models from the Ollama library it gives (our calculation):

The spread is the point. llama3.3:70b grows a cache on all 80 of its layers: 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, so its full context costs 40.0 GiB, as much as its weights. gpt-oss:20b keeps a growing cache on 12 of its 24 layers and qwen3.8:27b on 16 of 64, so a long context is cheap for both. qwen3.8:27b at its full 262,144 tokens needs 16.0 GiB.

Three settings scale those figures. Parallel requests multiply them, because Ollama starts the model with the context times OLLAMA_NUM_PARALLEL: four slots of 32,768 tokens are 40.0 GiB for llama3.3:70b. OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves them, for every model on the server. And the engine adds its own buffers on top, so treat the table as the floor and read the real figure from the KV buffer size line in the log. The LLM VRAM calculator does the sum for other models, and Ollama GPU requirements lists which models fit which GPU.

Why a larger context quietly moves layers to the CPU#

Ollama does not refuse a context that is too large for the GPU. It hands the model to llama.cpp with the context fixed, and llama.cpp fits what it can: it never shrinks a context it was given, so it takes layers off the GPU until the rest fits with 1,024 MiB to spare, and runs the remaining layers on the CPU. For a model with a vision projector, Ollama raises that margin to the projector's size plus 1 GiB. Nothing fails. The model answers, only slower, and the PROCESSOR column of ollama ps shows a CPU share where it should say 100% GPU.

llama3.3:70b on a 48 GB card is the example. Its 43 GB download is about 40 GiB. A card that reports 44.99 GiB has about 44 GiB to fill once llama.cpp keeps its 1,024 MiB free. With 8,192 tokens of context, model and cache come to about 42.5 GiB before the engine's own buffers, which is tight. With 32,768 tokens they come to about 50 GiB, and part of the model moves to the CPU (our calculation). On a 96 GB RTX PRO 6000, which reports 95.59 GiB, the same model fits with its full 131,072 tokens at about 80 GiB.

One safety net exists, and it covers only the automatic default. If the load itself fails for lack of memory and nobody set the context, Ollama retries once with the next tier down, 32,768 tokens, or 4,096 if it was already at 32,768 or less, and logs the warning llama-server load failed; reducing automatic context and retrying once. A context you set yourself is never reduced. Ollama not using the GPU covers the other reasons for a CPU share.

Silent truncation, and how to notice it#

Ollama does not refuse a conversation that outgrows the context either. By default it makes it fit. In 0.40.16, /api/chat drops the oldest messages one by one until the prompt fits, keeping the system messages and always keeping the newest message. It does this whether Ollama renders the prompt itself or llama.cpp applies the model's own chat template. The only trace is a debug-level log line, which a server at the default log level never prints. The model simply stops knowing what was said earlier.

A single message that is longer than the context cannot be saved by dropping history, and what happens then depends on how the model's prompt is built. Where llama.cpp applies the model's own chat template, the request fails with an error that contains exceeds the available context size. Where Ollama renders the prompt itself, it keeps the first few tokens and the end, cuts the middle and logs a warning with the message truncating input prompt. The exception is the deepseek2 model family, where 0.40.16 turns context shift off: there Ollama returns an error that begins the prompt is longer than the context length currently available to the model.

These are the ways to notice:

The OpenAI-compatible /v1/chat/completions truncates the same way and has no field to turn it off. /v1/responses is the exception: since 0.40.16 it does not truncate unless the request says "truncation": "auto".

Open WebUI adds one more way to end up with a small context. Its num_ctx control is unset by default, but switching it on pre-fills 2048, and Open WebUI then sends that value with every request, where it overrides the server's setting. Open WebUI's docs name the symptoms: blank responses and tool calls that come back as raw JSON. Leave the control unset, or type a real value, as the Open WebUI setup guide describes.

Ollama context length FAQ#

What is the default context length in Ollama?

In 0.40.16 it is 4,096 tokens when the server finds less than 23 GiB of GPU memory, 32,768 from 23 GiB, and 262,144 from 47 GiB, capped at the model's trained context. Ollama's docs round the thresholds to 24 and 48 GiB. Older answers of 2,048 and 4,096 describe releases before 0.15.5.

How do I set the context length for every model?

Set OLLAMA_CONTEXT_LENGTH in the environment of the server and restart it. Under systemd that is an Environment line in sudo systemctl edit ollama. In Docker it is -e OLLAMA_CONTEXT_LENGTH=32768 when you create the container.

Does OLLAMA_CONTEXT_LENGTH override num_ctx?

No, it is the other way round. num_ctx in a request wins over num_ctx in a Modelfile, and both win over OLLAMA_CONTEXT_LENGTH. The variable is the default for models and requests that say nothing.

What is the largest context I can set?

The model's trained context, which ollama show prints as context length. A larger value is lowered to it. Below that limit the ceiling is GPU memory: the cache has to fit next to the weights.

Why does ollama ps show a smaller context than I set?

Either the value was above the model's trained context and Ollama lowered it, or you did not set one and the automatic retry after a failed load stepped down a tier. The log has a warning for both.

My rule for Ollama's context: decide it, do not inherit it. On a 48 GB card I set OLLAMA_CONTEXT_LENGTH=32768 for chat and 65,536 for agents, check the CONTEXT and PROCESSOR columns of ollama ps after the first request, and move to a larger GPU when the context I need pushes a CPU share into that column. Open WebUI on QuantaCloud runs Ollama with no setup, and a Bare Metal instance is the place to set it up your own way.