An on-premise LLM is a model served from hardware you own, in a building you control. It is one of four places a model can run, and the only one that gives you physical custody of the machine. The other three are someone else's API, a GPU VM in a shared cloud, and dedicated single-tenant servers in a data center. Before you buy servers, check which of two requirements sits behind the word: prompts that no model provider sees, or hardware that nobody else shares. A rented GPU gives you the first, and a dedicated build gives you the second, without a server room.
On-premise and private are not the same thing#
On-premise describes where the hardware is. Private describes who it is for. NIST's definition of a private cloud makes the difference explicit: infrastructure "provisioned for exclusive use by a single organization", which "may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises".
So a private LLM does not have to be on your premises. It has to be a model whose weights, prompts and outputs stay on infrastructure used by you alone. Air-gapped is stricter again: no network path to the outside at all, which in practice means hardware you can walk up to.
The four places a model can run#
Each step to the right gives you more control and takes more money and time to start.
| Someone else's API | GPU VM in a shared cloud | Dedicated servers in a data center | Your own hardware, on premises | |
|---|---|---|---|---|
| Where the model runs | The provider's servers | A virtual machine you rent by the hour | Single-tenant servers built for you | Servers you own, in your building |
| Where prompts and outputs go | To the provider, under its terms | To your VM and no further, unless you send them on | To your servers | Nowhere outside your building |
| Which models | The ones the provider offers | Any open-weight model that fits the GPU | Any open-weight model | Any open-weight model |
| Who runs the hardware | The provider | The cloud | Set out in the written quote | You |
| Cost to start | A price per token, with little or nothing up front | Prepaid credit, from $5 on QuantaCloud, then an hourly rate | A written quote for a term | The purchase price, plus a room with the power and cooling for it |
| Time to start | Minutes | Most single-GPU VMs on QuantaCloud are running in about 3 minutes (median) | The lead time in the quote | Delivery, installation and commissioning |
| Network access | Over the internet | Over the internet, to the VM's public IP address | As set out in the brief and the written quote | Whatever you build, including no outside connection at all |
The second column is where most self-hosting starts, because it costs a few dollars to find out whether a model does the job. Self-hosting an LLM walks through that path, and LLM inference sizes the GPU for it.
What on-premise really requires#
Four things, and the hardware is the easiest of them to buy.
Hardware, by model size
GPU memory decides the hardware: the weights have to fit, with room left for the context you serve. These are the weights alone, read from each model's repository on 2026-11-12. For the largest three, the budget is 92% of the memory the GPUs report, which is what vLLM takes by default.
| Model | Weights | What holds them | Sizing guide |
|---|---|---|---|
| gpt-oss-20b | 13.8 GB | Its maker says it runs "within 16GB of memory" | gpt-oss GPU requirements |
| gpt-oss-120b | 65.3 GB | One 80 GB GPU, by its maker's statement, or one 96 GB card | gpt-oss GPU requirements |
| Llama 3.3 70B, FP8 | 72.7 GB | One 141 GB H200 with a full 128k-token context, 115.6 GB in all | How much VRAM you need |
| Kimi K2.6 | 595 GB, 554 GiB | More than the 517 GiB budget of four 141 GB H200s, so an 8-GPU server | Kimi K3 hardware requirements |
| DeepSeek V4 Pro | 893 GB, 831 GiB | An 8-GPU server: a budget of 1,033 GiB with H200s, 1,977 GiB with B300s | DeepSeek V4 GPU requirements |
| Kimi K3 | 1,561 GB, 1,454 GiB | More than the budget of an 8-GPU H200 or B200 server (1,033 and 1,318 GiB). An 8-GPU B300 server's is 1,977 GiB | Kimi K3 hardware requirements |
The table splits on-premise into two different projects. Models up to about 70B parameters fit one or two GPUs, which a workstation or a single server can hold. NVIDIA's desk-side DGX Spark is one machine for that class: it has up to 128 GB of unified memory, which NVIDIA says is enough for inference on models of up to 200 billion parameters, and the DGX Spark price guide compares it with renting. The frontier open models are a different matter: hundreds of gigabytes of weights mean an 8-GPU server, and that is data-center equipment. The LLM VRAM calculator does the sum for your own model and context, open-weight LLMs by GPU memory sorts the current models the same way, and AI server cost prices each class of machine.
Power and cooling
An 8-GPU server needs data-center power and cooling. NVIDIA lists the DGX B300 at about 14 kW and the DGX B200 at about 14.3 kW at maximum. Run flat out all month, 14 kW is 10,220 kWh, which costs $1,478.83 at the US average commercial price of 14.47 cents per kWh in August 2026 (our calculation: 14 x 730 x $0.1447). That is the electricity for the server alone, at its rated draw, before the cooling that has to remove the same 14 kW as heat.
Smaller machines have the same problem in miniature. The RTX PRO 6000 Blackwell Server Edition is a passive card of up to 600 W that needs 43 to 125 CFM of airflow from the chassis, depending on inlet temperature, so it belongs in a server with the fans for it, not in a desktop case.
People
Someone has to own the machine when it breaks. SemiAnalysis's October 2024 guide to running GPU clusters lists the most common failures: flapping transceivers, GPUs "falling off the bus" and GPU memory errors. On premises, spotting those, swapping the part, and keeping drivers, firmware and the operating system patched is your team's job, at whatever hour it happens.
The software stack
The software is the same wherever the GPU sits, which is the good news: what you learn on a rented VM carries over unchanged.
| Layer | Tool | Latest stable release on 2026-11-12 | Guide |
|---|---|---|---|
| An API for many overlapping requests | vLLM | v0.33.0, 2026-11-02 | Deploy vLLM with Docker |
| A model server for a small team | Ollama | v0.40.16, 2026-11-09 | Open WebUI and Ollama setup |
| GGUF files, with CPU offload when memory is short | llama.cpp | v0.9.0, 2026-11-04 | llama.cpp on a GPU |
| A chat front end | Open WebUI | v0.13.3, 2026-11-08 | Open WebUI |
Around those you still need TLS and a key in front of the endpoint, monitoring, and a plan for updates.
What it costs against renting#
Owning wins only when the hardware stays busy. A February 2025 comparison by a data-center research institute, for 8-GPU H100 systems in a US colocation facility, found owning cheaper than hyperscaler on-demand prices at 22% utilization or above, and cheaper than GPU-focused clouds only at 66% or above, a level it said "would be difficult for all but the most efficient AI deployments".
The share of the world's data center capacity on premises has been falling for years. Synergy Research Group puts enterprise on-premise data centers at 32% of the world's data center capacity at the end of 2025, down from 56% in 2018, and forecasts 19% by 2031. It also notes the counter-current this page is about: on-premise capacity "is receiving something of a boost thanks to GenAI applications and GPU infrastructure".
For a sense of the rented side, one RTX PRO 6000 Blackwell, a 96 GB card that holds gpt-oss-120b, cost $2.39 an hour on QuantaCloud on 2026-11-12 (today: $2.16/GPU-hr). Left running all month that is $1,744.70, and used 8 hours a day on 22 working days it is $420.64 (our calculation: 730 x $2.39 and 176 x $2.39). Today's prices for the GPUs that hold the first three models in the table, where the H200 NVL shows a price only when the catalog lists it:
| GPU | Memory | From | Available now |
|---|---|---|---|
| RTX A6000 | 48 GB | $0.48/GPU-hr | Yes |
| L40S | 48 GB | $1.09/GPU-hr | Yes |
| A100 PCIe 80GB | 80 GB | $1.48/GPU-hr | Yes |
| RTX PRO 6000 Blackwell | 96 GB | $2.16/GPU-hr | Yes |
| H200 NVL | 141 GB | $3.43/GPU-hr | Yes |
Prices checked 14 Nov 2026, 07:01 UTC
Reserved vs on-demand has the break-even formula, and it works for hardware you buy as well.
Where QuantaCloud fits, and where it does not#
QuantaCloud covers the two middle columns, and it is not on-premise. Here is the position exactly.
On demand, every QuantaCloud GPU instance is a virtual machine in a US region: us-east-1 in Virginia, or us-midwest-1, us-midwest-2 or us-midwest-4 in the Midwest. It has a public IP address, you log in over SSH as ubuntu, and you run the model server yourself, so prompts go from your client to your VM and are processed there. Stopping the instance terminates it and deletes its disk. There is no managed model endpoint, and QuantaCloud is not SOC 2 or HIPAA certified. Security covers access, isolation and data handling in detail.
Reserved builds are dedicated hardware built to order. We order and build the hardware to your spec, as dedicated GPU servers that are single-tenant and bare metal, or as a cluster, and the configuration, lead time and terms come back in writing before you commit. Where the servers sit, who can reach them and how the network is laid out belong in the brief.
What QuantaCloud does not offer is the fourth column. The hardware is not in your building, and an on-demand instance is not air-gapped. If your requirement is physical custody of the machine, a dedicated build does not meet it, and neither does anything else in a data center. If it is a certified provider, QuantaCloud does not meet it today either. I would rather say both here than in a sales call.
Which one to choose#
Match the requirement to the least expensive column that meets it.
| What you actually need | Where to run the model |
|---|---|
| To find out whether an open model does the job | A GPU VM by the hour |
| Prompts and outputs that no model provider sees | A GPU VM by the hour, with your own model server on it |
| Hardware nobody else shares, your own operating system, a fixed spec for months | Dedicated servers, built to order |
| A model with hundreds of gigabytes of weights | A dedicated 8-GPU server or a cluster |
| A certified processor, such as SOC 2 or HIPAA | Not QuantaCloud today. A certified provider, or your own audited environment |
| Physical custody of the machine, or an air gap | Your own hardware, on your premises |
On-premise LLM questions#
What is an on-premise LLM?
A large language model whose weights are loaded and served on hardware the organization owns, in its own building or data center space, instead of being called through a provider's API. It only works with open-weight models, because those are the ones you can download.
Is a private LLM the same as an on-premise LLM?
No. Private means the infrastructure is used by one organization, and NIST's definition says a private cloud "may exist on or off premises". A model on dedicated servers in a data center is private without being on-premise.
Can an LLM run air-gapped?
Yes, on hardware you control. Open-weight models are files, so you can carry the weights in and serve them with no outside connection. Test the whole stack with the network unplugged before you rely on it. A QuantaCloud on-demand instance is not air-gapped: it has a public IP address and you reach it over SSH.
What hardware does an on-premise LLM need?
Enough GPU memory for the weights plus the context. That is one GPU of 48 to 141 GB for models up to about 70B parameters, depending on the precision and the context length, and an 8-GPU server for models with hundreds of gigabytes of weights, as the sizing table above shows.
Is QuantaCloud an on-premise or private AI provider?
It is a GPU cloud in US regions. On demand it rents GPU virtual machines by the hour. For private AI infrastructure it builds dedicated, single-tenant servers and clusters to order in a data center, not on your premises.
Is QuantaCloud SOC 2 or HIPAA certified?
No. QuantaCloud holds neither certification today.
My rule: write down the requirement that made you say on-premise, then take the least expensive option that meets it. If it is "no model provider may see these prompts", rent a GPU and run the model yourself this week. If it is "nobody else may share the machine", send a brief for a dedicated build. If it is custody or an air gap, buy the servers, and budget for the power, the cooling and the person who gets paged.