Self-hosting LLMs: sovereignty, quantization, serving and cost

For many organisations, the question is no longer whether to use large language models but where they run. Hosted APIs are the fastest way to start. Yet banks, brokers, healthcare providers and public bodies often cannot send client data to an external provider, face restrictions on cross-border data transfer, or simply need guaranteed availability and predictable costs. Open-weight models have become good enough that self-hosting is a realistic option for a wide range of tasks. This article explains when self-hosting LLMs pays off and how to do it well.

This article explains the trade-offs and the technical levers: model choice, quantization, serving and cost.

Why self-host

• Data sovereignty and confidentiality. Prompts often contain the most sensitive data a company has: client conversations, contracts, trading positions. On your own infrastructure the data never leaves your control, and you can prove it to auditors.
• Regulatory fit. Data residency rules, sector regulators and internal policies are easier to satisfy when processing stays in a known jurisdiction and data centre.
• Availability and independence. No exposure to a provider's outages, rate limits, price changes, deprecations or access restrictions.
• Customisation. You can fine-tune, merge adapters, control decoding and pin exact versions so behaviour does not change underneath you.
• Cost at steady volume. At high, steady utilisation, owned or reserved GPUs can be cheaper per token than API pricing. At low or spiky volume, they usually are not.

The honest counterpoint: you take on operations, security patching, capacity planning and model evaluation yourself, and the very largest proprietary models may still be stronger on the hardest reasoning tasks. Many organisations end up hybrid: self-hosted models for sensitive or high-volume work, external APIs for non-sensitive tasks where they add value.

Choosing a model

Select by task, language and size, then test on your own data:

• Size versus need. Models with 7–14 billion parameters handle classification, extraction, summarisation and RAG answering well when given good context. Models of 30–70 billion parameters or mixture-of-experts models are better for complex reasoning, long documents and agentic tool use.
• Language coverage. For Persian or other less-represented languages, test fluency, tokenizer efficiency (how many tokens a typical sentence needs) and instruction following. Poor tokenization can double cost and halve effective context.
• Licence. Open-weight licences differ on commercial use, user thresholds and redistribution. Have legal review the licence before you build on it.
• Context length. Long context is useful, but memory for the attention cache grows with it. Do not pay for 128k tokens if your RAG pipeline sends 8k.

LLM quantization: the biggest lever on hardware

Model weights are trained in 16-bit precision. A 70-billion-parameter model therefore needs about 140 GB just for weights, before the key–value (KV) cache. Quantization stores weights with fewer bits:

• 8-bit (INT8 or FP8) halves memory with negligible quality loss for most tasks. FP8 is natively accelerated on recent data-centre GPUs.
• 4-bit methods such as GPTQ, AWQ and the GGUF formats used by llama.cpp cut memory to roughly a quarter. A 70B model fits in about 40 GB of weights. Quality loss is usually small for chat and RAG, but measure it on your tasks, especially for reasoning, mathematics and non-English text.
• KV-cache quantization reduces memory per concurrent request, increasing how many users a GPU can serve at once.

Rule of thumb for weight memory: parameters × bits ÷ 8 bytes, plus 10–30% overhead, plus KV cache that scales with batch size and context length. Always benchmark accuracy before and after quantization on your evaluation set; a model that is 2% worse on English benchmarks can be noticeably worse in Persian.

Serving: throughput comes from the engine

The serving engine matters as much as the GPU. Modern engines such as vLLM, Hugging Face TGI, NVIDIA TensorRT-LLM and SGLang provide:

• Continuous batching: new requests join the running batch at each step instead of waiting for a whole batch to finish. This alone multiplies throughput.
• Paged attention: KV cache stored in blocks, like virtual memory, reducing fragmentation so more requests fit.
• Prefix caching: reuse computation for shared prompt prefixes such as system prompts and RAG instructions.
• Speculative decoding: a small draft model proposes tokens that the large model verifies in parallel, reducing latency.
• Tensor and pipeline parallelism: split large models across several GPUs.
• OpenAI-compatible APIs: so applications can switch between hosted and self-hosted backends with a configuration change.

For CPU-only or edge deployments, llama.cpp with GGUF quantization is a practical option for smaller models and low concurrency.

Operationally, treat the model server like any production service: health checks, autoscaling based on queue depth, request timeouts, per-tenant rate limits, structured logs and metrics such as time to first token, tokens per second and GPU memory use. Put an internal gateway in front for authentication, routing between models, usage accounting and content policies.

Estimating on-premise LLM cost

Think in cost per million tokens at your real utilisation:

1. Measure sustained throughput (tokens per second) for your typical prompt and output lengths at your target latency, on your chosen GPU and quantization.
2. Compute monthly capacity: tokens/s × 3,600 × 24 × 30 × expected utilisation (often 30–60% for interactive workloads).
3. Divide the monthly cost of hardware (depreciation over three years, or reserved cloud pricing), power, cooling, hosting and an honest share of engineering time by that capacity.

Utilisation dominates. A GPU busy 70% of the time can be very economical; the same GPU busy 5% of the time is expensive. Batch workloads such as nightly document processing, call transcription summaries or data enrichment are excellent for filling idle capacity. Smaller, fine-tuned models for narrow tasks often deliver the best cost per useful answer.

Security and governance

Self-hosting is only sovereign if it is secure. Isolate model servers in private networks, encrypt traffic and storage, control who can load or change models, verify model files from trusted sources with checksums, and scan dependencies. Log prompts and outputs with care: they contain sensitive data, so apply the same retention and access rules as the source systems. Keep an inventory of models, versions, licences and evaluation results.

A pragmatic roadmap

Start with one valuable, sensitive use case, such as RAG over internal policies or summarising client conversations. Benchmark two or three open-weight models at 8-bit and 4-bit on your own evaluation set. Deploy on a modern serving engine behind an internal gateway with an OpenAI-compatible API. Measure quality, latency and real utilisation for a month, then size hardware for the next use cases. Sovereignty, done this way, becomes an engineering capability rather than a constraint.