Local LLM servers
A local LLM server runs language models on hardware you control. Its size depends on four things together: the model's size, the quantization you run it at, the context length, and how many people use it at once. Get those right and the hardware choice follows.
When it fits
The right choice when
- Prompts and documents should stay on infrastructure you manage.
- Usage is steady enough that owning hardware beats paying per token.
- You want to choose your own models and update on your own schedule.
- A team, not just one person, will use it.
Look elsewhere when
- You need the largest frontier models, which are not available to run locally or need a data centre.
- Usage is occasional or spiky; cloud inference may cost less.
- No one will look after updates, access control and backups.
What decides the configuration
The variables we work through with you before recommending anything.
| Criterion | Why it matters | What we check |
|---|---|---|
| Model size | Weights are the floor: parameters × bits per weight. | Which models, and whether a smaller one does the job. |
| Quantization | 4-bit needs about a quarter of the memory of FP16, with some quality loss. | Quality needs for your tasks. |
| Context length | Each token held in context costs KV-cache memory. | Longest prompts and documents you will send. |
| Concurrent users | Every simultaneous conversation needs its own cache. | Peak simultaneous users or agents. |
| Runtime | Ollama, llama.cpp and vLLM differ in batching, memory use and GPU support. | The runtime that fits your model, GPU and scale. |
| Access | A server people reach over the network needs authentication. | How users connect, and who administers it. |
Worked sizing examples
Illustrative estimates from the VRAM calculator's method: FP16 KV cache, every user at full context. Real usage is often lower with paged attention; we validate before quoting.
| Model | Settings | Estimated memory | Smallest catalogue fit |
|---|---|---|---|
| Qwen3 8B | 4-bit (Q4_K_M), 8K context, 1 user | 7 GB | 1× AMD Radeon AI PRO R9700 |
| Qwen3 32B | 4-bit (Q4_K_M), 16K context, 5 users | 44 GB | 2× AMD Radeon AI PRO R9700 |
| DeepSeek-R1 Distill Llama 70B | 4-bit (Q4_K_M), 32K context, 10 users | 157 GB | 2× NVIDIA RTX PRO 6000 Blackwell Workstation Edition |
| Qwen2.5 72B | 8-bit (Q8_0 / FP8), 16K context, 20 users | 193 GB | 4× NVIDIA RTX PRO 6000 Blackwell Server Edition |
Software such as Ollama, llama.cpp, vLLM and Open WebUI are examples we work with, not a promise that every model runs on every combination. We confirm the exact model, runtime and GPU together. We do not publish speed figures until we have measured them.
Run your own numbersReference builds
Configuration examples to start from. Every one is validated against your workload before quoting.
GPUs we build with
Related guides
Common questions
Can it run the model I want?
If the model's weights are available to download and its memory needs fit the hardware, generally yes. Use the calculator for a first estimate, then we check the exact runtime and GPU.
How many tokens per second will I get?
It depends on the GPU, model, quantization, runtime and batching. We only quote figures we have measured on the configuration, never estimates presented as results.
Is local cheaper than cloud?
For steady, high use it can be; for occasional use, often not. We can compare your expected usage against cloud pricing during the consultation.
Do I need a GPU server or will a tower do?
For a small team, a tower often does. Beyond roughly 20 simultaneous users or for very large models, a server is the better fit.
Size a local LLM server for your team
Give us your models, context and users. We will size the system, choose the runtime and quote it.