How much VRAM do you need for a local LLM?
Sizing · AI Node Works engineering team · Published
The short answer: Add three things: the model's weights (parameters × bits per weight ÷ 8), the KV cache for the context and number of simultaneous users, and runtime overhead. Then keep about 10% of the card free. Qwen3 32B at 4-bit needs about 23.9 GB for one user with an 8K context; serve ten users at 16K each and the same model needs about 66.3 GB, because every user adds KV cache.
The three parts of the estimate
- Weights: parameters × bits per weight ÷ 8. A 32-billion-parameter model at 16-bit is about 65 GB; at 4-bit, about 20 GB.
- KV cache: 2 × layers × KV heads × head size × context tokens × simultaneous users × bytes per value. It holds the attention keys and values for every token in every conversation being served.
- Overhead: the GPU runtime, compute buffers and memory fragmentation. The calculator allows 1 GB plus 4% of the rest, and 1 GB more for each extra GPU when a model is split.
Layers, KV heads and head size come from the model's published configuration file, not from its parameter count. Two models of the same size can need very different KV cache, because newer models share KV heads across attention heads to save memory.
Weights: precision matters most
Quantization stores each weight in fewer bits. It is the single biggest lever on memory, and 4-bit is the common default for running models locally. Quality falls slowly down to about 4 bits and faster below that.
| Format | Bits per weight | Qwen3 32B weights | Quality note |
|---|---|---|---|
| FP16 / BF16 | 16 | 65.6 GB | Full precision, as released |
| 8-bit (Q8_0 / FP8) | 8.5 | 34.8 GB | Near-lossless for most models |
| 6-bit (Q6_K) | 6.56 | 26.9 GB | Very close to 8-bit quality |
| 5-bit (Q5_K_M) | 5.69 | 23.3 GB | Good balance |
| 4-bit (Q4_K_M) | 4.85 | 19.9 GB | The common local default |
| 3-bit (Q3_K_M) | 3.91 | 16.0 GB | Noticeable quality loss |
Bits per weight for the GGUF k-quant formats follow llama.cpp's published figures and include the small per-block scaling data.
KV cache: context and users
The KV cache grows in a straight line with context length and with the number of conversations served at once. For one user with a short prompt it is small. For a team, it often becomes larger than the model.
| Simultaneous users | KV cache | Total estimate | Smallest launch configuration that fits |
|---|---|---|---|
| 1 | 4.3 GB | 26.1 GB | 1× AMD Radeon AI PRO R9700 |
| 4 | 17.2 GB | 39.5 GB | 2× AMD Radeon AI PRO R9700 |
| 10 | 42.9 GB | 66.3 GB | 1× NVIDIA RTX PRO 6000 Blackwell Workstation Edition |
| 20 | 85.9 GB | 111.0 GB | 2× NVIDIA RTX PRO 6000 Blackwell Workstation Edition |
Assumes every user is at full context at the same moment, which is the peak a system must survive. Serving runtimes such as vLLM allocate cache as conversations grow, so typical use is lower.
Two settings reduce it. Shorter maximum context is the first: most chat turns are far shorter than a model's limit. The second is a quantized cache. At 32K context, Qwen3 32B needs 8.6 GB of 16-bit cache for one user, or 4.6 GB with an 8-bit cache, at a small cost in quality that varies by model.
Overhead and headroom
Do not plan to fill the card. Drivers reserve memory, allocations fragment, and some runtimes briefly need extra memory while processing a long prompt. The calculator treats 90% of each card's memory as usable.
Splitting a model across two GPUs works in the common runtimes, but each GPU carries its own overhead, and both cards need the same software platform: CUDA for NVIDIA, ROCm for AMD.
Worked examples
| Model and setup | Weights | KV cache | Total | Smallest launch configuration that fits |
|---|---|---|---|---|
| Qwen3 8B, 4-bit (Q4_K_M), 8K, 1 user | 5.0 GB | 1.2 GB | 7.4 GB | 1× AMD Radeon AI PRO R9700 |
| Qwen3 14B, 4-bit (Q4_K_M), 16K, 1 user | 9.0 GB | 2.7 GB | 13.1 GB | 1× AMD Radeon AI PRO R9700 |
| Qwen3 32B, 8-bit (Q8_0 / FP8), 8K, 1 user | 34.8 GB | 2.1 GB | 39.5 GB | 2× AMD Radeon AI PRO R9700 |
| DeepSeek-R1 Distill Llama 70B, 4-bit (Q4_K_M), 16K, 1 user | 42.8 GB | 5.4 GB | 51.1 GB | 2× AMD Radeon AI PRO R9700 |
| DeepSeek-R1 Distill Llama 70B, 8-bit (Q8_0 / FP8), 16K, 1 user | 75.0 GB | 5.4 GB | 84.6 GB | 1× NVIDIA RTX PRO 6000 Blackwell Workstation Edition |
| Qwen3 30B-A3B (MoE), 4-bit (Q4_K_M), 32K, 5 users | 18.5 GB | 16.1 GB | 37.0 GB | 2× AMD Radeon AI PRO R9700 |
Estimates of memory only, with a 16-bit cache. They say nothing about speed, and a runtime may still need a specific version or setting to load a model.
What VRAM does not tell you
- Speed. Generating text one token at a time is usually limited by memory bandwidth, and reading a long prompt by compute. Two cards with the same memory can differ several-fold in speed. We publish speeds only after measuring them.
- Mixture-of-experts models. Only a few experts run for each token, which makes them fast, but every expert must still be in memory. Size them by total parameters.
- Software support. A model that fits may still need a recent runtime, and AMD cards need a runtime built for ROCm.
Sources Last verified 27 Oct 2026
- Hugging Face Transformers: How caching works
- Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180)
- llama.cpp: quantize tool README (bits per weight by format)
- vLLM documentation
- Hugging Face: Qwen/Qwen3-32B model card and configuration
- Hugging Face: Qwen/Qwen3-8B model card and configuration
- Hugging Face: Qwen/Qwen3-14B model card and configuration
- Hugging Face: Qwen/Qwen3-30B-A3B model card and configuration
- Hugging Face: DeepSeek-R1-Distill-Llama-70B model card and configuration