How to Estimate GPU Requirements for Local LLM Workloads

GPU capacity is usually the single largest line in an on-premises AI budget, and it’s the one most often gotten wrong in both directions. Under-provision, and the platform stalls under real load — requests queue, latency climbs, and users abandon it. Over-provision, and expensive hardware sits idle while finance asks why the capital was spent. Both mistakes come from the same root cause: sizing GPUs from a vague sense of “we need something big” rather than from the actual drivers of memory and compute demand.

This is a practical guide to estimating GPU requirements from the variables that actually determine them — model size, precision, context length, and concurrency — so infrastructure and finance teams can plan capacity with a defensible number instead of a guess. (New to the topic? Start with what a local LLM is, then come back for the arithmetic.)

What actually consumes GPU memory

The instinct is to equate “GPU requirement” with “model size.” Model weights are only one of three consumers, and treating them as the whole picture is exactly how sizing goes wrong.

• Model weights. This is the floor. The memory the weights occupy is approximately the parameter count times the bytes per parameter, which precision sets: roughly two bytes per parameter at 16-bit, about one byte at 8-bit, and roughly half a byte at 4-bit. A model’s parameter count and the precision you serve it at together fix this baseline.
• The KV cache. As the model generates, it stores intermediate state — the key-value cache — for every token in context. This grows with context length and with the number of requests being served at once. At high concurrency with long contexts, the KV cache can rival or exceed the weights themselves, and it’s the part teams most often forget to budget for.
• Serving overhead. The inference runtime, activations, and fragmentation all consume additional memory. It’s not the dominant term, but leaving no margin for it is how a model that “should fit” fails to load.

The takeaway: weights give you a floor, but real capacity is set by weights plus a KV cache that scales with how you actually use the model.

The four variables that drive your number

Every credible GPU estimate comes down to four inputs. Get these written down and the arithmetic follows.

1. Model size

Parameter count is the primary driver of weight memory. This is also the strongest argument for not defaulting to the largest available model: a smaller model that meets the accuracy bar for a task uses a fraction of the memory and compute. Many enterprise tasks are served well by small language models, and matching model size to task is the highest-leverage cost decision available.

2. Precision and quantization

Serving at 8-bit or 4-bit rather than 16-bit can cut weight memory enough to change how many GPUs you need. The cost is a possible accuracy impact that depends on the model and the task — sometimes negligible, sometimes not. Treat quantization as a lever to validate against your own evaluation set, not a free win to assume.

3. Context length

Longer contexts mean a larger KV cache per request. A workload that routinely processes long documents has a fundamentally different memory profile from one handling short queries, even on the identical model. Size context to what the workload genuinely needs rather than setting the maximum “to be safe” — that safety margin is paid for in GPU memory on every concurrent request.

4. Concurrency

How many requests must be served simultaneously multiplies the KV cache demand and drives compute throughput. This is the variable most disconnected from the model itself and most connected to your actual usage, which is why honest concurrency estimates matter more than headline model specs. Ten users occasionally querying an assistant and a thousand employees hitting an agent platform at 9am are different infrastructure problems.

A practical estimation sequence

Rather than a formula that pretends to more precision than the inputs support, work through this sequence:

1. Fix the weight floor. Multiply parameter count by bytes-per-parameter at your intended precision. This is the minimum before serving anything.
2. Add realistic KV-cache headroom. Estimate it from your expected context length and target concurrency — not from the maximum either could theoretically reach. This is where deliberately-chosen numbers beat worst-case defaults.
3. Add serving margin. Leave room for runtime overhead and fragmentation so models load reliably under load, not just in a clean benchmark.
4. Map to physical GPUs. Compare the total against the memory of the GPUs you can actually procure. If it exceeds a single card, you’re into multi-GPU serving, which adds interconnect and sharding considerations to the plan. The Kimi K3 deployment notes walk through that arithmetic for a trillion-parameter-class MoE model.
5. Validate empirically. Load the model, drive it at target concurrency and context, and measure. An estimate sets the budget; a measurement confirms it. Treat the calculated number as a starting hypothesis, not a final answer.

The goal isn’t a single perfect figure — it’s a defensible range you can put in a capacity plan and then confirm against real behavior.

Sizing by number of users

Most sizing requests start from a headcount rather than a model. Two threads on the Level1Techs forum show the pattern: one asked how to serve about 200 users in an academic setting, with up to 40 at once, and another scoped hardware for five people running an agentic RAG workflow over tax documents. The table summarises the worked examples below. Every figure is a memory estimate from stated assumptions, not a benchmark.

To run your own numbers through the same method, use the LLM VRAM calculator. GPU memory sizes come from NVIDIA’s product pages for the H100, H200, L40S and RTX PRO 6000 Blackwell Server Edition (verified October 2026). Choosing the server around those GPUs is covered in the on-premise AI server guide.

From headcount to GPU memory

1. Turn users into concurrent sequences. Assume 10–20% of named users send a request at the busiest moment. The reference architecture’s sizing tiers work out at roughly 10%, and the 200-user forum thread planned for 20%. In a team of under ten people, assume everyone can be active at once.
2. Work out the KV cache per sequence. KV bytes per token = 2 × layers × key-value heads × head dimension × bytes per value, where bytes per value is 2 at 16-bit and 1 at FP8 (NVIDIA’s derivation). Multiply by the context length you allow. Grouped-query attention means a model stores far fewer key-value heads than query heads, so take the number from its config.json.
3. Add the weights. Parameters × bits per weight ÷ 8.
4. Leave headroom. Divide the total by 0.9 so about a tenth of GPU memory stays free for activations and the runtime. By default, vLLM lets its engine use 92% of each GPU’s memory (verified October 2026).

The examples use three published model shapes:

Worked example: 5 users

• Assumptions: five named users who can all be active together, document analysis with 16,384-token contexts, and a 32B dense model with Qwen3-32B’s shape at FP8.
• Weights: 32.8B × 8 ÷ 8 ≈ 32.8 GB.
• KV cache: 262,144 bytes × 16,384 tokens ≈ 4.3 GB per sequence, so ≈21.5 GB for five.
• Total: (32.8 + 21.5) ÷ 0.9 ≈ 60 GB.
• Result: one 80 GB H100 or one 96 GB RTX PRO 6000. With 4-bit weights (about 16–20 GB once format overhead is included) and an FP8 KV cache (≈10.7 GB), the total falls to roughly 30–34 GB, which fits a single 48 GB L40S.

Worked example: 50 users

• Assumptions: 50 named users with 20% active at peak, so 10 concurrent sequences, 8,192-token contexts, and a 70B dense model with Llama 3’s shape.
• KV cache: ≈2.7 GB × 10 ≈ 27 GB at 16-bit.
• With 4-bit weights: (35–40 GB + 27 GB) ÷ 0.9 ≈ 69–74 GB. That is one 80 GB H100 with little to spare, or one 96 GB RTX PRO 6000.
• With FP8 weights: (70 GB + 27 GB) ÷ 0.9 ≈ 108 GB. That is one 141 GB H200, or two 80–96 GB GPUs with tensor parallelism.
• Result: one or two GPUs. Add a second server if the service has to survive a hardware failure.

Worked example: 200 users

• Assumptions: the forum thread’s own figure of 40 concurrent sequences, and 8,192-token contexts.
• With the 7–14B models that thread considered, using Qwen3-8B’s shape at 16-bit: weights ≈16.4 GB, KV cache ≈1.2 GB × 40 ≈ 48 GB, so (16.4 + 48) ÷ 0.9 ≈ 72 GB. One 80 GB or 96 GB GPU holds it. An FP8 KV cache cuts the total to about 45 GB, which squeezes onto a 48 GB L40S with little headroom.
• With a 70B model at FP8: the KV cache is ≈2.7 GB × 40 ≈ 107 GB at 16-bit, or ≈54 GB at FP8. The totals are (70 + 107) ÷ 0.9 ≈ 197 GB, which means two H200 or four H100, or (70 + 54) ÷ 0.9 ≈ 137 GB, which fills one H200 to its limit; two 80–96 GB GPUs are the safer choice.
• Result: from one GPU for an 8B model to two to four data-centre GPUs for a 70B model. Replies in that thread also recommended vLLM over Ollama at 40 simultaneous users, since vLLM is built to batch many concurrent requests.

What changes the answer

• Context length. The KV cache grows in proportion to it. At 32,768 tokens, 40 sequences on the 70B model need about 430 GB of 16-bit KV cache, four times the 8,192-token figure.
• Attention design. Hybrid-attention models keep a KV cache in only some of their layers. Qwen3.8-27B uses full attention in every fourth layer (16 of 64), with 4 key-value heads of dimension 256, so it stores about 64 KiB per token at 16-bit, a quarter of Qwen3-32B’s figure. Its other layers hold a fixed-size state per sequence instead.
• Agents. One agent task can make several model calls in a row, each with a longer context, so count running agents as extra concurrent sequences.
• Speed. These steps size memory only. Throughput depends on memory bandwidth, batch size and the serving engine, so confirm it with a load test on your own prompts before you buy.

Where teams waste GPU budget

A few patterns account for most over-spend, and all are avoidable at planning time:

• Defaulting to the largest model everywhere. Routing every request to a flagship model, including trivial ones, burns capacity on tasks a smaller model would handle. Model routing that directs lightweight requests to lightweight models is one of the most effective ways to lower the GPU bill without lowering quality.
• Provisioning for a theoretical peak. Sizing every GPU for the busiest imaginable minute leaves hardware idle almost always. Plan a sustainable baseline with peak headroom, not peak-everywhere.
• Ignoring quantization. Skipping a validation pass on 8-bit or 4-bit serving can mean buying more GPUs than the workload actually requires.
• Forgetting the KV cache. Sizing only for weights produces a plan that works in a demo and falls over under concurrent production load.

GPU capacity is one input to total cost of ownership, alongside storage, platform staff, and operations — the fuller picture is in the On-Premise AI Platform Cost and TCO Guide.

How VDF AI helps right-size GPU capacity

VDF AI is designed so that GPU capacity maps to actual workload rather than to the largest model on the roster. VDF AI Networks lets teams register multiple local models at different sizes and precisions, and routes each request to the appropriate one, so lightweight tasks don’t consume flagship-model capacity. Because the platform runs entirely on-premises, capacity planning is a direct function of your own workload — models, context, and concurrency you control and can measure — not an opaque consumption bill. That makes the estimate you build at planning time something you can verify in production and adjust deliberately, which is the difference between a capacity plan and a hopeful purchase order.

Start from the four variables, calculate a defensible range, then validate it under real load. GPU sizing done this way turns the largest line in the budget from a guess into a number you can defend to finance.