Most on-premises AI budgets are decided by one number: how much GPU memory the platform needs. That number is not set by the model you chose. It is set by the numeric format you serve it in.
The same open-weight model can occupy roughly two bytes per parameter in 16-bit precision, about one byte in an 8-bit format, and around half a byte in a 4-bit weight-only format. On a fixed hardware footprint, that difference decides whether a department gets one model or four, whether a second model can be held resident for failover, and how many concurrent sessions a node can serve before requests start queuing.
Quantization is therefore an infrastructure procurement decision that is usually made by an engineer editing a launch flag. This is a guide to making it deliberately.
What the numeric format actually changes
Three things move together when precision drops.
Memory footprint. Weights shrink roughly in proportion to bits per parameter. This is the effect that shows up in the hardware quote.
Memory bandwidth pressure. Decoding a token requires reading the weights. Smaller weights mean fewer bytes read per token, which is why low-precision formats often improve latency on bandwidth-bound single-stream generation.
Available key-value cache. Whatever memory the weights do not occupy is available for the KV cache that holds active sessions. In practice this is the effect that matters most for a shared enterprise platform, because concurrency is what determines whether one node serves a business unit or three.
The trade is accuracy, and it is not uniform. A format that is invisible on summarisation can be visible on numeric extraction from a scanned invoice, or on a long tool-calling chain where one malformed argument ends the run.
The formats worth evaluating
| Format | Typical use | What to watch |
|---|---|---|
| BF16 / FP16 | Reference baseline for evaluation | Highest memory cost per model |
| FP8 (weights and activations) | Default on accelerators with native FP8 support | Requires recent GPU generations |
| INT8 (weights and activations) | Equivalent step on older accelerators | Integer scaling is more sensitive to outlier activations |
| 4-bit weight-only (AWQ, GPTQ) | Fitting larger models on smaller cards; many resident models | Calibration data quality; task-specific regressions |
A large study of the Llama-3.1 family, spanning more than 500,000 evaluations across academic benchmarks and real-world tasks, reported FP8 weight-and-activation quantization as effectively lossless across model sizes, INT8 at roughly 1-3% degradation, and 4-bit weight-only quantization as more competitive than expected. The same work found 4-bit weight-only most cost-efficient for synchronous single-stream setups, while 8-bit weight-and-activation formats performed better under asynchronous continuous batching.
Read that as a prior, not a result. It tells you which experiments are worth running first. It does not tell you how a 4-bit build will handle your policy documents, your German-language claims correspondence, or your JSON schema.
Serving runtimes have made the experiment cheap. vLLM supports FP8, AWQ, GPTQ, compressed-tensors and a long list of other schemes, and FP8 can be produced through several underlying algorithms, so a team can hold a baseline and two candidates and compare them on the same hardware.
Evaluate on your own tasks, not on leaderboards
A quantization decision needs a task-level evidence set before it goes near production traffic. Build it once and reuse it for every model and format change:
- A frozen prompt set per workload — retrieval answers, document extraction, classification, tool-calling, and any workload with a strict output schema.
- Deterministic scoring where possible. Schema validity, field-level extraction accuracy and tool-argument correctness are countable. Reserve human review for the cases that are not.
- Long-context cases. Degradation often appears first at the far end of the context window, not on short prompts.
- Non-English cases, if the platform serves them. Multilingual behaviour is not always preserved evenly across formats — the same caution that applies to multilingual private RAG.
- A latency and concurrency profile, measured at the concurrency the platform is actually sized for rather than at batch size one.
The comparison that matters is the same prompt set against the same runtime on the same hardware, changing only the numeric format. Anything else compares two variables at once.
Where the memory really goes
Quantizing weights does not remove the rest of the budget. When sizing a node, account for:
- Weights, in the chosen format;
- KV cache, which scales with context length and concurrent sessions and is often the binding constraint on a shared platform;
- Activation and runtime overhead for the serving engine;
- Embedding and reranking models, if they share the same accelerator — see embedding models and rerankers for private RAG;
- Failover headroom, so losing one node does not take the service below its committed concurrency.
This is why quantization should be decided alongside, not before, GPU requirement estimation and admission control. A 4-bit build that frees memory for KV cache can raise usable concurrency more than a hardware upgrade would, and it can also mask the fact that the platform has no queueing policy at all.
Treat a quantized build as a governed artifact
The common failure is not accuracy loss. It is losing track of which build produced an output.
A quantized model is a new artifact. It has its own file digest, its own calibration dataset, its own producing toolchain version, and its own evaluation record. If it enters production as a runtime flag rather than a registered version, the audit trail records a model name that no longer identifies what actually ran.
Practical controls:
- Register the quantized build as a distinct version with its own digest, and carry it through the model artifact supply chain like any other imported weight file.
- Record the calibration data used to produce it, including its sensitivity classification. Calibration samples are enterprise data.
- Store the evaluation report against that version, not against the base model, following the same discipline as local LLM versioning and upgrades.
- Log the serving format in each execution record so a reviewer can reconstruct which numeric build answered a given request.
- Re-run evaluation on upgrade. A new base model version invalidates the prior quantized comparison.
Under the EU AI Act’s provider and deployer documentation expectations, “which model version handled this decision” is a question that needs an answer from the log, not from an engineer’s memory. Precision format is part of that identity.
How this fits a routed platform
Quantization is most useful when it is not a single global choice. A platform that can route requests across several local models can hold a 4-bit build for high-volume classification and summarisation, and reserve a higher-precision build for extraction and approval-bearing steps, with the routing rule recorded rather than implied.
VDF AI runs local models, private retrieval and agent orchestration inside the customer’s environment. VDF AI Agents call whichever registered local endpoint policy assigns to a step, and VDF AI Networks records the model, tool and fallback decisions taken across a workflow — so the serving build behind each step stays visible after the fact.
The engineering rule is simple: quantize to buy concurrency, prove it on your own tasks, and register the result as a version rather than a flag.
Sources and further reading
- “Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization
- vLLM quantization documentation
- LLM Compressor: choosing a compression scheme
- How to estimate GPU requirements for local LLM workloads
- Selecting a local LLM for enterprise workloads
Sizing GPU capacity for an on-premises AI platform? Book a VDF AI architecture review to set precision targets per workload, build the evaluation set, and plan capacity against real concurrency.