Enterprise Local LLM Deployment Handbook

Model size alone does not determine capacity. Plan the full memory envelope: model weights (parameters × bits per weight ÷ 8 as a lower bound), KV cache (roughly 2 × layers × KV heads × head dimension × active tokens × bytes per element), runtime activations and graphs, plus explicit headroom for fragmentation and bursts. These are planning formulas only — grouped-query attention, cache precision, paging and prefix caching change the result materially, so benchmark peak allocated memory before committing to any capacity or SLO.

No. Local processing can support data minimisation, privacy by design and sovereignty objectives, but compliance still depends on purpose, roles, lawful basis, data rights, security, documentation, oversight and sector obligations. Local deployment narrows some external data paths while insiders, misconfiguration, vulnerable dependencies, over-privileged retrieval and ungoverned outputs remain material risks. For the EU AI Act, determine whether the system or its downstream use is in scope and apply the current timeline.

Six steps: effective demand equals peak admitted request rate × measured tokens per request by profile; serviceable throughput is the lowest throughput that still meets quality and percentile latency SLOs; required replicas equal effective demand ÷ serviceable throughput, rounded up per model and priority pool; add resilience capacity for the agreed failure domain; add explicit growth and operational headroom rather than hiding it in an optimistic benchmark; then revalidate whenever model, runtime, quantisation, context, prompt, GPU or workload mix changes.

No. A downloadable checkpoint may still carry use restrictions, redistribution terms or missing training-data freedoms. The Open Source Initiative’s Open Source AI Definition is a useful reference point, but legal review must assess the actual terms supplied with each release — covering weights, code, tokenizer and dataset terms, attribution, use restrictions, redistribution and downstream duties.

It is a release change and should be governed as one. Record why the method was chosen, what changed (calibration data, group size, compute and cache precision, conversion tooling), what regressed across task, language, long-context, grounding, safety and structured-output comparisons, what runtime and accelerator generations are supported, whether the result is reproducible from a pinned toolchain with signed hashes, and whether the prior approved release is retained and load-tested within the recovery objective.

The retrieval system must preserve source permissions at query time, not merely at ingestion: apply identity and security trimming before retrieval, then evaluate relevance and leakage separately. The non-negotiable test is that a user who loses source access must also lose retrieval and cached-result access within a defined propagation objective — tested continuously, not only before launch.

A controlled import path treated as a product: stage, verify (hash, signature, provenance, licence, scan), approve, transfer under dual control, mirror to immutable internal registries, then promote by canary. The readiness gate is operational, not architectural — the environment is not ready until the team can import a critical fix, verify and promote it, reconstruct an existing release, export a redacted diagnostic pack and roll back, within defined times and without unplanned connectivity.

Observe from user outcome down to accelerator, not the reverse. A healthy GPU does not prove a useful, authorised or safe answer. Cover outcome (task success, groundedness, abstention, escalation, harmful-action prevention), service (success ratio, time to first token, completion latency, queue time, availability), runtime (prefill/decode throughput, batch size, cache use, eviction, preemption), accelerator (memory, utilisation, power, thermal, faults), dependencies and change versions — each segmented so regressions can be attributed.