How to Select a Local LLM for Enterprise Workloads

The question “which local LLM should we run?” usually arrives framed as a benchmark comparison. It rarely stays that way. By the time a model is actually serving traffic inside an enterprise data centre, the decision has been shaped far more by license terms, GPU memory, serving stack maturity, and how the model behaves on the organisation’s own documents than by any published leaderboard position.

That is not a criticism of benchmarks — they are a useful first filter. But selecting a model for on-premises production is closer to a procurement decision than a technical preference. This guide sets out a selection framework that starts with the workload, screens hard constraints early, and reserves capability testing for the candidates that could actually be deployed.

Start from the workload profile, not the model list

Before comparing anything, write down what the model is being asked to do. A workload profile does not need to be elaborate, but it needs to be specific enough to disqualify candidates:

• Task shape. Summarising long policy documents, extracting structured fields from forms, answering grounded questions over a retrieval index, and driving multi-step tool-calling agents are four different jobs with different failure modes. A model that writes fluent summaries may be unreliable at emitting valid structured output on every call.
• Context length actually required. Not the largest document in the corpus — the amount of text the model needs in one window after chunking and retrieval. Overspecifying here is expensive, because context length drives key-value cache memory, which drives concurrency.
• Latency and concurrency targets. An interactive assistant used by 400 people has a very different serving profile from an overnight batch that processes 50,000 documents. The second tolerates slow tokens; the first does not.
• Language coverage. If the corpus or the users span languages, confirm the model was trained for those languages rather than assuming a strong English model generalises.
• Sensitivity class of the data. This determines which environment the model runs in — general on-premises, a restricted segment, or an air-gapped enclave — and therefore how the weights get in and how updates are handled.

Two or three profiles usually cover an enterprise’s initial workloads, and they will not all point at the same model — itself a useful early finding.

Screen the license before anything else

License terms are the cheapest constraint to check and the most expensive to discover late. A model that fails legal review after an integration is built is a wasted quarter.

Open-weight models are not uniform in this respect. Apache 2.0 and MIT are the straightforward cases: broad commercial rights, redistribution permitted, and — in Apache 2.0’s case — an explicit patent grant, which enterprise counsel tends to value. Vendor community licenses are a different category. They are usually workable, but they carry conditions worth reading in full: user or revenue thresholds above which a separate agreement is required, attribution and naming requirements, acceptable-use policies that bind you contractually, and sometimes territorial restrictions.

The territorial point is not hypothetical. Meta’s Llama 4 Community License does not grant rights to the multimodal models in that family to individuals domiciled in, or companies with a principal place of business in, the European Union — a restriction Meta attributed to regulatory uncertainty. Text-only paths are unaffected, and end users of a product that already incorporates such a model are carved out, but for an EU-based enterprise planning to deploy multimodal capability itself, that is a hard stop rather than a negotiating point.

A small number of models marketed as “open” are released under non-commercial terms and cannot be used in production at all. Others permit use but restrict using outputs to train competing models — relevant if fine-tuning or distillation is part of the plan.

Four questions settle most of this: Can we use it commercially, without a user threshold we will cross? Can we fine-tune it and keep the resulting weights? Can we run it in every jurisdiction where we operate? And can we redistribute it internally to the subsidiaries that need it?

Confirm the model fits the hardware you have

Capability that does not fit in memory is not capability. Three variables set the envelope: parameter count, quantisation, and context length.

Quantisation is what makes larger models practical on modest hardware, and the trade-off is often acceptable — a well-executed 8-bit or 4-bit quantisation of a larger model frequently outperforms a smaller model at full precision on the same hardware budget. What matters is testing the quantised build you intend to deploy, not the full-precision version that produced the published scores.

Context length is the variable teams most often underestimate, because its cost is paid in key-value cache memory per concurrent request rather than in weights. A model that fits comfortably at one concurrent user can exhaust GPU memory at thirty. Size against peak concurrency, not average. The arithmetic for this is covered in How to Estimate GPU Requirements for Local LLM Workloads.

The related question is whether the largest available model is needed at all. For narrow, high-volume tasks — classification, extraction, routing — a smaller specialist model often matches a general-purpose one at a fraction of the serving cost, as discussed in Small Language Models in Enterprise AI Infrastructure.

Evaluate on your own tasks, with a fixed test set

Once two or three candidates have survived the license and hardware screens, capability testing becomes worth the effort — and it should be done on your material.

Build a golden set: real questions, real documents, and known-correct answers or expected extractions, assembled with the people who will actually use the system. A few hundred well-chosen items outperform a few thousand synthetic ones. Then score the things that break production rather than the things that read well in a demo:

• Grounding. Does the model answer from the retrieved context, or drift into its own priors when retrieval is thin?
• Structured output reliability. If a workflow expects JSON on every call, measure the failure rate across the whole test set. A model that is correct 97% of the time and malformed 3% of the time needs a validation and retry path designed in, not discovered later.
• Tool-calling behaviour. For agent workloads, test whether the model selects the right tool, supplies well-formed arguments, and stops when it should. This varies far more between models than general benchmark scores suggest.
• Refusal and hedging patterns. Some models decline routine enterprise tasks — reviewing a contract clause, summarising a claims note — often enough to be disruptive.
• Behaviour at production concurrency. Quality measured on a quiet GPU is not the quality users get at peak.

Keep the test set fixed and version-controlled, so successive models are compared on identical ground. That test set later doubles as governance evidence: it is the artefact that turns “we selected this model carefully” into something a compliance reviewer can inspect, alongside the documentation obligations discussed in EU AI Act Enforcement Timeline and Deployer Readiness.

Weigh the operational criteria that decide long-term cost

Two models with similar benchmark scores can differ enormously in what they cost to operate over three years:

• Serving stack support. Whether mainstream inference servers support the architecture, and how quickly they add support for new releases, determines how much bespoke engineering you carry.
• Weight provenance and offline distribution. For restricted environments, you need a verifiable way to obtain, checksum, mirror, and promote weights without internet access at the point of deployment.
• Documentation quality. A thorough model card — training data description, intended use, known limitations, evaluation results — is the raw material for the technical documentation regulated deployers maintain.
• Release cadence and deprecation behaviour. A model family with a predictable upgrade path is easier to plan around than one that ships irregularly and abandons prior versions, and a mature fine-tuning ecosystem matters later even if you do not fine-tune on day one.

How VDF AI approaches model selection

VDF AI is deliberately model-agnostic: the platform is the governed layer around whichever local models an enterprise runs, not a wrapper on one vendor’s model. Registering and governing a local model makes it available to workflows under policy, with version pinning and access controls attached, and the Model Evaluation Suite runs candidates against your own test sets so selection rests on evidence from your data rather than a published score.

Because most environments end up running a mix, request routing across local models lets the model choice sit in policy rather than application code — so a better model, or a cheaper specialist, can be adopted without rewriting the workflows that depend on it. Model selection stops being a one-time bet.