Model Governance for Local LLMs, SLMs, and Specialist Models

Adding a local model to an enterprise platform is a solved problem. The weights land on a GPU host, an inference server comes up, the platform registers an endpoint, and a first workflow starts routing to it. Teams that have done it once reasonably conclude that the hard part is behind them.

The hard part starts at the second model and the second version. Once there are four or five models in the estate — a general instruction-tuned model, a smaller one for classification and routing, an embedding model, a reranker, maybe a code or domain-specialist model — and once the first of them needs upgrading, a set of questions appears that “just load the weights” has no answer to. Which workflows depend on this model? What happens to their behaviour when it changes? Which model produced the outputs someone is asking about six months later?

This is model governance, and on-premises estates need it more than cloud consumers do, not less. When you own the models, nobody else is maintaining the version history for you.

A model upgrade is a change, not a patch

The most consequential misunderstanding in local model operations is treating a model version bump as infrastructure maintenance. Operationally it looks like one: pull new weights, restart the server, done. Behaviourally it is nothing of the sort.

The same prompt, the same retrieval context, and the same tool definitions can produce materially different results across versions. Extraction that reliably found a field may start missing it in an edge format. Refusal behaviour tightens or loosens. Tool-calling patterns shift, so an agent that reliably called a validation step now sometimes answers directly. Output formatting drifts in ways that break downstream parsing.

None of that shows up in a code diff. A model change alters what a production system does while every artefact under version control stays identical — which is precisely why it needs to enter change management rather than bypass it.

The model register

Governance starts with a record. A model register is the authoritative list of what is permitted to run, and each entry needs to carry enough to answer questions asked long after the fact:

• Identity and version. The exact model, the specific version or weights hash. Not a moving tag. A register entry that names a model family but not a pinned version cannot tell you what produced a given output.
• Provenance and licence. Where the weights came from, under what terms, and any restrictions on use. This is a procurement and legal question as much as a technical one.
• Serving configuration. Quantisation, context length, serving runtime and version, and the hardware profile. A model quantised differently is, for behavioural purposes, a different model.
• Approved scope. Which workflows, departments, and data classifications may route to it. A model approved for internal summarisation is not thereby approved for a customer-facing or regulated decision path.
• Evaluation basis. The results that supported approval, on the organisation’s own representative tasks rather than public leaderboards.
• Ownership and state. Who approved it, when, and whether it is in evaluation, active, deprecated, or retired.

Most of this is knowable at registration time and nearly impossible to reconstruct later, which is the argument for capturing it at the point a model is added and governed rather than when someone asks.

Pin versions, and promote deliberately

Two practices do most of the work.

Pin the version everywhere. Workflows should reference a specific approved version, never a floating “latest” pointer. Floating references mean a change to shared infrastructure silently propagates into every dependent workflow at once, and the resulting behaviour change has no changelog entry anyone can find.

Promote through environments. A new model or version enters as a candidate, runs against the organisation’s own evaluation set alongside the incumbent, and is compared on tasks that reflect actual production work — the specific extractions, classifications, and agent trajectories that matter, including the failure cases that motivated previous fixes. Only then does the routing policy change. This is the same discipline that applies to separating development, testing, and production AI environments, applied to the model as an artefact in its own right.

Public benchmark scores are a filter for which candidates to evaluate, not a substitute for evaluating them. A model that leads a general leaderboard can perform worse than a smaller incumbent on one organisation’s document formats.

Routing is where governance is enforced

A register describes intent. Routing is where intent becomes control.

If every workflow can address every model directly, the register is documentation with no enforcement behind it. If model selection runs through a routing layer, that layer becomes the point where approved scope is applied: this workflow class routes to this model, that data classification never reaches that model, this department’s traffic stays on the models approved for it. Because the routing decision is recorded, the log answers the question that governance actually asks — not “which models exist” but “which model handled this request, and was it entitled to.”

That also makes upgrades tractable. Changing a routing policy for one workflow class is a reviewable, reversible change with a defined blast radius. Swapping the weights behind a shared endpoint is neither. It is the same reason compliance-aware model routing is treated as a control rather than a cost optimisation.

Deprecation and retirement

Estates accumulate models. Retirement is the least glamorous part of the lifecycle and the part most often skipped, which is how organisations end up serving a model nobody owns, on a runtime nobody patches, for a workflow nobody remembers approving.

A workable retirement path has four steps: mark the model deprecated in the register so no new workflow can route to it; identify every workflow still pinned to it; migrate those workflows deliberately, with the same evaluation applied to the replacement; then withdraw the serving capacity. What must survive retirement is the record — which version was active over which period, and on what basis it was approved. Decommissioning the model does not decommission the obligation to explain outputs it produced while it was live.

What supervisors and standards expect

Model governance is increasingly not just good practice. Under the EU AI Act, high-risk systems must technically allow automatic logging of events over their lifetime (Article 12), and providers must operate a documented post-market monitoring plan that tracks performance across the deployed lifetime (Article 72). Both assume you can attribute behaviour to a specific system configuration over time.

Article 25 is the provision most often overlooked in this context: a party that makes a substantial modification to a high-risk AI system already on the market can be reclassified as a provider, inheriting the fuller obligations that role carries. Whether a given model upgrade rises to that threshold is a legal assessment specific to the system and its purpose — but an organisation that cannot describe what changed, when, and how it was evaluated is not in a position to make that assessment at all. ISO/IEC 42001 pushes in the same direction, framing AI management as a lifecycle running from development through deployment, monitoring, and decommissioning, with documentation expected at each stage.

The common requirement across all of them is unremarkable: know what is running, know why it was approved, and be able to show both. See our fuller treatment of EU AI Act readiness for on-premises architectures for how this fits the wider obligation set.

Why on-premises makes this achievable

There is a genuine advantage to owning the models. In a hosted API arrangement, the provider changes the model behind an endpoint on their schedule, deprecates versions on their timeline, and exposes only as much version information as they choose — so a defensible version history is difficult to build by construction.

When the weights, the serving stack, and the routing layer all sit inside your own environment, version pinning means what it says: nothing changes until you change it. Governance stops being a matter of reconstructing what a supplier did and becomes a matter of recording what you did, which is the position local AI infrastructure is meant to put you in.

How VDF AI supports model governance

VDF AI treats models as governed, registered assets rather than interchangeable endpoints. Local models — general, specialist, embedding, and reranking — are registered with their version, serving configuration, and approved scope, so the estate has one authoritative list of what may run. VDF AI Router enforces that scope at request time, directing each workflow to the models it is entitled to use and recording the routing decision, so approvals are applied rather than merely documented. VDF AI Model Evaluation supports the promotion gate, comparing candidate models against the incumbent on your own representative tasks before a routing policy changes. And because all of it runs inside your environment, the version history, evaluation evidence, and inference logs stay under the controls the rest of your infrastructure already meets.