Embedding Models and Rerankers: The Overlooked Accuracy Layer in On-Premises RAG

Enterprise conversations about on-premises AI accuracy tend to focus on one thing: which large language model is generating the answer. That’s the visible part. But in a retrieval-augmented generation (RAG) system, the generation model only ever sees what retrieval hands it — and retrieval quality is set upstream, by an embedding model and, in the better-built systems, a reranker. Get those two components wrong and no amount of prompt engineering on the generator will fix the answers.

This is the layer that gets the least attention in on-premises AI planning, and it’s also one of the more tractable to run entirely locally, on-prem — most production-grade embedding and reranker models are small enough to serve on modest hardware, next to the retrieval index they support.

What each component actually does

A private RAG pipeline has roughly three stages: embed, retrieve and rerank, then generate. Each stage has a distinct job, and conflating them is where most quality problems start.

The embedding model turns a chunk of text — a paragraph, a clause, a support ticket — into a vector: a fixed-length list of numbers positioned so that semantically similar text ends up nearby in that vector space. At query time, the same model embeds the user’s question, and a vector search finds the nearest chunks. This step has to run fast, because it’s scanning across potentially millions of vectors.

The reranker is a second, more expensive model that looks at a shortlist — typically the top 20 to 100 candidates the embedding search returned — and scores each one directly against the query. Where an embedding model compares two vectors it computed independently, a reranker reads the query and each candidate together, which lets it catch relevance signals a vector comparison misses: negation, specificity, whether a passage actually answers the question versus merely shares vocabulary with it.

The generation model, discussed in most on-prem AI planning, comes last — and only ever works with what these two steps decided was relevant. Related patterns for the full agent workflow around this pipeline are covered in RAG Agent Patterns.

Why retrieval quality is a bigger lever than most teams assume

It’s tempting to treat “the AI gave a wrong answer” as a generation problem and reach for a bigger or better generation model. In a RAG system, that’s frequently the wrong fix. If the retrieval step never surfaced the passage that contains the answer, no generation model — however capable — can produce a grounded response. It will either say it doesn’t know, or worse, generate a plausible-sounding answer from its own training data that has nothing to do with your documents.

This matters more, not less, in regulated environments. A private RAG system is usually being trusted to answer from a specific, governed corpus — policy documents, clinical guidelines, contract language — precisely because free-form generation isn’t acceptable. If retrieval quality is weak, the system quietly degrades into the thing it was built to avoid: an LLM answering from its own priors with a thin veneer of citations. Sharpening the embedding and reranking layer is often the single highest-leverage change available before touching the generation model at all, as discussed more broadly in RAG Best Practices for Enterprise AI Systems.

Choosing a local embedding model

A handful of practical criteria matter more than benchmark leaderboard position when selecting an embedding model to run on-premises:

• Domain fit over generic benchmark rank. Public retrieval benchmarks are dominated by web and Wikipedia-style text. A model that ranks well there can still underperform on dense contract language, clinical notes, or internal jargon. Evaluate candidates against a sample of your own documents and real questions, not just a published leaderboard.
• Dimensionality and index cost. Higher-dimensional embeddings can capture more nuance, but every additional dimension increases vector index storage and search latency at scale. For most enterprise corpora, a mid-sized, well-tuned model outperforms a larger one on cost-adjusted accuracy.
• Multilingual coverage, if relevant. If the corpus spans languages, confirm the model was actually trained for multilingual retrieval rather than assuming a strong English-language model generalizes.
• License terms for commercial, on-premises use. Not every open-weight model on a leaderboard is licensed for unrestricted commercial deployment — check this before it becomes a procurement blocker late in a project.
• Footprint that matches what you’re already running. Most competitive open-weight embedding models are small enough to run on CPU or a single modest GPU, so the addition rarely changes the on-premises hardware plan described in Local AI Infrastructure Best Practices.

Choosing and deploying a reranker

Not every RAG deployment needs a reranker, but most that handle anything beyond narrow, well-structured lookups benefit from one. The decision criteria differ slightly from embedding selection:

• Cross-encoder architecture, evaluated on your query shapes. Rerankers that jointly encode the query and candidate together are generally more accurate than lighter-weight scoring approaches, at the cost of more compute per candidate. Test with the kind of questions your users actually ask, including short, ambiguous, and multi-part queries.
• Shortlist size as a tuning knob. Reranking the top 20 candidates is far cheaper than reranking the top 200, and for most corpora the accuracy gain plateaus well before 200. This is one of the cheapest levers available to control latency without giving up the reranking step entirely.
• Batching for throughput. Rerankers benefit disproportionately from batched inference. Plan the serving layer around this rather than scoring candidates one at a time.
• A clear fallback path. If the reranker service is unavailable, the pipeline should degrade to embedding-only retrieval rather than failing the query outright — a resilience pattern worth designing in from the start, not retrofitting later.

Operating both models on-premises

Running embedding and reranker models locally is one of the more straightforward parts of an on-premises AI stack, precisely because these models are small relative to a generation LLM. A few operational points are worth planning for regardless:

• Re-embedding is a migration, not a config change. Vectors from two different embedding models are not comparable, so switching models — or even upgrading a model’s version — means re-embedding the entire corpus and rebuilding the index. Treat this as a planned migration with its own testing pass, not something to do casually.
• Version-pin both models in production. An unannounced model update that changes vector geometry can silently degrade retrieval quality across the whole platform. Pin exact versions and roll changes forward deliberately.
• Log retrieval and rerank scores, not just final answers. When a user reports a wrong or missing answer, the fastest diagnosis path is seeing which candidates were retrieved and how they were scored — not re-running the whole pipeline blind. This ties into the broader observability practice described in AI Agent Observability: Logs, Traces, and Audit Trails.
• Keep evaluation data current. A test set of real questions with known correct passages, refreshed periodically, is what turns “we think retrieval quality is fine” into an evidence-based claim you can show a compliance reviewer.

How VDF AI approaches this layer

VDF AI runs the full private RAG pipeline — embedding, retrieval, reranking, and generation — inside the customer’s own environment, so no document fragment or query passes through a third-party API at any stage. VDF AI Networks lets teams select and version-pin embedding and reranker models alongside generation models, logs retrieval and rerank scores as part of the same audit trail used for agent decisions, and supports re-embedding workflows when a corpus or model changes. The goal is to make the retrieval layer a deliberately engineered, evaluated part of the platform — not an unexamined default sitting quietly upstream of the model everyone is actually paying attention to.