Most private RAG guidance quietly assumes an English corpus. European enterprises rarely have one. Group policy is written in English, the employment contracts are in French, the plant maintenance manuals are in German, the local regulatory correspondence is in Turkish or Polish, and the same procedure exists in three languages with two of them out of date.
A retrieval pipeline tuned on English documents does not fail loudly on a corpus like that. It fails selectively: it works for the head office, mostly works for one or two large subsidiaries, and quietly returns nothing useful for a plant manager asking in the language the manual was written in. Because the failure is uneven, it usually survives the pilot and surfaces after rollout.
Multilingual retrieval is not a translation problem bolted onto the side of a RAG system. It is a set of design decisions running through the embedding model, the chunking strategy, the query path, the evaluation, and — for regulated European organisations — the deployment boundary.
Three architectures, and what each costs you
Translate then index. Machine-translate everything into one pivot language and run a monolingual pipeline. Simple, and it lets you use the strongest single-language components. The costs are substantial: every answer inherits translation error the user cannot inspect, regulated terminology drifts, the corpus doubles and the copies fall out of sync, and — unless translation runs locally — documents leave the boundary during ingestion.
Cross-lingual retrieval with multilingual embeddings. One shared vector space, so a German query can retrieve a French passage. Source documents remain the retrievable unit, keeping citations honest and terminology intact. Everything then depends on embedding quality per language, which varies more than teams expect.
Per-language indexes with query routing. Detect the query language, search the matching index, optionally fan out to others. Predictable, and it allows a different model per language where one justifies it. It fragments recall when the answer exists only in another language — common in multinational groups, where the authoritative policy is in one language and everything else is a local rendering.
The workable default is the second, with per-language handling introduced only where measurement shows a language is underserved. That keeps one index, one permission model and one place to fix problems.
Choosing the embedding model is the decision that matters most
In a multilingual pipeline, the embedding model is the ceiling on everything downstream. A reranker cannot recover a document that retrieval never surfaced, and a strong generation model will produce a confident answer from the wrong passages.
Public benchmarks are the right way to build a shortlist. MMTEB, the massive multilingual extension of MTEB, spans more than 500 evaluation tasks across over 250 languages, and its findings are directly relevant to on-premises deployment: the authors report that the strongest publicly available model in their evaluation was a roughly 560-million-parameter multilingual model rather than a multi-billion-parameter one. For a private deployment that is welcome news — a competitive multilingual embedding model does not necessarily demand a large slice of the GPU capacity you have.
A shortlist is not a decision. Benchmarks measure general-domain retrieval; your corpus is insurance clauses, maintenance procedures or credit policy in a specific register. Run the candidates on your own per-language evaluation sets, as described in the private RAG accuracy framework, paying particular attention to the cross-lingual case — query in one language, correct answer in another — because that is the capability the architecture rests on and the one most likely to be weak. Rerankers deserve the same treatment: a multilingual cross-encoder earns its keep most clearly when query and document languages differ, precisely the scenario a monolingual reranker handles worst.
Chunking is not language-neutral
Chunking rules calibrated on English documents transfer badly. Token counts inflate for non-Latin scripts and for agglutinative languages such as Turkish and Finnish, where a single word can carry what English spreads across a phrase — so a fixed token-size chunk holds materially less meaning in one language than another, and a context budget that comfortably fits five English passages may fit three.
Two adjustments help. Set boundaries on document structure — clause, section, procedure step — rather than on a fixed token count, so a chunk is a unit of meaning in any language. And measure the realised chunk size per language after ingestion; a large discrepancy is an early signal that one language is being systematically disadvantaged at retrieval time. The broader boundary strategies are covered in chunking enterprise documents for private RAG.
The query path needs explicit policy
Three decisions belong in the design rather than in whatever the framework does by default.
Answer language. The safe rule is to answer in the user’s language while citing sources in their original language, and to say so in the interface. A user reading a German answer built from a French contract needs to know that, particularly where the original wording is the one that governs.
Terminology. Regulated vocabulary does not map one-to-one across languages, and the near-translation is often wrong in a way that only a specialist notices. A maintained glossary of domain terms and their accepted equivalents, applied as query expansion and captured in document metadata, closes more retrieval gaps than a model change usually does.
Ambiguous queries. Short queries, product codes and technical identifiers defeat language detection. Where detection is uncertain, searching the whole index and letting the reranker sort the result beats routing on a confident guess.
Access control does not change, and that is the point
The one thing multilingual design must not touch is the permission model. Language is an attribute of a document, not a security boundary. A French translation of a restricted policy carries exactly the restrictions of the original, and any derived artefact — a translation, a summary, an extracted table — has to inherit its source’s access tags at creation rather than acquire them later.
The enforcement mechanism is unchanged: permissions live in the metadata schema and are applied as a pre-filter before retrieval, never as an instruction to the model. That is the model described in metadata filtering and permission-aware retrieval, and the failure to test for is a derived document losing its tags in a multilingual ingestion path that was added after the access design was signed off.
Evaluate per language, or you will not see the failure
A single blended accuracy number is the most common way multilingual RAG problems stay hidden. If 80% of the corpus and 80% of the traffic are in one language, that language dominates the average, and a second language performing badly barely moves it — while the users of that language experience a system that does not work.
Build the evaluation set per language, from questions those users actually ask, include cross-lingual cases deliberately, and report the metrics separately. Then treat the suite as a regression gate: an embedding upgrade that improves the overall average can degrade one language, and without per-language reporting that trade stays invisible until someone complains.
Generation quality needs the same separation. A local model’s fluency in a lower-resource language is usually weaker than its English fluency, and it degrades faster as the model gets smaller — a real consideration when smaller specialist models are attractive for cost reasons.
Why this argues for keeping everything inside the boundary
Multilingual gaps are usually closed with something that sends text somewhere: a translation API during ingestion, a hosted embedding endpoint for a language the local model handles poorly, a cloud reranker. Each is a small engineering decision, and collectively they can undo the sovereignty position the project was built on — often without appearing in an architecture diagram.
Running the multilingual components locally removes the temptation structurally. Embedding models, rerankers, any translation step and the generation model all execute inside the same environment as the documents, so the derived artefacts — vectors, translations, logs — stay where the source material stays. For European enterprises this is generally the reason the deployment was scoped as private in the first place, and it is the distinction examined in data sovereignty versus data residency.
How VDF AI supports multilingual private RAG
VDF AI runs the full retrieval stack inside the customer’s environment, including multilingual embedding models and rerankers registered alongside the generation models. Because models are registered rather than fixed, a team can evaluate several multilingual embedding models on its own corpus and change the choice without rebuilding the pipeline around it.
Documents keep their source language and their access metadata, and permission filters are enforced at retrieval regardless of the language of the chunk or the query. Retrieval and generation are recorded with the model versions involved, so a per-language evaluation can be re-run against real traffic. And because every component — embedding, reranking, any translation, generation — executes locally, adding a language does not add a data transfer.
Further reading
- Embedding Models and Rerankers in On-Premises Private RAG
- Chunking Enterprise Documents for Private RAG
- How to Measure Private RAG Accuracy
- How to Use Metadata Filters in Private RAG
- Keeping Enterprise Knowledge Sources in Sync
Sources
- MMTEB: Massive Multilingual Text Embedding Benchmark (arXiv:2502.13595)
- MMTEB — ICLR 2025 proceedings
Retrieving across several languages inside your own environment? See how VDF AI builds private RAG on-premises, or book a demo.