Long-Context Models vs Private RAG: What On-Premises Buyers Should Evaluate

Every few months a model ships with a context window an order of magnitude larger than the last one, and a version of the same question lands in the architecture review: if the model can read a million tokens, why are we still building a retrieval pipeline?

It is a fair question, and the honest answer is not “long context is hype.” Long context is genuinely useful, and for some enterprise work it is clearly the better tool. But the framing of “context window versus retrieval” hides the two things that actually decide the answer on private infrastructure: who pays for the tokens, and who enforces the permissions.

The advertised window is a ceiling, not a capability

The first correction is empirical. A stated context length describes what the model will accept without erroring, not the length at which it still reasons well.

NVIDIA’s RULER benchmark was built specifically to test this. It generates synthetic tasks at configurable lengths across four categories — retrieval, multi-hop tracing, aggregation and long-form question answering — precisely because the popular needle-in-a-haystack test is too easy to be diagnostic. Models that score close to perfect on plain needle retrieval degrade sharply on the harder categories as sequences grow, and of the models evaluated that claimed 32k tokens or more, only a minority held quality at 32k.

The related finding from the “lost in the middle” work is just as practical: performance depends on where in the input the relevant passage sits, with evidence at the beginning or end used more reliably than evidence buried in the middle. Stuffing forty documents into a prompt and trusting position-independence is not a design; it is an assumption that has been measured and found wanting.

None of this means long context fails. It means the number on the datasheet is not a substitute for measuring effective length on your own document types, with your own question mix, using the same evaluation framework you would apply to a retrieval pipeline.

On-premises, context length is a memory budget

In a cloud API, long context arrives as a larger invoice. On your own hardware it arrives as a claim on GPU memory that your other users were relying on.

The key-value cache holds the attention keys and values for every token processed so far, and it grows linearly with sequence length. The arithmetic is checkable:

KV bytes = 2 (K and V) × layers × kv_heads × head_dim × tokens × bytes_per_value

For a 70B-class open-weight model with 80 layers, 8 key-value heads under grouped-query attention and a head dimension of 128, at half precision that is 2 × 80 × 8 × 128 × 2 bytes = 320 KiB per token. At 128,000 tokens, a single request’s cache is roughly 40 GiB — most of an 80 GB accelerator, before the model weights themselves are counted.

Three consequences follow, and they are the ones that show up in production rather than in the demo:

• Concurrency collapses. Memory spent on one long conversation is memory unavailable for batching. A tier sized for fifty concurrent short sessions can be reduced to a handful by a few users who habitually paste in large documents.
• Time-to-first-token grows faster than input length. Prefill attention cost scales quadratically with sequence length, so the wait before the first word appears is where users feel a long prompt, not in the generation.
• The cost is invisible in the usual budget line. There is no per-token charge to notice. The symptom is a queue, which is why admission control and per-request context limits belong in the platform rather than in user guidance.

Sizing work that assumed short prompts — the kind covered in estimating GPU requirements for local LLM workloads — has to be redone against a realistic distribution of input lengths, not the average. Techniques like quantisation and cache compression move the line, but they do not change its slope.

The governance argument is the one that settles it

The cost argument is a trade-off. The access-control argument usually is not.

In a private retrieval pipeline, the filter that decides which documents a given user may see runs before generation. Permission-aware retrieval evaluates the asker’s entitlements against document ACLs and returns only what they were already allowed to open. The model never sees the rest, so it cannot leak it, and every answer carries citations to specific source locations that a reviewer can open.

A “just put it all in context” design has no equivalent step. Whatever is assembled into the prompt is visible to the model, and any post-hoc instruction not to use restricted material is a request, not a control. For regulated environments this is decisive: an HR policy corpus, a claims file or a credit file cannot be handed to a model wholesale because one authorised user asked a broad question.

There is a second, quieter governance benefit. Retrieval produces a record of what was considered, not only what was answered — which documents were candidates, which were filtered out by permission, which were reranked into the final set. That record is the raw material for decision receipts and for the evidence an auditor asks for. A long prompt assembled ad hoc produces no such artefact.

Where long context is genuinely the better tool

Against all that, there are tasks where retrieval is the wrong instrument and a large window is exactly right:

• Single-document deep work. Reviewing one 200-page agreement, one technical specification, one incident timeline. The document is the unit of analysis; chunking it only creates the risk of missing a cross-reference between clause 4 and schedule 9.
• Questions with no retrievable anchor. “Summarise the obligations this contract imposes on us” has no query terms to match — the answer is distributed across the whole document rather than concentrated in a passage.
• Cross-cutting consistency checks. Finding contradictions between sections, or confirming that a defined term is used consistently, requires seeing the sections together.
• Structured material that chunking damages. Long tables and dependency-heavy documents suffer badly from fragmentation, as private RAG on tables and spreadsheets sets out in detail.

Retrieve wide, read long

The architecture that survives contact with both the finance team and the compliance team is not a choice between the two. It is a division of labour.

Retrieval does selection and access control: it applies the permission filter, narrows a corpus to a defensible candidate set, and reranks that set so the most relevant material lands where the model attends to it best. The long window then does reasoning: instead of three timid 400-token fragments, the model receives whole sections — or whole documents — from the material the user was entitled to see.

Practically, that means:

1. Set an explicit per-request context budget in the platform, enforced centrally rather than trusted to prompt templates.
2. Keep the permission filter and citation trail in retrieval, always, regardless of how much context the model can accept.
3. Route by task type: single-document analysis gets a long-window model; corpus questions go through retrieval first. This is an ordinary model-routing decision, not a platform rewrite.
4. Measure effective context on your own data before you promise it to users, and re-measure after every model upgrade.
5. Use semantic caching carefully for repeated long inputs, respecting the same permission boundary as retrieval.

The question to bring to a vendor is therefore not “what context length do you support.” It is: what happens to concurrency when ten users send 100k-token requests at once, where does the permission filter run, and what record survives the answer.