Most private RAG deployments do well on policies, procedures, contracts and manuals, and then hit a wall the first time someone asks about a number. The document is indexed. The number is in it. The answer comes back wrong, or right for the wrong period, or confidently attached to the wrong row.
This is not a model problem, and turning up the context window does not fix it. It is a pipeline problem, and it has three distinct causes that need three distinct fixes.
Cause one: the chunker destroys the table
A retrieval chunk is supposed to be self-contained: enough context that a reader landing on it cold can understand what it says. Prose satisfies this reasonably well. A table row does not satisfy it at all.
Take a row reading Q3 | 14,204 | 3.1% | (812). Without the header row it is meaningless. Without the units note it is ambiguous — thousands or millions, local currency or reporting currency. Without the scope footnote it may be excluding a business unit. Without the table caption it might be forecast rather than actual. And a chunker splitting on character counts will happily separate all four from each other.
The failure is compounded by the embedding step. A fragment of a financial table embeds close to almost any financial question, because it is dense with the right vocabulary. So the shredded row retrieves well and reads badly — high recall, no meaning. The chunking strategy that works for narrative documents is actively harmful here.
The fix is structural. Treat tables as a separate document type, extracted before chunking rather than during it. Detect table regions during layout analysis, recover the structure — including merged cells and multi-level headers — and then serialise each table into a form where every row carries its own context: the column names, the units, the period, the caption and the source location in the original document. A row that reads “Revenue, Q3 FY26, actual, EUR thousands: 14,204 (source: FY26 management pack, p.12, table 3)” retrieves for the right reasons and can be checked by the person reading it.
Scanned tables need the same treatment one layer earlier, where OCR quality and layout recovery determine whether structure recognition has anything to work with.
Cause two: some questions are not retrieval questions
The second cause is the one no amount of chunking fixes. “What was the loss ratio for commercial property in Q3?” is a lookup — the answer exists in a document, and retrieval is the right tool. “How has the loss ratio moved across all lines over the last eight quarters?” is not a lookup. The answer exists in no chunk, because it has to be computed across many.
Retrieval systems answer these questions anyway, which is the problem. They return the chunks that look most like the question, and the model produces a plausible narrative from whatever subset it got — usually a partial one, with no signal that it was partial.
The design answer is routing rather than better retrieval. Classify the question: point lookups go to the document index; aggregates, comparisons and trends go to a query tool over the system of record. This is where the argument for connecting the database directly rather than indexing exports of it becomes concrete — the numbers are live, complete and computed rather than remembered.
Generated queries deserve appropriate scepticism. On the BIRD benchmark, which spans 12,751 question-SQL pairs across 95 real databases in 37 professional domains, leading published systems reach the low eighties in execution accuracy against a measured human benchmark of 92.96%. That is genuinely useful and clearly not autonomous. Treat the generated query as a draft: run it read-only, against a constrained view rather than the raw schema, and show the SQL and the returned rows next to the answer so the reader can check the work.
Cause three: spreadsheets are applications, not documents
Spreadsheets pulled from a shared drive are the hardest input in the corpus, and teams routinely underestimate them.
A workbook is not a table. It is several tables at arbitrary positions, plus formulas, plus hidden rows, plus named ranges, plus three years of overwritten assumptions, plus a version that someone forwarded by email and edited. Flattening a sheet to text loses the formulas — which is where the business logic lives — and preserves the values, which are only meaningful with the formulas. Hidden and filtered rows serialise as visible ones unless you handle them explicitly. Merged cells become blanks that quietly shift columns.
Three rules keep this manageable. Extract each contiguous region as its own table with its own header inference rather than treating a sheet as one grid. Capture the formula alongside the computed value for any cell you intend to expose, because “this is a hard-coded override of the model” is exactly what a reviewer needs to know. And carry the workbook’s provenance — path, owner, last modified, sheet name and cell range — into the retrieval metadata, so a user can filter to the authoritative copy instead of retrieving whichever of the nine near-identical versions embedded best.
Where a spreadsheet is genuinely the system of record for a process, the honest answer is often that it should be queried rather than indexed, or migrated before it is indexed. RAG over a corpus of conflicting workbooks produces authoritative-sounding answers derived from someone’s abandoned draft.
Evaluate tables separately
A retrieval pipeline can score well overall and be unreliable on exactly the questions that matter most, because generic evaluations weight document and section relevance and table failures are a small slice of a mixed test set.
Build a table-specific evaluation slice: questions whose answers live in tables, with the correct value, the correct unit and the correct source location recorded as the expected result. Score three things — was the right table retrieved, was the right row and column identified, and was the number reported with its correct unit and period. A pipeline that gets the first two right and the third wrong is the dangerous case, because the answer looks checked.
Track that slice separately in your retrieval accuracy framework. It moves independently of the prose numbers, and it is the one that regresses silently when a parser or a model is upgraded.
How VDF AI handles structured data
VDF AI treats tables and databases as first-class inputs rather than as text to be flattened. Table extraction runs as part of ingestion inside the customer’s environment, with structure, headers, units and source location preserved in the indexed record. Enterprise databases can be connected directly and exposed to agents as governed query tools, so aggregate questions are computed against the system of record rather than reconstructed from retrieved fragments, with read-only access, constrained views and the query itself recorded in the audit trail.
Because the pipeline runs on-premises, the documents, the extracted tables, the embeddings and the query logs stay inside the same boundary as the underlying systems — which matters more for financial and operational data than for almost anything else in the corpus.
Further reading
- Private RAG Across SQL Databases and Document Repositories
- Chunking Enterprise Documents for Private RAG
- Scanned Documents in Private RAG: OCR and Layout
- How to Measure Private RAG Accuracy
- How to Connect an Enterprise Database to VDF AI
Sources
- BIRD-Bench — A Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (arXiv)
- PubTables-1M: Towards comprehensive table extraction from unstructured documents (arXiv)
Numbers buried in tables your assistant cannot read? See how VDF AI builds private RAG pipelines inside your own environment, or book a demo.