Scanned Documents in Private RAG: On-Premises OCR, Layout, and Tables

Most private RAG projects start with the easy half of the corpus. Wiki pages, policy documents, digitally produced PDFs, and office files all carry a clean text layer, and a weekend of ingestion produces something that demos well.

The half that matters in a regulated business often has no text layer at all. Signed contracts scanned at 200 dpi. Claims correspondence that arrived by fax. Twenty-year-old inspection reports. Loan files assembled from photocopies. Forms with handwritten annotations in the margin that change the meaning of the printed clause above them.

Retrieval quality over that material is decided long before the embedding model is chosen. It is decided in the document parsing stage — and that stage is a real piece of infrastructure, not a library call.

Words are not enough

An OCR engine returns characters and where they sit on the page. That is necessary and nowhere near sufficient, because retrieval depends on structure:

• Reading order. A two-column page read left to right produces sentences that alternate between unrelated columns. The text looks plausible, chunks cleanly, and answers questions incorrectly.
• Tables. A premium schedule, a rate card, or a lab result loses all meaning when the row and column relationship is flattened into a stream of numbers. The model will still match one — just not the right one.
• Headings and hierarchy. Section numbering is what lets a chunk carry the context of the clause it belongs to.
• Repeating furniture. Headers, footers, page numbers, and watermarks pollute every chunk if they are not identified and stripped.
• Forms and checkboxes. The information is in which box is ticked, and that is a layout fact rather than a text fact.
• Stamps, signatures, and handwriting. These often carry the legally significant content — an approval, a date, a condition written by hand.

This is the difference between OCR and document parsing, and it is where the accuracy of a private RAG system over scanned material is actually won.

The stages worth building explicitly

Two of these are routinely skipped and both hurt later. Path classification saves the cost of running OCR over documents that already carry exact character positions — digitally produced PDFs are faster and more accurate read directly, and IBM Research reports large speed gains from avoiding unnecessary OCR in Docling. Confidence routing is what stops a badly recognised page from entering the index as though it were clean.

The chunking that follows should respect the structure this stage recovered, along the lines described in chunking enterprise documents for private RAG, and the whole thing sits inside the secure ingestion pipeline with its access control and provenance obligations.

Choosing an on-premises stack

Three families of tooling are viable inside the boundary, and most serious pipelines use more than one.

Classical OCR engines. Tesseract and PaddleOCR are mature, run on CPU, and are excellent on clean flatbed scans of ordinary text. PaddleOCR 3.0 is Apache-licensed and ships text recognition, hierarchical document parsing, and key information extraction with models under 100 million parameters, which its technical report positions as competitive with far larger vision-language models.

Layout-aware parsers. Docling is MIT-licensed, converts PDF, Office formats, HTML, and images, and uses a layout analysis model trained on tens of thousands of hand-labelled pages together with TableFormer for table structure. It documents local execution for sensitive data and air-gapped environments, and it runs within a modest resource budget.

Document vision-language models. Compact models such as Granite-Docling handle messy layouts, unusual formats, and handwriting better than classical pipelines, at higher inference cost. Docling can invoke a VLM pipeline directly, which makes a mixed strategy practical: cheap path for the bulk, expensive path for the difficult minority.

Pick per document class, not per platform. A claims archive and an engineering drawing repository do not deserve the same pipeline, and the routing decision should be data-driven rather than architectural taste.

Provenance is the feature auditors care about

For a regulated workload, the answer is only half the deliverable. The other half is being able to open the page the answer came from — the actual scan, at the actual region — and see the stamp, the signature, or the table cell for yourself.

That means carrying document identifier, version, page number, and bounding box through chunking and embedding, and rendering citations that resolve back to the source image. It costs a little metadata design at ingestion and it changes how the system is received: an underwriter or claims handler who can verify a citation in two seconds will use the assistant, and one who cannot will quietly stop trusting it.

The same metadata makes filtered retrieval possible — by document class, effective date, entity, jurisdiction, or confidence band, as covered in metadata filters for private RAG.

Evaluate on your worst documents

Public OCR benchmarks are built from documents that are nothing like an enterprise archive. Build a labelled set of a few hundred pages drawn deliberately from the difficult end of your own corpus: the faded photocopies, the rotated scans, the multi-language pages, the forms with handwriting.

Measure at two levels. At the extraction level: character error rate on the fields that matter, table cell accuracy, reading order correctness, and the proportion of pages routed to review. At the answer level: whether the assistant retrieves the right passage and answers correctly, using the approach in measuring private RAG accuracy. Extraction metrics tell you where the pipeline is broken; answer metrics tell you whether it matters for the questions people actually ask.

Re-run the set whenever a model, engine, or preprocessing step changes. Document parsing stacks move quickly, and an upgrade that improves average accuracy can easily regress the document class your business depends on.

Capacity and containment

Parsing is a batch workload with a very different shape from interactive serving: long-running, GPU-hungry when a vision model is involved, and prone to consuming everything available during a backfill. Give it its own workload class and admission policy rather than letting an archive migration degrade the assistant that people use during the day — the mechanics are in GPU admission control.

The containment argument is simpler still. Scanned documents are usually the most sensitive material an organisation holds, complete with signatures, identifiers, medical details, and commercial terms. Sending them to a hosted document intelligence service moves exactly that material outside the boundary, in bulk, at ingestion time — while the local alternatives are mature, permissively licensed, and documented for air-gapped operation.

How VDF AI fits

VDF AI runs ingestion, parsing, embedding, retrieval, and generation inside the customer’s environment, with local models and no external document service in the path. Parsed output keeps its page and region provenance through to the citation, retrieval respects the source system’s access controls, and every run leaves an execution trace the enterprise holds.

The practical effect is that the difficult half of the corpus becomes usable without becoming a disclosure. A claims handler asks a question, gets an answer, clicks the citation, and lands on the scanned page — which never left the building.