Document AI: OCR, Layout and VLMs

Reading documents like a person does

Documents carry meaning in their layout: a number means "total" because of where it sits on an invoice. Classic document AI uses OCR to detect and read text boxes, then a layout-aware model or rules to map boxes to fields. OCR pipelines are fast, cheap, deterministic and give exact character positions, which is useful for highlighting and audit. Their weak spots are complex tables, handwriting, stamps and mixed languages. VLM-based extraction sends the page image (sometimes with OCR text as a hint) to a vision-language model and asks for structured output. It handles messy layouts and instructions in plain language, but it can hallucinate plausible values, costs more per page, and gives no guaranteed coordinates. A sound production design combines both: OCR for text and positions, a VLM for interpretation, and cross-checks such as "line items must sum to the total" with human review on failures.

Rendering PDF pages to images

VLMs read images, so PDF pages are rendered first. PyMuPDF does this without external tools; it needs a PDF file, so no output is shown.

import pymupdf  # PyMuPDF

doc = pymupdf.open("invoice.pdf")
for i, page in enumerate(doc):
    pix = page.get_pixmap(dpi=150)        # enough for most printed text
    pix.save(f"invoice-page-{i + 1}.png")
    print(i + 1, pix.width, pix.height)

The CA's assistant

OCR is the junior assistant who types every number from a client's bills exactly as printed. The VLM is the senior who understands which number is GST and which is the total, but occasionally fills a gap from memory. A good CA office uses both and checks the totals.

त्वरित जाँच: Which check best catches a VLM hallucinating values on an invoice?

• Asking the model whether it is sure
• Using a higher temperature
• Verifying that extracted line items sum to the extracted total
• Converting the invoice to grayscale