Skip to content

ocr.md

Latest commit

 

History

History
131 lines (101 loc) · 6.27 KB

ocr.md

File metadata and controls

131 lines (101 loc) · 6.27 KB
Raw

OCR pipeline — architecture

Step 1.4 introduces region-aware OCR to AgentContextOS. This document covers the design choices that the reference doc doesn't spell out — why the SPI evolved the way it did, how engines with very different output shapes are normalised, and how OCR sits in the ingest pipeline alongside parsers and PolicyEngine.

Why region-aware

The Step 0.3 OCR SPI returned only (text, confidence, page_number) — a single flat string per image. That worked for proof-of-concept ingest but breaks down for two real workloads:

  1. Structure-aware chunking (Step 1.5) wants natural split points. Lines and paragraphs in a scanned page are the equivalent of Blocks in a parsed Word doc — the chunker needs them to decide where to break chunks. Without per-region bboxes, every page becomes one undifferentiated text blob.
  2. Faithfulness scoring & UI overlays. Per-region coordinates let downstream consumers highlight the exact span of a cited chunk on the source image. The hallucination guard (Step 4.3) and Status GUI (Step 3.11) both depend on this.

So Step 1.4 evolves the SPI to carry a regions: list[OCRRegion] payload. The legacy text / aggregate confidence fields remain so callers that only care about full-page text continue to work.

Normalising engine output

The two shipped engines emit very different output shapes; we normalise on the way out so downstream code never has to special-case the backend.

Concern Tesseract (via pytesseract) PaddleOCR Normalised form
Granularity per-word per-line informational only — both flow through OCRRegion
Coordinates (left, top, width, height) axis-aligned 4-point quadrilateral (supports rotated text) BoundingBox(x0, y0, x1, y1) axis-aligned
Confidence per-word 0–100 per-line 0.0–1.0 per-region 0.0–1.0
Aggregate confidence not provided not provided text-length-weighted mean (1.0 when no regions)
Empty input empty data dict [None] OCRResult(text="", regions=[], confidence=1.0)

Why we drop PaddleOCR's rotation metadata

PaddleOCR returns each detection as a quadrilateral — four (x, y) points — so it can represent rotated text. The Step 1.4 SPI deliberately collapses this to the enclosing axis-aligned BoundingBox because:

  1. The first downstream consumer (Step 1.5 chunker) operates on text + offsets, not pixels. Rotation is invisible to it.
  2. The second consumer (Step 3.11 Status GUI region overlay) renders simple pixel rectangles in v0; axis-aligned bboxes are sufficient.
  3. Tesseract has no rotation output. Carrying rotation in the SPI would make consumers handle two cases (rotated / not) without a use case yet.

If a future consumer needs rotation, the path is: add quad as an optional attribute on OCRRegion and have PaddleOCR populate it. The current bbox normalisation keeps the contract uniform until then.

Why aggregate confidence defaults to 1.0 on empty input

When an OCR pass finds no text, returning confidence=0.0 would look like an engine failure to consumers. confidence=1.0 reads correctly as "fully confident that there is no text here". Engine failures raise IngestionError instead — there's no ambiguity at the SPI surface.

Pipeline placement

Connector → Parser → OCR → Chunker → Enricher → PII → Embedder → Store
                      ↑
                      Only invoked when Parser flags an image-bearing block
                      (scanned PDF page, image-only DOCX shape, raw image
                      file from a Connector).

OCR sits after the parser and before the chunker:

  • Parsers emit Blocks with BlockType.other and attributes={"image": ...} for image regions they can't text-extract. OCR processes those images and inserts OCR'd Blocks back into the document at the right offsets. (Wiring lands in Step 1.10 — the OCR SPI is ready now.)
  • The chunker (Step 1.5) sees OCR'd blocks identically to natively-parsed blocks, with the added per-region bbox metadata in attributes.

Policy boundary

OCR runs upstream of PolicyEngine in the ingest direction, like parsers and connectors. The bytes haven't become Chunks yet, so there's no ACL or PII to enforce on them. The Step 1.10 ingest pipeline is the enforcement point — per-document quota check before OCR (so a quota-busted tenant doesn't burn an OCR call), per-chunk ACL + PII after chunking.

The rag-ocr package therefore depends only on rag-core, not on rag-policy. The PolicyEngine coverage linter (tests/policy/coverage.py) allowlists OCR plugins for this reason.

Hot-path discipline

OCR is the slowest stage in the ingest pipeline by a wide margin (tens to hundreds of milliseconds per image vs. sub-millisecond for parsers). performance.md classifies OCR as a cold path — Pydantic validation at the SPI boundary is fine; the engines themselves dominate the latency budget.

The only hot-path concern is the per-region loop in _data_to_regions / _raw_to_regions. Both use plain dicts/lists with no Pydantic overhead in the loop body — OCRRegion is constructed once per detection and the construction itself dwarfs the surrounding dict access.

Cross-platform

The default rag-ocr install is pure-Python (rag-core + Pillow). Both engine extras have cross-platform stories:

  • Tesseract — system binary, install via brew (macOS), apt (Linux), or the UB Mannheim installer (Windows). pytesseract is pure-Python and has no native deps beyond Pillow.
  • PaddleOCR — paddlepaddle ships wheels for all three OS+arch combos in the CI matrix. Heavy (~500 MB installed) so it stays optional.

Unit tests stub both engines at the sys.modules boundary, so the test suite is hermetic — no system binaries or heavy wheels needed in CI.

Related ADRs

  • ADR-0005 (PolicyEngine PDP) — defines the egress enforcement boundary that OCR sits upstream of.
  • ADR-0007 (tiered storage with BlobRef) — OCR'd text follows the same trust-level rules as parsed text; OCR doesn't grant additional trust.
  • ADR-0009 (vector index strategy + quantization) — OCR'd chunks are embedded identically to parsed chunks; no special path.