Skip to content

parsers.md

Latest commit

 

History

History
209 lines (165 loc) · 7.86 KB

parsers.md

File metadata and controls

209 lines (165 loc) · 7.86 KB
Raw

rag-parsers — reference

Built-in document parsers for AgentContextOS. Step 1.3 delivers ten format-specific parsers plus a content-type detector that the ingest pipeline (Step 1.10) will wire up after the chunker (Step 1.5) lands.

Overview

A Parser consumes raw bytes and returns a ParsedDocument —

  • the embedded Document with content_hash, tenant_id, corpus_id, mime_type, and status=DocumentStatus.ready set
  • the extracted plain-text text
  • a list of structural Block records (heading / paragraph / list_item / table_row / code / quote / caption / other) with offsets into text, optional heading level, and parser-specific attributes like {"page": 3} or {"sheet": "Q1", "row": 5}
  • the detected_mime — the parser's normalized MIME if it sniffed past the caller-supplied hint

The downstream chunker (Step 1.5) consumes blocks for structure-aware boundary detection. Parsers that cannot recover structure (plain text, opaque blobs) emit a single BlockType.paragraph spanning the full extracted text.

Supported formats

MIME Parser Library Heading detection
text/plain and other text/* PlainTextParser stdlib none — blank-line paragraph split
text/markdown MarkdownParser markdown-it-py #–###### → level 1–6
text/html, application/xhtml+xml HtmlParser beautifulsoup4 + stdlib html.parser <h1>–<h6>
application/json, application/x-ndjson JsonParser stdlib none — re-emitted as indented text
text/csv, text/tab-separated-values CsvParser stdlib first row treated as header (attributes.header=True)
application/yaml, text/yaml, text/x-yaml, application/x-yaml YamlParser PyYAML (safe_load) none
application/pdf PdfParser pypdf none — one paragraph block per page
application/vnd.openxmlformats-officedocument.wordprocessingml.document DocxParser python-docx Heading 1–Heading 6 styles
application/vnd.openxmlformats-officedocument.presentationml.presentation PptxParser python-pptx slide title → level-2 heading
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet XlsxParser openpyxl (read-only) sheet name → level-2 heading

All libraries are pure Python (or distributed as wheels), so the package installs identically on Windows, macOS, and Linux without WSL.

Usage

from rag_core.types import (
    CorpusId, DocumentId, Principal, PrincipalId, PrincipalKind,
    RequestContext, TenantId,
)
from rag_parsers import default_registry, detect_mime

tenant = TenantId("acme")
ctx = RequestContext(
    tenant_id=tenant,
    principal=Principal(
        id=PrincipalId("svc"),
        kind=PrincipalKind.service,
        display_name="svc",
        tenant_id=tenant,
    ),
)

with open("memo.docx", "rb") as f:
    data = f.read()

mime = detect_mime(data, filename="memo.docx")
parser = default_registry().select(mime)
parsed = await parser.parse(
    ctx, data, mime,
    DocumentId("d1"), CorpusId("c1"), "file:///memo.docx",
)

print(parsed.text[:200])
for block in parsed.blocks[:5]:
    print(block.type, block.level, block.text[:60])

MIME detection

detect_mime(data=..., filename=..., hint=...) accepts any combination of bytes / filename / caller hint. Resolution order:

  1. Extension — if filename is given, the lowercase extension wins for known formats. Office and PDF extensions are deterministic, so no content sniffing is needed.
  2. Content sniffing via puremagic on the first 4 KiB. When puremagic returns the generic application/zip for an OOXML file, the ZIP central directory is inspected to distinguish docx / pptx / xlsx.
  3. Caller hint (e.g. an HTTP Content-Type header) as a last-resort fallback.
  4. application/octet-stream if all three fail.

Registry

default_registry() returns a ParserRegistry pre-populated with every built-in parser in the order specific → generic: Markdown, HTML, JSON, CSV, YAML, PDF, DOCX, PPTX, XLSX, PlainText. Use registry.select(mime) to resolve a parser; it raises ParseError when no registered parser accepts the MIME type.

You can also build a custom registry to override default ordering or to inject your own Parser implementation:

from rag_parsers import ParserRegistry, PlainTextParser
registry = ParserRegistry()
registry.register(MyHtmlParser())   # takes precedence over the default
registry.register(PlainTextParser())

ragctl parse

A smoke-test command ships with the CLI:

$ ragctl parse memo.docx
file:    memo.docx
mime:    application/vnd.openxmlformats-officedocument.wordprocessingml.document
parser:  DocxParser
bytes:   24,816
sha256:  a1b2c3...
text:    1,847 chars
blocks:  18 (heading=3, paragraph=12, table_row=3)

first blocks:
  [heading h1] @0: Section One
  [paragraph] @13: First body paragraph.
  ...

Useful for design-partner demos and triaging parse problems before the full ingest pipeline lands in Step 1.10.

Error handling

  • ParseError is raised when input bytes are malformed (corrupt PDF, invalid JSON, broken OOXML, undecodable bytes).
  • Empty input returns an empty ParsedDocument — text="", blocks=[], document.content_hash still set (sha256 of b""). Callers should decide whether to skip empty documents downstream rather than have parsers raise.
  • Encrypted PDFs are rejected with a ParseError — password-protected ingestion is deferred to Step 6.7 (BYOK / secrets).

Internals

Block taxonomy

BlockType is deliberately small:

  • heading — carries level 1–6
  • paragraph — generic flowing text
  • list_item — bullet / numbered list element
  • table_row — one row of a table (HTML <tr>, DOCX table, CSV/TSV row, XLSX row)
  • code — preformatted code block
  • quote — <blockquote> / Markdown > quote
  • caption — figure caption (HTML <figcaption>) or PPTX speaker notes
  • other — escape hatch for parser-specific fragments

Parsers normalize text inside blocks (collapsed whitespace, stripped leading/trailing spaces) but do not merge across structural boundaries — the chunker owns higher-level grouping decisions.

Offsets and reproducibility

Block start offsets index into ParsedDocument.text, the same string that was used to compute document.content_hash-adjacent text length metrics. Parsers guarantee:

  • text[block.start : block.start + len(block.text)] == block.text for plain-text / Markdown / HTML, where reading order matches input order.
  • For tabular formats (CSV / DOCX tables / XLSX) and OOXML where text is re-assembled by the parser, offsets index into the re-assembled text, not the source bytes. The chunker should treat start as an intra-document anchor, not a byte offset.

Performance

Parsers are designed to be cheap to construct (registry-friendly) and to do real work inside parse(). Tests on the developer laptop (M1):

  • text-family: < 5 ms per file
  • PDF (single-page): ~30 ms
  • DOCX / PPTX / XLSX: ~50–150 ms
  • 10-page PDF: ~200 ms

Step 1.5's chunker is expected to dominate; Step 1.10's pipeline will wrap parsers in a Batcher so connector-triggered concurrent fetches coalesce into batched parses.

Extension points

Adding a new parser:

  1. Implement rag_core.spi.Parser returning a ParsedDocument.
  2. Use rag_parsers._base.build_document to assemble the embedded Document with content_hash set correctly.
  3. Register via ParserRegistry.register(MyParser()) — first match wins.
  4. Add the new MIME to rag_parsers.mime._EXT_MIME if a file-extension shortcut is appropriate.
  5. Add a conformance test under tests/parsers/ and update docs/reference/parsers.md.

See docs/architecture/parsers.md for the design rationale and the policy / pipeline integration plan.