Skip to content

enricher.md

Latest commit

 

History

History
129 lines (97 loc) · 4.43 KB

enricher.md

File metadata and controls

129 lines (97 loc) · 4.43 KB
Raw

rag-enricher — reference

Metadata enrichment for AgentContextOS. Step 1.6 introduces the Enricher SPI and ships DefaultEnricher as the default implementation. The enricher sits between the chunker (Step 1.5) and the PII detector (Step 1.7) / embedder (Step 1.8) in the ingest pipeline.

Overview

An Enricher consumes the chunk list produced by the chunker and returns chunks whose metadata dict has been augmented with auto-detected facts. The contract is non-mutating: a new Chunk (frozen Pydantic) is constructed for each input; identifying fields (id, position, parent_id, content, tenant_id, …) are preserved.

The Enricher SPI (rag_core.spi.enricher.Enricher):

  • Output length equals input length; order preserved.
  • Identifying fields preserved per chunk.
  • Only metadata may differ.
  • Enrichers compose: running A.enrich then B.enrich is valid; implementations should namespace their metadata keys to avoid collisions when composed with unrelated enrichers.

DefaultEnricher

from rag_enricher import DefaultEnricher

enricher = DefaultEnricher()                 # langdetect, seed=0
enriched = await enricher.enrich(ctx, chunks, parsed)

Tags written into chunk.metadata:

Key Type Source
language str (ISO 639-1) langdetect (deterministic seed); absent if text < 20 chars
doc_type str Document.mime_type mapped via doc_type_from_mime()
created_at str (ISO-8601) Document.metadata["created_at"] else Document.created_at
modified_at str (ISO-8601) Document.metadata["modified_at"] else Document.updated_at
author str Document.metadata["author"] if present (else absent)
title str Document.title if present (else absent)
source_uri str Document.source_uri
document_id DocumentId Document.id
section_path list[str] walk parent chain via section_path() helper
reading_level float Flesch-Kincaid grade; body chunks with detected English ≥ 80 chars only

Pre-existing metadata (e.g., kind, level from the chunker) is preserved.

LanguageDetector

from rag_enricher import LangdetectDetector, LanguageDetector

class LanguageDetector(Protocol):
    def detect(self, text: str) -> str | None: ...
    def name(self) -> str: ...

Default is LangdetectDetector(seed=0). langdetect is non-deterministic by default — the seed makes results repeatable, which matters for retrieval-cache keys built from chunk metadata. Swap in your own detector by implementing the two-method protocol.

doc_type_from_mime

from rag_enricher import doc_type_from_mime

doc_type_from_mime("text/markdown")           # "markdown"
doc_type_from_mime("application/pdf")         # "pdf"
doc_type_from_mime("image/png")               # "image" (fallback to head)
doc_type_from_mime(None)                      # "unknown"

Known MIME types map to short tags (markdown, pdf, docx, pptx, xlsx, json, csv, yaml, html, text, ocr); unknown MIMEs fall back to the head before /.

section_path

from rag_enricher import section_path

by_id = {c.id: c for c in chunks}
section_path(chunk, by_id)   # ["Top", "Sub A"]

Walks chunk.parent_id repeatedly through the lookup table, collecting the content of any parent whose metadata["kind"] == "heading". Returns root-first; excludes the chunk itself.

O(d) where d is the depth of the heading hierarchy. Defensively guards against cycles and missing parents.

CLI

ragctl enrich path/to/doc.md
ragctl enrich doc.txt --max-tokens 256
ragctl enrich doc.md -n 10

Pipes parser → chunker → enricher and prints a tag-coverage summary plus first-N enriched chunks with language, reading level, and section path.

Errors

All failures wrap to rag_core.errors.IngestionError with the original exception chained via from. The pipeline error handler can react uniformly.

Extension points

To plug in a custom enricher (NER, topic classification, custom taggers):

  1. Subclass rag_core.spi.enricher.Enricher.
  2. Implement enrich(ctx, chunks, parsed) -> list[Chunk] and health().
  3. Honour the SPI invariants (preserve length / order / identifying fields; only metadata differs).
  4. Namespace your metadata keys (e.g., ner.entities rather than just entities) so you compose cleanly with DefaultEnricher.
  5. Map third-party exceptions to IngestionError.