Skip to content

Feat/add ingestion scaffold - #2

Merged
bartrosa merged 21 commits into
mainfrom
feat/add-ingestion-scaffold
May 3, 2026
Merged

Feat/add ingestion scaffold#2
bartrosa merged 21 commits into
mainfrom
feat/add-ingestion-scaffold

Conversation

@bartrosa

@bartrosa bartrosa commented May 3, 2026

Copy link
Copy Markdown
Owner

Context

On the OmniDocBench academic_paper subset, the default DocumentConverter (StandardPdfPipeline) did not emit formula blocks (n_formulas_pred = 0) despite LaTeX-rich ground truth. Docling exposes a separate VLM path (VlmPipeline + granite_docling preset), which must be configured explicitly.

Goals

  • Provide an optional parsing mode with VLM (default off — backward compatible).
  • Keep cache keys disjoint (version suffix +vlm when VLM is enabled).
  • Re-baseline academic_paper only as v4 (formula-heavy subset); keep baselines v1–v3 unchanged.
  • Document the actual Docling 2.92.x API and the v3 → v4 comparison.

Technical changes

  • DoclingBackend: enable_vlm: bool = False. When True: VlmPipeline, VlmPipelineOptions, VlmConvertOptions.from_preset("granite_docling"), with PdfFormatOption for PDF and ImageFormatOption for images (avoids deprecation when processing images).
  • Output mapping: handle TextItem with label FORMULA; heuristic for LaTeX-like text when the model returns formulas as plain text.
  • CLI: bigos parse and bigos eval — --vlm / --no-vlm (Docling backend only).
  • Discovery scripts: scripts/discover/01_docling_vlm_api.py, 02_vlm_smoke_test.py; outputs under eval/results/discovery/.
  • Docs: docs/eval/docling-vlm-api.md; updates to README.md, CHANGELOG.md, docs/eval/omnidocbench.md.
  • Eval artefacts: baseline-omnidocbench-academic_paper-v4.{md,json}, dumps in eval/results/dumps/academic-v4/, eval/results/comparison-v3-vs-v4-academic.md.

Results (summary)

Metric academic v3 (standard) academic v4 (VLM)
Mean NED ~0.78 ~0.52
Mean len(pred)/len(gt) ~0.26 ~1.02
Sum of formula preds / GT (20 samples) 0 / 248 ~98 / 248 (~40%)

VLM materially improves formula coverage and headline text metrics; mean formula_edit_dist stays high due to LaTeX formatting differences vs GT. Details and qualitative notes: eval/results/comparison-v3-vs-v4-academic.md.

Testing

  • uv run ruff check . / ruff format --check
  • uv run mypy src/
  • uv run pytest (including slow, e.g. VLM smoke on a tiny PDF)

Compatibility

  • Default behaviour unchanged unless --vlm is passed.
  • Different backend.version with VLM — no mixing cached parses between standard and VLM runs.

bartrosa added 21 commits May 3, 2026 10:51
@bartrosa
feat: implement schema and backend for document processing
c72e86e 
- Added `Backend` protocol to define the interface for processing sources into documents.
- Introduced `Source`, `Block`, and `Document` models in `schema.py` for structured data representation.
- Updated `__init__.py` to re-export new classes as part of the public API.
- Created tests for schema properties and document functionality to ensure data integrity and correctness.
@bartrosa
chore: update dependencies in project configuration
206edc3 
- Added `pydantic` version `>=2.10` to `pyproject.toml` for data validation.
- Updated `uv.lock` to include `pydantic` and `annotated-types` with their respective versions and dependencies.
@bartrosa
chore: update CHANGELOG with new features for upcoming release
717afc0 
- Added Pydantic v2 schemas: Source, Block, Document.
- Introduced Backend Protocol abstraction.
- Implemented Markdown and JSON export functionality for Document.
@bartrosa
feat: implement device detection and hashing functionality
38c9f41 
- Added `_device.py` for auto-detecting the best available device (CPU, CUDA, MPS) based on environment variables and PyTorch availability.
- Introduced `_hashing.py` with a `sha256_file` function to compute SHA-256 hashes of files.
- Created `cli.py` for command-line interface to parse documents with support for different output formats and backends.
- Implemented `docling.py` backend for document processing, integrating with the Docling library.
- Added tests for CLI commands and Docling backend functionality, ensuring robust document parsing and output handling.
- Included fixture generation for testing with sample PDF documents.
@bartrosa
feat: add sample PDF fixture for testing document processing
dc9c0d1 
- Introduced a new PDF file `simple_text.pdf` in the fixtures directory to support testing of document parsing and processing functionalities.
@bartrosa
feat: enhance project configuration and dependencies
846807d 
- Added `typer` as a dependency in `pyproject.toml` for command-line interface support.
- Introduced optional dependency group for `docling` with version `>=2.0`.
- Defined a new script entry point for `bigos.cli:app`.
- Updated `uv.lock` with new package dependencies including `accelerate`, `annotated-doc`, and others.
- Modified `pytest` command in CI workflow to exclude slow tests.
- Added `reportlab` to development dependencies for PDF generation support.
- Updated mypy configuration to ignore missing imports for specific modules.
@bartrosa
feat: update CHANGELOG and README with new features and usage examples
d023955 
- Added new entries to CHANGELOG for DoclingBackend integration, CLI enhancements, cross-platform device detection, and generated test fixtures.
- Expanded README with a quick example for using the CLI and updated supported formats section to include PDF via Docling.
@bartrosa
feat: add sanity check script and disk caching for document processing
3d26cbe 
- Introduced `sanity_check.py` script to process all PDFs in a directory and print processing statistics.
- Implemented `DiskCache` class for caching parsed documents, improving performance by reducing redundant processing.
- Updated `DoclingBackend` to support caching of documents based on SHA-256 and backend version.
- Added tests for cache functionality and performance improvements when using cached documents.
@bartrosa
feat: enhance documentation and add sanity check directory
f2bb336 
- Updated CHANGELOG to include new features: DiskCache, CLI flags for caching, and the addition of the sanity check script.
- Expanded README with instructions for using the sanity check script on local documents.
- Created a new README in the data directory to guide users on managing document samples for sanity checks.
@bartrosa
feat: update project dependencies and enhance .gitignore
8fa8516 
- Added `diskcache` as a new dependency in `pyproject.toml` and `uv.lock` for improved caching functionality.
- Updated `.gitignore` to exclude all files in the `data` directory except for `README.md` and `.gitkeep`, streamlining document sample management.
@bartrosa
feat: add evaluation command and metrics for OmniDocBench
def2f41 
- Introduced a new CLI command `eval` for evaluating document processing benchmarks using OmniDocBench.
- Implemented evaluation metrics including Character Error Rate (CER) and Tree Edit Distance Similarity (TEDS) in `metrics.py`.
- Added support for generating evaluation reports in Markdown and JSON formats.
- Created new modules for OmniDocBench evaluation and metrics, enhancing the overall evaluation framework.
- Added tests for evaluation metrics and OmniDocBench functionality to ensure accuracy and reliability.
@bartrosa
feat: add OmniDocBench evaluation documentation and baseline results
6b364a3 
- Created a new documentation file for OmniDocBench evaluation, detailing the benchmark, metrics, and caveats.
- Added baseline results for the `academic_paper` and `tables` subsets in both JSON and Markdown formats, including per-sample metrics and aggregate statistics.
- Introduced a `.gitkeep` file in the results directory to maintain structure.
@bartrosa
feat: update project dependencies and enhance evaluation capabilities
99f8f39 
- Added new dependencies in `pyproject.toml` for evaluation, including `datasets`, `huggingface-hub`, `jiwer`, `rapidfuzz`, `apted`, and `pillow`.
- Updated `uv.lock` with new packages `aiohappyeyeballs` and `aiohttp`, along with their respective dependencies and wheel files.
- Enhanced module imports to include new evaluation-related packages, improving the overall functionality of the project.
@bartrosa
feat: enhance evaluation features and documentation for OmniDocBench
9928e1c 
- Added evaluation metrics including character error rate, TEDS, and edit distance to the evaluation harness.
- Integrated OmniDocBench v1.5 via HuggingFace dataset repo and updated CLI to support `bigos eval --benchmark=omnidocbench`.
- Documented evaluation setup and baseline results for `tables` and `academic_paper` subsets in README and CHANGELOG.
@bartrosa
feat: enhance evaluation metrics and CLI options for OmniDocBench
94d79cb 
- Added new command-line options for evaluation, including `--dump-dir` for diagnostic JSON dumps and `--gt-strategy` to select between legacy and json2md ground-truth assembly.
- Updated evaluation logic to incorporate the new options and improved error handling for ground-truth strategy validation.
- Enhanced metrics calculations by including mean NED and mean length ratio in the evaluation report.
- Refactored text metrics in `metrics.py` to improve clarity and added normalization functions for better comparison of predicted and ground-truth text.
@bartrosa
feat: add diagnostic scripts for OmniDocBench evaluation
ae8bb24 
- Introduced multiple diagnostic scripts to analyze and visualize data from OmniDocBench, including statistics on length distributions, sample previews, and ground truth extraction logic.
- Implemented a deep-dive script for detailed comparisons of predicted and ground truth outputs.
- Added functionality to search for ground truth extraction logic in the reference repository and compare legacy extraction methods with new approaches.
- Enhanced the sanity check script to include Polish character detection and improved output reporting for processed documents.
- Updated evaluation metrics tests to cover new functionalities and ensure accuracy in diagnostic outputs.
@bartrosa
chore: update .gitignore and enhance evaluation documentation
4038393 
- Added `eval/results/` to .gitignore to exclude evaluation result files from version control.
- Updated CHANGELOG with new evaluation features, including ground-truth markdown rebuilding and per-category metric breakdown.
- Enhanced README with instructions for running evaluations and inspecting individual sample results.
- Introduced new documentation file `metrics-explained.md` to clarify evaluation metrics used in OmniDocBench.
@bartrosa
feat: add VLM support to Docling backend and CLI
878afef 
- Introduced `enable_vlm` option in CLI for using the Granite-Docling VLM pipeline, allowing for improved handling of STEM PDFs and LaTeX output.
- Updated `DoclingBackend` to support VLM processing, including necessary adjustments to the document converter and versioning.
- Enhanced tests to verify VLM functionality and ensure stability with simple PDFs.
- Updated documentation to reflect new VLM capabilities and usage instructions.
@bartrosa
feat: add discovery scripts for VLM pipeline
bdaccaa 
- Introduced `01_docling_vlm_api.py` to discover and log the VLM pipeline API in the installed Docling version, including module inspection and class/function signatures.
- Added `02_vlm_smoke_test.py` to perform a smoke test on the VLM pipeline using local PDF and image inputs, logging conversion results and detected formula-like items.
- Enhanced output handling by writing results to markdown files for easier inspection.
@bartrosa
feat: add new evaluation documentation for OmniDocBench
8e1a428 
- Introduced three new documentation files: `diagnosis-2025.md`, `docling-vlm-api.md`, and `json2md-reference.md` to enhance understanding of evaluation metrics, VLM pipeline API, and JSON to Markdown conversion processes.
- Updated `omnidocbench.md` to reference the new documentation files, improving navigation and accessibility of evaluation resources.
- The new documents provide detailed insights into the evaluation process, API usage, and conversion logic, supporting better analysis and understanding of the OmniDocBench framework.
@bartrosa
feat: enhance Docling with VLM support and update documentation
60a8cc8 
- Added `enable_vlm` option in `DoclingBackend` for formula-friendly parsing with the new `granite_docling` preset.
- Introduced CLI commands `bigos parse --vlm` and `bigos eval --vlm` for VLM processing.
- Updated `CHANGELOG.md` to reflect new features and changes related to VLM integration.
- Enhanced `README.md` with instructions and comparison metrics for the academic paper subset using VLM, showcasing performance improvements.
@bartrosa
bartrosa merged commit ce42204 into main May 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bartrosa