Skip to content

fix(eval): use unique image_rel as sample_id to stop silent dump_dir overwrites - #6

Merged
bartrosa merged 1 commit into
mainfrom
cursor/critical-bug-inspection-ffbf
May 23, 2026
Merged

fix(eval): use unique image_rel as sample_id to stop silent dump_dir overwrites#6
bartrosa merged 1 commit into
mainfrom
cursor/critical-bug-inspection-ffbf

Conversation

@cursor

@cursor cursor Bot commented May 9, 2026

Copy link
Copy Markdown

Bug and impact

Running bigos eval --benchmark=omnidocbench --dump-dir=... silently dropped most per-sample diagnostic JSON dumps. With the tables subset (~hundreds of pages spread across many PDFs), users would see only one or two *.json files in dump_dir/ instead of one per processed sample. The aggregate report numbers were correct, but the per-sample diagnostic data — used to inspect failing pages, compare pred_preview vs gt_preview, etc. — was lost.

Root cause

In bigos.eval.omnidocbench.evaluate, sample_id was built from:

        sample_id = str(
            page_info.get("page_no") or row.get("page_id") or image_rel or f"row_{idx}",
        )

page_info.page_no is the page number within a source PDF (1, 2, 3, …) and is not unique across manifest rows: every multi-page document starts at page 1, so picking page_no first meant every "first page" sample collapsed to sample_id == "1". The dump path is computed as dump_dir / f"{_safe_dump_stem(sample_id)}.json", so all those samples wrote to dump_dir/1.json and silently overwrote each other.

I reproduced this with 5 manifest rows sharing page_no=1: the eval reported 5 samples, but only 1 dump file was written (the last one).

Fix and validation

  • Reorder the sample_id priority so image_rel (e.g. paper_2401.05459_page_001.png) — which is unique per row and carries both document and page info — is preferred over page_no. page_id (which would also be unique if present) keeps top priority.
  • Add a regression test (test_dump_dir_writes_one_file_per_sample) that processes 5 rows sharing page_no=1 and asserts dump_dir contains 5 distinct files and 5 distinct sample_ids.

Validation:

  • New regression test passes; previously it would have written 1 file.
  • Full non-slow test suite passes (43 passed).
  • ruff check clean.
  • Manual rerun confirms: 5 samples → 5 dump files (doc0_page1.png.json, …).

No public API change; affects only the value used for result.sample_id and the per-sample dump filename.

Open in Web View Automation 

OmniDocBench's page_info.page_no is the page number within a source PDF
(1, 2, 3, ...), not a unique identifier across the manifest. Picking it
first for sample_id meant every PDF's first page collapsed to
sample_id == '1', and per-sample diagnostic dumps written to
dump_dir/{sample_id}.json silently overwrote one another. With the
'tables' subset (~hundreds of rows), users got only one or two JSON
dumps instead of one per processed sample.

Reorder the priority to prefer image_rel (e.g. paper_xxx_page_001.png),
which is unique per row and carries both document and page info.

Add a regression test that processes 5 manifest rows sharing
page_no == 1 and asserts the dump_dir contains 5 distinct files.

Co-authored-by: Bartłomiej Rosa <bartrosa@users.noreply.github.com>
@bartrosa
bartrosa marked this pull request as ready for review May 23, 2026 12:45
@bartrosa
bartrosa merged commit 4328bcd into main May 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants