Skip to content

fix: BEIR corpus _id and qrels mismatch when use_group_id_in_eval=True - #65

Merged
oliverholworthy merged 2 commits into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:bugfix/convert-beir-corpus-id-and-qrels-mismatch-when
Aug 18, 2026
Merged

fix: BEIR corpus _id and qrels mismatch when use_group_id_in_eval=True#65
oliverholworthy merged 2 commits into
NVIDIA-NeMo:mainfrom
andrewwhitecdw:bugfix/convert-beir-corpus-id-and-qrels-mismatch-when

Conversation

@andrewwhitecdw

@andrewwhitecdw andrewwhitecdw commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What

Fix BEIR evaluation output when use_group_id_in_eval=True so qrels corpus-id values match the _id values written to corpus.jsonl.

The qrels writer now always uses the document ID mapping established while writing the corpus, keeping ID selection in one place.

Why

Previously, hash-ID mode wrote sequential corpus _id values such as d0, while qrels referenced hash IDs such as d_<hash>. Standard BEIR consumers therefore could not match relevant documents to corpus entries.

Behaviour

  • Default mode continues to use sequential IDs.
  • Hash-ID mode uses the content hash as both _id and group_id.
  • Qrels always reference the corresponding corpus _id.

Validation

  • Added a parametrized regression test covering both ID modes.
  • GitHub lint, test, validate, and check jobs pass.

Signed-off-by: andrewwhitecdw <andrewwhitecdw@users.noreply.github.com>
Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>

@oliverholworthy oliverholworthy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying this. I added regression coverage for both evaluation ID modes and centralized qrels on the IDs written to corpus.jsonl. The hash-mode test fails against the previous implementation, while the default sequential mode remains unchanged. This restores the BEIR qrels-to-corpus identity contract.

@oliverholworthy
oliverholworthy merged commit a5db42d into NVIDIA-NeMo:main Aug 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants