Skip to content

fix(commit0): count errored report instances - #772

Open
onatozmenn wants to merge 2 commits into
OpenHands:mainfrom
onatozmenn:fix/commit0-error-reporting
Open

onatozmenn wants to merge 2 commits into
OpenHands:mainfrom
onatozmenn:fix/commit0-error-reporting

Conversation

@onatozmenn

Copy link
Copy Markdown

Fixes #493

Commit0 builds its report from output.jsonl, but runner errors are written to output_errors.jsonl. Those runs disappear from the submitted and error counts.

This reads the sibling error file, ignores IDs with a completed final result, and deduplicates repeated errors. The report now includes submitted and incomplete IDs and counts like GAIA.

Tested with the Commit0 evaluator, report summary, and metrics suites (46 passed), plus Ruff, pycodestyle, and Pyright. The regression test also fails on current main.

Copilot AI lite review requested due to automatic review settings September 26, 2026 14:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@onatozmenn

Copy link
Copy Markdown
Author

Hi maintainers — both the Pre-commit checks and Run tests workflows are awaiting maintainer approval on this fork PR. Could someone approve the current runs so CI can execute? I’ll address any failures once results are available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Commit0 report undercounts error instances (reports 0 instead of actual 6)

2 participants