Skip to content

fix(swebench): build reports after Modal evaluations - #770

Open
onatozmenn wants to merge 2 commits into
OpenHands:mainfrom
onatozmenn:fix/modal-swebench-report
Open

onatozmenn wants to merge 2 commits into
OpenHands:mainfrom
onatozmenn:fix/modal-swebench-report

Conversation

@onatozmenn

Copy link
Copy Markdown

Fixes #623

Modal returns before SWE-bench writes its aggregate report, so the wrapper tries to move a file that was never created.

When that file is missing after a Modal run, this calls SWE-bench's own make_run_report. Non-Modal runs keep the existing path. The same fallback is used by SWE-bench and SWE-bench Multilingual.

Tested with pytest tests/test_swebench_eval_infer.py (10 passed), Ruff, pycodestyle, and Pyright.

Copilot AI lite review requested due to automatic review settings September 26, 2026 14:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@onatozmenn

Copy link
Copy Markdown
Author

Hi maintainers — both the Pre-commit checks and Run tests workflows are awaiting maintainer approval on this fork PR. Could someone approve the current runs so CI can execute? I’ll address any failures once results are available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

swebench-eval modal runs fail after successful evaluation because final run report is never written

2 participants