contextbench/parsers/gold.py:23 ends with return p.lstrip("./"). str.lstrip takes a character set, not a prefix, so every leading . is consumed:
.goreleaser.yml → goreleaser.yml
.github/workflows/macos.yml → github/workflows/macos.yml
Those files exist in the repository. Normalisation renames them into non-existence.
This is two-sided, and it penalises a correct retriever. evaluate._filter_step_to_repo resolves predicted paths against the worktree and drops any that do not exist, while Gold.files() applies a different function and drops nothing. So a mangled gold path stays in the coverage denominator permanently — and a system that correctly returns the real dot-path is scored as wrong for it.
Measured on SWE-Bench-Pro__go__maintenance__bugfix__997c7afd (flipt-io/flipt; 13 gold files, 4 of them dot-paths, all four present in the repo):
| replay |
File coverage |
File precision |
| gold through current normalisation |
0.6923 |
1.0000 |
| gold with the dataset's true paths |
0.6923 |
0.6923 |
10 tasks in the public release are affected. Because it changes the denominator, it also affects published leaderboard numbers.
Suggested fix: p[2:] if p.startswith("./") else p
contextbench/parsers/gold.py:23ends withreturn p.lstrip("./").str.lstriptakes a character set, not a prefix, so every leading.is consumed:.goreleaser.yml→goreleaser.yml.github/workflows/macos.yml→github/workflows/macos.ymlThose files exist in the repository. Normalisation renames them into non-existence.
This is two-sided, and it penalises a correct retriever.
evaluate._filter_step_to_reporesolves predicted paths against the worktree and drops any that do not exist, whileGold.files()applies a different function and drops nothing. So a mangled gold path stays in the coverage denominator permanently — and a system that correctly returns the real dot-path is scored as wrong for it.Measured on
SWE-Bench-Pro__go__maintenance__bugfix__997c7afd(flipt-io/flipt; 13 gold files, 4 of them dot-paths, all four present in the repo):10 tasks in the public release are affected. Because it changes the denominator, it also affects published leaderboard numbers.
Suggested fix:
p[2:] if p.startswith("./") else p