Skip to content

Gold path normalisation uses lstrip("./"), which eats leading dots and renames real files out of existence #2

Description

@jbendercb

contextbench/parsers/gold.py:23 ends with return p.lstrip("./"). str.lstrip takes a character set, not a prefix, so every leading . is consumed:

  • .goreleaser.ymlgoreleaser.yml
  • .github/workflows/macos.ymlgithub/workflows/macos.yml

Those files exist in the repository. Normalisation renames them into non-existence.

This is two-sided, and it penalises a correct retriever. evaluate._filter_step_to_repo resolves predicted paths against the worktree and drops any that do not exist, while Gold.files() applies a different function and drops nothing. So a mangled gold path stays in the coverage denominator permanently — and a system that correctly returns the real dot-path is scored as wrong for it.

Measured on SWE-Bench-Pro__go__maintenance__bugfix__997c7afd (flipt-io/flipt; 13 gold files, 4 of them dot-paths, all four present in the repo):

replay File coverage File precision
gold through current normalisation 0.6923 1.0000
gold with the dataset's true paths 0.6923 0.6923

10 tasks in the public release are affected. Because it changes the denominator, it also affects published leaderboard numbers.

Suggested fix: p[2:] if p.startswith("./") else p

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions