RF tasks: exclude /builddir when generating model patch?

[jiaweihhuang]

Hi,

I found that a common failure mode for those tasks in language c/c++ in RF task is that, the agent tried to compile or build the project during thre resolution process. Therefore, it generated /builddir and many other artifacts, and all of them are included into the agent.patch in the evaluation stage.

As a result, the patch is too large and in the llm judge stage, it will raise an exception regarding "prompt is too long" and the rubrics fail directly. In many cases, those artifacts in the agent patch results in the failure in cmake clean/build, and the agent fails in test directly.

Even when I run the evaluation with opus 4.7/4.8, the above issue happens.

I wonder if this is what we expected? It seems more reasonable to me that we should exclude those artifacts.

I'm also curious about the reported score in your official website: did you exclude those artifacts in agent.patch or not during the evaluation?

For reference, here are the tasks with this issue (maybe not completed):

task-69b7c2a04b6f8ff9ed98812c
task-69b7c2a04b6f8ff9ed98812d
task-69b7c2a04b6f8ff9ed98812e
task-69d196f015a150488265afc1
task-69d196f015a150488265afc0

Thanks!