Question about reproducing WebArena-Lite-v2 reported results

[QianhuiWu]

Hi team,

Congratulations on the CILR acceptance — awesome work! I tried setting up the WebArena-Lite-v2 evaluation from this repo. After fixing a few small issues (e.g., parsing actions like answer(xxxx) and handling write after cleaning up), I ran the 7B checkpoint with a 15-step budget and got the following results:

Domain	Pass@1	Pass@4	Count
Gitlab	0.2833	0.4000	30
Map	0.1635	0.2308	26
Reddit	0.2632	0.3684	19
Shopping	0.2330	0.3182	44
ShoppingAdmin	0.3143	0.3714	35
---	---	---	---
Average	0.2532	0.3377	154

This is lower than the reported number (~37). Do you have any insight into what settings or evaluation details might explain the gap (e.g., step limit, prompts, environment versions, or action parsing)?

Thanks a lot!

[QianhuiWu]

P.S., I'm running with the "native model" setting.

[heroding77]

Hi Dr. Wu @QianhuiWu , sorry for the late reply! I've already asked my colleague in charge to check on this.

[G-JWLee]

Hi, any updates after this? I am trying to use ScaleCUA-32B model, but it is far below the reported performance, even though I added the answer(xxx) action parsing, where the model outputs an action of "answer('xxx')" instead of "response(answer='xxx')".