Hi authors,
Thank you for open-sourcing CityNavAgent. I am trying to reproduce the evaluation on the AirVLN-S val-seen split, and I have a few questions about memory graph files.
I downloaded the data from the link provided in the README and placed it under ./data with the following structure:
gt_by_env
mem_graphs
mem_graphs_pruned
I noticed that the released memory graph files seem to be available only for some scenes, such as scene 3 and scene 14. In my reproduction, these two scenes can achieve non-zero SR, but for other scenes such as 2, 5, 8, 10, 12, and 17, the agent mainly falls back to the exploration branch and the SR becomes 0 in my runs.
Could you also clarify whether the memory graphs for these scenes are missing from the released package, or whether they were not used in the original experiments? If they are required for reproducing the reported AirVLN-S val-seen results, would it be possible to release the full memory graphs or the scripts used to construct them?
For reference, I am currently using a Qwen-series API instead of GPT-4o/GPT-4V, so I understand that the absolute performance may be lower than the paper. However, I would like to confirm whether the missing memory graph files are expected.
Thanks again for your great work.
Hi authors,
Thank you for open-sourcing CityNavAgent. I am trying to reproduce the evaluation on the AirVLN-S val-seen split, and I have a few questions about memory graph files.
I downloaded the data from the link provided in the README and placed it under
./datawith the following structure:gt_by_envmem_graphsmem_graphs_prunedI noticed that the released memory graph files seem to be available only for some scenes, such as scene 3 and scene 14. In my reproduction, these two scenes can achieve non-zero SR, but for other scenes such as 2, 5, 8, 10, 12, and 17, the agent mainly falls back to the exploration branch and the SR becomes 0 in my runs.
Could you also clarify whether the memory graphs for these scenes are missing from the released package, or whether they were not used in the original experiments? If they are required for reproducing the reported AirVLN-S val-seen results, would it be possible to release the full memory graphs or the scripts used to construct them?
For reference, I am currently using a Qwen-series API instead of GPT-4o/GPT-4V, so I understand that the absolute performance may be lower than the paper. However, I would like to confirm whether the missing memory graph files are expected.
Thanks again for your great work.