Skip to content

Question about AirVLN-S val-seen missing memory graph files #8

@Savartfr

Description

@Savartfr

Hi authors,

Thank you for open-sourcing CityNavAgent. I am trying to reproduce the evaluation on the AirVLN-S val-seen split, and I have a few questions about memory graph files.
I downloaded the data from the link provided in the README and placed it under ./data with the following structure:

  • gt_by_env
  • mem_graphs
  • mem_graphs_pruned

I noticed that the released memory graph files seem to be available only for some scenes, such as scene 3 and scene 14. In my reproduction, these two scenes can achieve non-zero SR, but for other scenes such as 2, 5, 8, 10, 12, and 17, the agent mainly falls back to the exploration branch and the SR becomes 0 in my runs.

Could you also clarify whether the memory graphs for these scenes are missing from the released package, or whether they were not used in the original experiments? If they are required for reproducing the reported AirVLN-S val-seen results, would it be possible to release the full memory graphs or the scripts used to construct them?

For reference, I am currently using a Qwen-series API instead of GPT-4o/GPT-4V, so I understand that the absolute performance may be lower than the paper. However, I would like to confirm whether the missing memory graph files are expected.

Thanks again for your great work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions