I attempted to reproduce the nuPlan results using this repository by training the model according to the settings described in the paper (200 epochs, learning rate 5e−4, batch size 1024 instead of 2048). However, when evaluating on the Test-14-hard split, the performance was significantly lower than expected — I obtained a score of 0.52 for closed loop non reactive variant.
I also tried switching NumPy to version 1.26, but the results remained roughly the same.
Is there a hyper parameter, configuration detail, or additional step I might be missing, or could there be another issue affecting the evaluation? I had enabled find_unused_parameters=True, as I was getting DDP error while training.
I attempted to reproduce the nuPlan results using this repository by training the model according to the settings described in the paper (200 epochs, learning rate 5e−4, batch size 1024 instead of 2048). However, when evaluating on the Test-14-hard split, the performance was significantly lower than expected — I obtained a score of 0.52 for closed loop non reactive variant.
I also tried switching NumPy to version 1.26, but the results remained roughly the same.
Is there a hyper parameter, configuration detail, or additional step I might be missing, or could there be another issue affecting the evaluation? I had enabled
find_unused_parameters=True, as I was getting DDP error while training.