Question about hard-coded validation index (idx=0) in evaluation

[yaoyao-jpg]

I noticed that in dataset/dataset.py, the validation process uses a hard-coded index idx=0 to select only the first sample from each scene's validation data. Could you help clarify what was the rationale behind this design choice instead of using the full validation set? And does this partial validation affect metrics' reliability or model comparison?

This implementation might limit the evaluation completeness, as models are only validated on a single sample per scene. I'd appreciate any insights into this design decision and suggestions for modifying it to use the full validation set if needed.

Thank you for your work and clarification!

[gusongen]

You are correct—this design is intentional to enable fast validation during the training process. You can run the full validation set by changing the type of val_dataset_config in the configuration file from nuScenesSceneDatasetLidar to nuScenesSceneDatasetLidarTraverse.

[yaoyao-jpg]

Thank you for your response! May I ask if the metrics obtained during the ablation experiments in your paper were evaluated using a hard-coded index idx=0 ?