Question of dynamic_mask apllication

[zhangshuoneu]

Hello, an interesting work and application of 3D reconstruction models. In the paper, it mentions, "In our approach, we replace static confidence by directly aggregating pairwise dynamic mask predictions into image dynamic masks." I checked the code, and it seems that the dynamic_mask predicted by the dpt head is used when calculating the optical flow loss. Are there any other places where dynamic_mask is utilized to enhance the accuracy of 3D reconstruction?

[kai422]

Hi, thank you for your question. You're correct that the dynamic_mask is utilized to enhance the accuracy of 3D reconstruction. This is because the 2D reprojection loss is only valid in static areas, and the dynamic_mask helps exclude dynamic regions from this calculation.

For other losses such as the global alignment loss, since the alignment is based on predicted 3D points and the corresponding world points within the same frame, there is no need to mask out dynamic objects.

Nevertheless, in addition to the optical flow loss, we also use the dynamic_mask to mask out dynamic regions for camera pose optimization during Gaussian Splatting training. the loss is only computed in static region to make sure camera pose recieve gradients only from static region.

[zhangshuoneu]

Thank you for your response. I noticed that the camera pose estimation accuracy in Table 2 of the paper differs significantly from the accuracy provided in MonST3R. I would like to confirm which specific sequence of data you used for the camera pose evaluation. For example, in MonST3R, for the TUM dataset, every 3rd frame is taken, and the first 270 frames are used to extract 90 frames for evaluation.


As shown in the two images above, the first one represents the accuracy provided in the MonST3R paper, while the second one shows the accuracy from the DAS3R paper.

[kai422]

Hi,
At the version of Monst3R we tested, there are some reproduction issues with TUM-dyanmics, see Junyi42/monst3r#42 (comment). We will try their lastest version and update the results.
Regards,
Kai

[zhangshuoneu]

Thank you for your prompt and detailed response.