How to train on my own dataset?

[lazyDamon]

Hi, thank for your great work! I tried to train on my own data, but result was very bad. Can you give me some advice? My data is a 360-degree shot of the shoe. I made the background transparent and extracted the depth. Intrinsic and extrinsic of colmap format.

[Fictionarry]

Hi, actually I have not test situations with transparent background yet. It seems that the voxels in the background areas are propagated abnormally. May I know what the config is in use? DTU config should be most suitable for this case.
And it seems that the depth estimator does not work very well for such a background. Maybe you can first turn off lambda_depthanythingv2 and see if the model can converge at about 2k iterations under the DTU pipeline.

[lazyDamon]

I used DTU config. It is normal at about 2k iterations when I Turn off lambda_depthanythingv2. I set the background to transparent or black because my scene is fixed camera, and the shoes rotate and flip on the turntable.

[lazyDamon]

I tried to train one of the sides of the shoes, using a transparent background, and found that result was normal.

[Fictionarry]

I tried to train one of the sides of the shoes, using a transparent background, and found that result was normal.

Yeah, it seems the depth estimation on the virtual background to be the problem. The training may pay too much attention to the background that does not need to be reconstructed. I think excluding the unnecessary depth map by the same background mask would help (set to a maximum position, or just mask it when calculating losses), if external depth supervision is still required.

[lazyDamon]

I'll try it, thanks