Instance segmentation

[abrahamezzeddine]

Hello,

I am using your wonderful code to train a deep learning model. Just wondering, if I would like to add, in addition to XYZ, color and class labeling, instance labeling as well, how would be the best setup for this with your code if I want to perform instance semantic segmentation? Does your code support instance segmentation (like a room for example).

Not sure if I should add it together with XYZ and RGB ?

Thank you!

[kentechx]

It doesn't support instance segmentation.

XYZ represents the geometry of a point cloud and should be kept separate from the network input. Therefore, it's better not to add XYZ with RGB. PointNext already utilizes XYZ in the form of relative positions of neighboring points, similar to the way CNNs operate. Note that CNNs do not require the XY coordinates of image pixels; instead, the convolution process inherently encodes neighboring pixel information.

Another drawback of including XYZ as part of the network input is that it largely increases the time required to fit the dataset. This is because XYZ values will change after each transformation (such as rotation and translation), even though the scene and objects remain the same.

[abrahamezzeddine]

Thank you!

Do you then have any suggestion on how to solve the S3DIS dataset with your code where values such as RGB, Normals and segments are included? (Not the S3DIS dataloader, but rather how to define the input to your PointNeXt model).

Would this be correct?

outputs = model(colors, coords)?
And if I want to add normals etc, I just concatenate colors with normals and so on?

features = rgb+normals

outputs = models(features, coords)? Because in a previous thread you mentioned that
"X is the feature of points, consisting of XYZ, normals, colors, etc., whereas XYZ coordinates are used for sampling." So it should not consist of XYZ? :D

[kentechx]

Thank you!

Do you then have any suggestion on how to solve the S3DIS dataset with your code where values such as RGB, Normals and segments are included? (Not the S3DIS dataloader, but rather how to define the input to your PointNeXt model).

Would this be correct?

outputs = model(colors, coords)? And if I want to add normals etc, I just concatenate colors with normals and so on?

features = rgb+normals

outputs = models(features, coords)? Because in a previous thread you mentioned that "X is the feature of points, consisting of XYZ, normals, colors, etc., whereas XYZ coordinates are used for sampling." So it should not consist of XYZ? :D

Yes, you’re correct. In the issue you mentioned, I meant that if X doesn’t include XYZ coordinates, the network will not take XYZ as input.

[abrahamezzeddine]

Ok thanks.

Would you say it is better to feed a whole room as a single segment and let pointnext perform ball query processing, etc… or should I split up a room in patches/segments of meter x meter and feed those and nearby patches with an overlap?

what do you think is the best? Thanks!!

[kentechx]

For room scenarios, it's better to feed a whole room if possible. The sampling rate (voxel size) of the entire point cloud is more important. You should use the sample sampling rate for both training and inference, as it influences the field of view of the network kernel.

An example is https://github.com/kentechx/s3dis_sem_experiments/blob/main/code/train_pointnext.py