D435i depth image: How to properly resize and normalize 640x480 to 64x64 for simulation alignment?

[TaoJiang-code]

Hi,

I have successfully reproduced sim-to-sim results, but I am having trouble with sim-to-real.

I am using an Intel RealSense D435i camera to capture depth images at 640x480 resolution. In my simulation (MuJoCo), the depth camera outputs 64x64 images. I want to process the real depth images to match the simulation input.

My current processing pipeline is:

# Get 640x480 depth image
depth_image = np.asanyarray(depth_frame.get_data())

# Resize to 64x64 to match MuJoCo simulation
depth_image = cv2.resize(depth_image, (64, 64), interpolation=cv2.INTER_AREA)

# Normalize to 0~1 and center to -0.5~0.5
depth_far_clip = 2.0
depth_near_clip = 0.0
depth_image = np.clip(depth_image, depth_near_clip, depth_far_clip)
depth_image = (depth_image - depth_near_clip) / (depth_far_clip - depth_near_clip) - 0.5

Convert to torch tensor and crop/interpolate to 64x64

depth_image = torch.tensor(depth_image, dtype=torch.float32)
clip_left = 10
clip_top = 20
clip_right = 10
clip_bottom = 2
cropped = depth_image[clip_top:-clip_bottom, clip_left:64-clip_right]
cropped = depth_image
cropped = cropped.unsqueeze(0).unsqueeze(0)  # [1,1,H,W]
resized = F.interpolate(cropped, size=(64, 64), mode='bilinear', align_corners=False).squeeze(0).squeeze(0)

# Display
cv2.namedWindow('depth image', cv2.WINDOW_NORMAL)
cv2.imshow('depth image', (resized.detach().numpy() + 0.5) * 255)
cv2.waitKey(1)

Is there anything wrong with this approach?
Is there a better way to align the real depth image with the simulation input?

[dwei-k]

Please refer to this issue #11 for how to process depth images in real world.
The general pipeline involves processing the raw images using pyrealsense’s built-in filters, followed by clipping the depth values to align with the simulation (typically with a 2m maximum). Depending on the requirements, gaussian blur or inpaint is applied. Finally, the data is normalized to the [-0.5, 0.5] range and resized to (64, 64)