Support Image Rendering for Vision-Based Research

[iitimii]

Add support for rendering RGB (and optionally depth/segmentation) images in the simulation using MuJoCo’s camera API.

[iitimii]

Curious if you have any suggestions or pointers on how to approach this, @amacati .

For camera setup — should it be fixed, attached to the drone’s body, or tracking the drone?

Also, should image rendering be implemented inside the simulation core or on top of the gymnasium environments? Any other nuances to be aware of?

[amacati]

It should definitely be implemented as a sensor on the drones using native mujoco functions. One would probably have to include the sensor in the drone xml.

One thing that is critical is that the simulation should not pay the cost of adding the sensor if images are not being rendered. This includes potential buffers for images. Otherwise, we probably won't be able to initialize sims with 1M drones anymore, and the regular sim would slow down for no reason.

It's probably best to look at some examples that include cameras in mujoco, then port that to mjx and finally integrate it into our stack