Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning

Tianchong Jiang¹, Jingtian Ji¹, Xiangshan Tan¹, Jiading Fang^2,*, Anand Bhattad³, Vitor Guizilini^4,†, Matthew R. Walter^1,†

¹TTIC   ²Waymo   ³Johns Hopkins University   ⁴Toyota Research Institute

Installation

First, clone the repo and cd into it.

git clone https://github.com/ripl/CamPoseOpensource
cd CamPoseOpensource

Then, run the setup script. It will setup the conda environment and download data.

bash setup.sh

Activate the conda environment with

conda activate know_your_camera

If you only need ManiSkill or robosuite, comment out the lines to install other one.

How to run

You can run training in robosuite with

python policy_robosuite/train.py

or in ManiSkill with

python policy_maniskill/train.py

Plücker Snippet

To add camera conditioning to your policy, you can use the following minimalist snippet to get Plücker raymap from intrinsics and extrinsics. (It assumes OpenCV convention i.e. image origin at top-left, +z is forward.)

import torch

def get_plucker_raymap(K, c2w, height, width):
    """intrinsics (3,3), cam2world (4,4), height int, width int"""
    vv, uu = torch.meshgrid(
        torch.arange(height, device=K.device, dtype=K.dtype) + 0.5,
        torch.arange(width, device=K.device, dtype=K.dtype) + 0.5,
        indexing="ij",
    )
    rays = torch.stack([uu, vv, torch.ones_like(uu)], dim=-1)
    d_world = torch.nn.functional.normalize(
        (rays @ torch.linalg.inv(K).T) @ c2w[:3, :3].T,
        dim=-1,
        eps=1e-9,
    )
    o = c2w[:3, 3].view(1, 1, 3)
    m = torch.cross(o, d_world, dim=-1)
    return torch.cat([d_world, m], dim=-1)                         

BibTeX

If you find this work useful, please cite:

@article{jiang2025knowyourcamera,
  title     = {Do You Know Where Your Camera Is? {V}iew-Invariant Policy Learning with Camera Conditioning},
  author    = {Tianchong Jiang and Jingtian Ji and Xiangshan Tan and Jiading Fang and Anand Bhattad and Vitor Guizilini and Matthew R. Walter},
  journal   = {arXiv preprint arXiv:2510.02268},
  year      = {2025},
}