Skip to content

feat: add remote sim and isaac support #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
blankey1337 wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat: add remote sim and isaac support #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6e360a5
Improve AlohaMini URDF with realistic meshes
blankey1337 Nov 30, 2025
e9ce986
Fix AlohaMini gripper orientation and Rerun camera visualization
blankey1337 Dec 1, 2025
52b9bc9
Improve URDF meshes and fix Rerun visualization
blankey1337 Dec 1, 2025
281cd6e
Remove obsolete dashboard app
blankey1337 Dec 1, 2025
a763d90
Remove accidental files and update .gitignore
blankey1337 Dec 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.env
AlohaMini.pem
25 changes: 12 additions & 13 deletions docs/remote_simulation_workflow.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ This guide outlines how to develop, train, and test AlohaMini using a remote clo
## Architecture

* **Cloud Server (The "Lab")**: Runs NVIDIA Isaac Sim. Handles physics, rendering, and training.
* **Local Machine (The "Mission Control")**: Runs the Dashboard and Teleoperation scripts. Connects to the cloud via SSH.
* **Local Machine (The "Mission Control")**: Runs the Teleoperation script with Rerun visualization. Connects to the cloud via SSH.

## Prerequisites

Expand Down Expand Up @@ -51,23 +51,22 @@ This guide outlines how to develop, train, and test AlohaMini using a remote clo
ssh -L 5555:localhost:5555 -L 5556:localhost:5556 ubuntu@<CLOUD_IP>
```

3. **Launch Dashboard (Local)**
Start the web dashboard to see what the robot sees.
3. **Launch Teleop & Visualization (Local)**
Start the teleop script which includes Rerun for visualization.
```bash
# On Local Mac
python software/dashboard/app.py
python software/examples/alohamini/remote_teleop_rerun.py
```
Open `http://localhost:5001` in your browser.
This will open the Rerun viewer and a terminal interface for control.

4. **Teleoperate & Record**
* Use the Dashboard to see the camera feed.
* Run the teleop script in another terminal to control the robot with your keyboard:
```bash
python software/examples/alohamini/standalone_teleop.py --ip 127.0.0.1
```
* **To Record**: Click the **"Start Recording"** button on the Dashboard.
* **Controls**:
* WASD: Move Base
* Q/E: Rotate Base
* U/J: Lift Up/Down
* **To Record**: Press **'r'** in the terminal window to toggle recording.
* Perform the task (e.g., pick up the object).
* Click **"Stop Recording"**.
* Press **'r'** again to stop recording.
* Repeat 50-100 times. The data is saved to `AlohaMini/data_sim/` on the **Cloud Server**.

### Phase 2: Training
Expand Down Expand Up @@ -102,7 +101,7 @@ Test the trained model in the simulator to see if it works.
* *Coming Soon: `eval_sim.py` which loads the safetensor and drives the ZMQ robot.*

3. **Watch (Local)**:
Use the Dashboard to watch the robot perform the task autonomously.
Use `software/examples/alohamini/remote_teleop_rerun.py` (without touching keys) to watch the robot perform the task autonomously.

## Troubleshooting

Expand Down
188 changes: 188 additions & 0 deletions scripts/launch_remote_sim.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
#!/usr/bin/env python3
import subprocess
import sys
import time
import threading
import os
import signal
import re

# Configuration
PEM_FILE = "AlohaMini.pem"
REMOTE_HOST = "ubuntu@64.181.241.126"
REMOTE_IP = "64.181.241.126"

# Commands
CMD_SIM_DOCKER = [
"ssh", "-i", PEM_FILE, REMOTE_HOST, "-t",
'sudo docker run --name isaac-sim --privileged --entrypoint bash -it --gpus all '
'-e "ACCEPT_EULA=Y" -e "NVIDIA_VISIBLE_DEVICES=all" -e "NVIDIA_DRIVER_CAPABILITIES=all" '
'--rm --network=host -v ~/AlohaMini:/isaac-sim/AlohaMini nvcr.io/nvidia/isaac-sim:4.2.0 '
'-c "cd /isaac-sim && ./python.sh -m pip install \\"numpy<2.0.0\\" opencv-python-headless pyzmq && '
'./python.sh AlohaMini/software/examples/alohamini/isaac_sim/isaac_alohamini_env.py '
'--/app/livestream/enabled=true --/app/livestream/proto=ws"'
]

CMD_PORT_FWD = [
"ssh", "-i", PEM_FILE, "-N",
"-L", "5555:localhost:5555",
"-L", "5556:localhost:5556",
"-L", "8211:localhost:8211",
REMOTE_HOST
]

CMD_LOCAL_TELEOP = [
sys.executable, "software/examples/alohamini/remote_teleop_rerun.py"
]

def stream_reader(pipe, prefix, stop_event, on_match=None, match_pattern=None):
"""Reads from a pipe and prints to stdout, optionally triggering a callback on match."""
try:
for line in iter(pipe.readline, ''):
if stop_event.is_set():
break
if not line:
break
# Print output with prefix
# Clean up the line: handle \r and whitespace
# Use \r\n to ensure we move to the next line properly
clean_line = line.replace('\r', '').strip()

if clean_line:
print(f"[{prefix}] {clean_line}\r")

if match_pattern and on_match:
if match_pattern in line:
on_match()
# We only trigger once
on_match = None
except Exception:
pass

def cleanup_remote():
"""Stops the remote docker container and ensures no lingering processes."""
print("Cleaning up remote environment...")
cmd = ["ssh", "-i", PEM_FILE, REMOTE_HOST, "sudo docker stop isaac-sim || true && sudo docker rm isaac-sim || true"]
try:
subprocess.run(cmd, check=False, timeout=10, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
except Exception as e:
print(f"Warning: Remote cleanup failed: {e}")

def sync_code():
print("Syncing local code to remote server...")
# Sync software folder to ensure remote has latest changes
# This creates ~/AlohaMini/software if it doesn't exist, or updates it
cmd = ["scp", "-i", PEM_FILE, "-r", "software", f"{REMOTE_HOST}:~/AlohaMini/"]
try:
subprocess.run(cmd, check=True)
print("Sync complete.")
except subprocess.CalledProcessError as e:
print(f"Error syncing code: {e}")
sys.exit(1)

def main():
if not os.path.exists(PEM_FILE):
print(f"Error: {PEM_FILE} not found in current directory.")
print("Please run this script from the project root where the .pem file is located.")
sys.exit(1)

print("Starting Remote Isaac Sim Workflow...")
print("-------------------------------------")

# Initial cleanup to ensure fresh state
cleanup_remote()

# Sync code
sync_code()

# Events and State
stop_event = threading.Event()
sim_ready_event = threading.Event()

processes = []

def on_sim_ready():
print("\n>>> SIMULATION READY DETECTED! Starting Port Forwarding... <<<\n")
sim_ready_event.set()

# 1. Start Simulation (Remote Docker)
print(f"Step 1: Launching Isaac Sim container on {REMOTE_HOST}...")
sim_process = subprocess.Popen(
CMD_SIM_DOCKER,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1
)
processes.append(sim_process)

# Start monitoring thread for Sim output
sim_thread = threading.Thread(
target=stream_reader,
args=(sim_process.stdout, "REMOTE", stop_event, on_sim_ready, "Isaac Sim AlohaMini running")
)
sim_thread.daemon = True
sim_thread.start()

try:
# Wait for Sim to be ready
print("Waiting for simulation to initialize (this may take a minute)...")
while not sim_ready_event.is_set():
if sim_process.poll() is not None:
print("Error: Simulation process exited prematurely.")
sys.exit(1)
time.sleep(1)

# 2. Start Port Forwarding
print("Step 2: Starting SSH Port Forwarding...")
fwd_process = subprocess.Popen(
CMD_PORT_FWD,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
processes.append(fwd_process)

# Give it a moment to establish
time.sleep(3)
if fwd_process.poll() is not None:
print("Error: Port forwarding failed to start.")
# Check stderr
print(fwd_process.stderr.read().decode())
# Continue anyway? No, teleop needs it.
sys.exit(1)
else:
print("Port forwarding established.")

# 3. Start Local Teleop
print("Step 3: Starting Local Teleop...")
print("-------------------------------------")
print("Use the keyboard commands below to control the robot.")

teleop_process = subprocess.Popen(CMD_LOCAL_TELEOP)
processes.append(teleop_process)

# Wait for teleop to finish (user quits)
teleop_process.wait()

except KeyboardInterrupt:
print("\nStopping workflow...")
finally:
stop_event.set()
print("Cleaning up processes...")

# Kill all started processes
for p in processes:
if p.poll() is None:
p.terminate()
try:
p.wait(timeout=2)
except subprocess.TimeoutExpired:
p.kill()

# Explicit remote cleanup
cleanup_remote()

print("Done.")

if __name__ == "__main__":
main()
58 changes: 58 additions & 0 deletions scripts/setup_remote_sim.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/bin/bash
set -e

# Configuration
REPO_URL="https://github.com/blankey1337/AlohaMini.git"
ISAAC_IMAGE="nvcr.io/nvidia/isaac-sim:2023.1.1"

echo ">>> Starting Remote Setup..."

# 1. Install Docker & NVIDIA Container Toolkit if missing
if ! command -v docker > /dev/null 2>&1; then
echo ">>> Installing Docker..."
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker "$USER"
fi

if ! dpkg -l | grep -q nvidia-container-toolkit; then
echo ">>> Installing NVIDIA Container Toolkit..."
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
fi

# 2. Docker Login
if [ -z "$NVIDIA_API_KEY" ]; then
echo ">>> Please export NVIDIA_API_KEY before running this script."
exit 1
fi

echo ">>> Logging into NGC..."
echo "$NVIDIA_API_KEY" | sudo docker login nvcr.io -u '$oauthtoken' --password-stdin

# 3. Clone Repo
if [ -d "AlohaMini" ]; then
if [ -d "AlohaMini/.git" ]; then
echo ">>> AlohaMini already cloned. Pulling latest..."
cd AlohaMini && git pull && cd ..
else
echo ">>> AlohaMini directory exists but is not a git repo. Removing and cloning..."
rm -rf AlohaMini
git clone "$REPO_URL"
fi
else
echo ">>> Cloning AlohaMini..."
git clone "$REPO_URL"
fi

# 4. Pull Isaac Sim Image
echo ">>> Pulling Isaac Sim Image ($ISAAC_IMAGE)..."
sudo docker pull "$ISAAC_IMAGE"

echo ">>> Setup Complete!"
Binary file added software/examples/alohamini/__pycache__/remote_teleop_rerun.cpython-310.pyc
View file
Open in desktop
Binary file not shown.
7 changes: 7 additions & 0 deletions software/examples/alohamini/evaluate_bi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
#!/usr/bin/env python3

import sys
import os

# Add repo root to path
repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../.."))
sys.path.append(repo_root)

from lerobot.datasets.lerobot_dataset import LeRobotDataset
from lerobot.datasets.utils import hw_to_dataset_features
from lerobot.policies.act.modeling_act import ACTPolicy
Expand Down
Binary file added software/examples/alohamini/isaac_sim/__pycache__/isaac_alohamini_env.cpython-310.pyc
View file
Open in desktop
Binary file not shown.
Loading