myst

Myst is a combination of Stable Diffusion and VGGT (Visual Geometry Grounded Transformer)/DepthAnything/Metric3D to create worlds that are 3D aware and go beyond outpainting.

Simply, Myst lets you start with a text prompt, "an image of a kitchen", generates a 2D image from it using diffusion, lifts the 2D image to 3D, and then lets you navigate around the scene. As you rotate/move to reveal unexplored parts of the scene, you can run additional diffusion steps and new images are generated and then combined into a single, coherent, 3D scene.

Update: We've replaced Dust3r/Mast3r with VGGT, the CVPR 2025 Best Paper Award winner, for faster and more accurate 3D reconstruction.

We can create infinite 3D scenes, for use as a potential dataset. We can manually create these worlds, or do it automatically.

Architecture: Showing how we created these worlds.

Real-time Generation: A real-time recording of me creating a kitchen scene using Stable Diffusion and Dust3r.

Automatic Dataset: Showing a few automatic datasets.

---

A Few Scenes

LOTS of bay windows..	What happens in a long hallway?	Monastery tunnels
Full 360 beach views	Kitchen meets a fireplace	Nice wood oak paneling
Super mario kitchen-land	Severance hallway?	More beach and ocean views
Spacious bedroom kitchens	Fireplace bedroom kitchens	Interesting ceilings

Synthetic Dataset

Urban spook	Mountains and ducks
Marble, books, plants	Buddha, cape town, aerial
Venice and ruins	More kitchens

Installation

# Clone the repository
git clone https://github.com/your-repo/myst.git
cd myst

# Run setup (installs all dependencies and downloads model checkpoints)
chmod +x setup_env.sh
./setup_env.sh

Prerequisites: Python 3.12+, CUDA 12.x+, uv package manager

Run

Default mode now uses VGGT for 3D reconstruction:

python run.py --depth vggt

You can also use other depth estimation methods:

• --depth vggt (default): VGGT - fastest and most accurate (CVPR 2025 Best Paper)
• --depth metric: Metric3D
• --depth da: Depth Anything
• --depth dust: Dust3r (requires ./setup_env.sh --with-dust3r)
• --depth mast3r: Mast3r (requires ./setup_env.sh --with-dust3r)

Command Line Options

python run.py [OPTIONS]

Options:
  --headless              Don't show GUI
  --depth DEPTH           vggt / metric / da / dust
  --renderer RENDERER     raster / py3d
  --prompt PROMPT         me / doors / auto / combo / default
  --control CONTROL       me / doors / auto
  --intrinsics INTRINSICS dummy / pf
  --image IMAGE           gen / path
  --model MODEL           sd2 / if

Examples

# Interactive mode with VGGT
python run.py --depth vggt --prompt auto --control auto

# Headless mode for dataset generation
python run.py --headless --depth vggt --prompt auto --control auto

# Use specific image as starting point
python run.py --image path/to/your/image.jpg --depth vggt