Skip to content

Conversation

@yoshikisd
Copy link
Collaborator

The spiritual successor to #17, this PR enables reconstruction scripts to be executed as a multi-GPU job initialized with torchrun or torch.multiprocessing.spawn.

Currently, only AdamReconstructor can take advantage of multi-GPU acceleration

Multi-GPU processing is based off of distributed data parallelism, where each subprocess/GPU

  1. works on the same model
  2. works on different parts of the dataset but, together, samples all the diffraction patterns within each epoch
  3. synchronizes gradients (after loss.backwards() calls), losses, and learning rates (if scheduler is enabled) across all participating GPUs

What's new

  • cdtools.tools.multigpu: Contains functions that enables CDTools reconstructions to run as multi-GPU jobs
  • Plotting and saving is compatible with multi-GPU: Ptycho2DDataset, CDataset, and CDIModel only saves/plots if they're running on the 'Rank 0' GPU subprocess (i.e., plots are generated only by one subprocess rather than all subprocesses).
  • Multi-GPU jobs can be started with torchrun or spawn: Two example scripts gold_ball_ptycho_torchrun.py and gold_ball_ptycho_spawn.py shows how to set up the scripts accordingly.
  • PyTests for multi-GPU: The flag --runmultigpu starts up the multi-GPU pytests, which uses up to 2 GPUs.
  • Check GPU-dependent reconstruction performance with cdtools.tools.multigpu.run_speed_test: The example script gold_ball_ptycho_speedtest.py shows how to set up the test. The implementation is also much simpler than in [feature] Multi-GPU support #17.

@yoshikisd yoshikisd linked an issue Nov 9, 2025 that may be closed by this pull request
@yoshikisd yoshikisd added the enhancement New feature or request label Nov 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for multi-GPU

2 participants