Does NPKit support PyTorch?

[adu-test]

When using PyTorch for training, I found that PyTorch does not call the NCCL ncclCommDestroy method, so it cannot write the events recorded by NPKit to a file. Does NPKit support PyTorch? Your prompt response would be highly appreciated.

[adu-test]

May I ask again, what are the scenarios for using NPKit in MSCCL? Is it used to test with nccltest after a performance issue occurs?
How do you interpret the results? Do you just look at which pair of cards has relatively low communication bandwidth?

[yzygitzh]

Hi @adu-test ,

Latest PyTorch might call ncclAbort instead of ncclDestroy, so NPKit might fail to collect events in current MSCCL. Can you help try whether Azure/msccl-executor-nccl#52 fixes your issue?

NPKit is used for performance diagnosis of collective operations in applications. You can check the latency of different events (primitives) to understand where the bottleneck lies.

[adu-test]

Hi @yzygitzh ,

Thank you very much for your answer. I trained the llama3_8b model using pai-megatron-patch with nccl-2.19.3 without any issues, but encountered the following error when using msccl-executor-nccl:

Traceback (most recent call last):
  File "/workspace/Pai-Megatron-Patch/examples/llama3/../llama2/pretrain_megatron_llama.py", line 124, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/workspace/Pai-Megatron-Patch/examples/llama3/../llama2/pretrain_megatron_llama.py", line 124, in <module>
  File "/workspace/Pai-Megatron-Patch/examples/llama3/../llama2/pretrain_megatron_llama.py", line 124, in <module>
  File "/workspace/Pai-Megatron-Patch/examples/llama3/../llama2/pretrain_megatron_llama.py", line 124, in <module>
    pretrain(train_valid_test_datasets_provider, model_provider,
  File "/workspace/Pai-Megatron-Patch/megatron_patch/training.py", line 90, in pretrain
    pretrain(train_valid_test_datasets_provider, model_provider,
pretrain(train_valid_test_datasets_provider, model_provider,pretrain(train_valid_test_datasets_provider, model_provider,
  File "/workspace/Pai-Megatron-Patch/megatron_patch/training.py", line 90, in pretrain

  File "/workspace/Pai-Megatron-Patch/megatron_patch/training.py", line 90, in pretrain
  File "/workspace/Pai-Megatron-Patch/megatron_patch/training.py", line 90, in pretrain
    initialize_megatron(extra_args_provider=extra_args_provider,
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 86, in initialize_megatron
    initialize_megatron(extra_args_provider=extra_args_provider,
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 86, in initialize_megatron
    initialize_megatron(extra_args_provider=extra_args_provider,
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 86, in initialize_megatron
        _compile_dependencies()_compile_dependencies()

  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 150, in _compile_dependencies
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 150, in _compile_dependencies
    initialize_megatron(extra_args_provider=extra_args_provider,
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 86, in initialize_megatron
    _compile_dependencies()
  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 150, in _compile_dependencies
    torch.distributed.barrier()
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    torch.distributed.barrier()
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    torch.distributed.barrier()
_compile_dependencies()  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 72, in wrapper

  File "/workspace/Pai-Megatron-Patch/Megatron-LM-231007/megatron/initialize.py", line 148, in _compile_dependencies
    torch.distributed.barrier()
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3745, in barrier
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3745, in barrier
    return func(*args, **kwargs)
      File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3745, in barrier
return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3745, in barrier
    work = default_pg.barrier(opts=opts)
        work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)

torch.distributed.    DistBackendErrorwork = default_pg.barrier(opts=opts)torch.distributedtorch.distributed
: ..DistBackendErrorDistBackendErrorNCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1729, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.19.4
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
MSCCL Internal Scheduler: open algorithm in share directory /usr/share/nccl/msccl-algorithms failed: :
NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1729, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.19.4
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
MSCCL Internal Scheduler: open algorithm in share directory /usr/share/nccl/msccl-algorithms failedNCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1729, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.19.4
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
MSCCL Internal Scheduler: open algorithm in share directory /usr/share/nccl/msccl-algorithms failedtorch.distributed

.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1729, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.19.4
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
MSCCL Internal Scheduler: open algorithm in share directory /usr/share/nccl/msccl-algorithms failed
[2024-12-24 09:50:56,467] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 12784) of binary: /usr/bin/python
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.2.0a0+81ea7a4', 'console_scripts', 'torchrun')())
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 351, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
../llama2/pretrain_megatron_llama.py FAILED
------------------------------------------------------------

Part of the code in /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp is as follows:

1724 // [Note 2 ]
1725 #ifndef NCCL_HAS_COMM_NONBLOCKING
1726 C10D_NCCL_CHECK(ncclGroupEnd(), c10::nullopt);
1727 #else
1728 if (!nccl_use_nonblocking()) {
1729 C10D_NCCL_CHECK(ncclGroupEnd(), c10::nullopt);
1730 } else {
1731 C10D_NCCL_CHECK_TIMEOUT_GROUPEND(ncclGroupEnd(), ncclComms, c10::nullopt);
1732 }
1733 #endif