Asymmetric bandwidth in bidirectional Host-Device transfers: H2D drops to ~11 GB/s while D2H maintains ~25 GB/s

[chanpapa]

Environment

Hardware:

• GPU: 8x NVIDIA A100-SXM4-80GB
• CPU: Dual socket (NUMA 0: GPU0-3, NUMA 1: GPU4-7)
• Memory: DDR4 8-channel per CPU socket
• Topology: PCIe 4.0 (see below)

Software:

• nvbandwidth Version: v0.8
• CUDA Runtime: 12.6.0
• CUDA Driver: 12.6.0
• Driver Version: 560.35.05
• OS: Linux

GPU Topology:

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      PXB     NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0
GPU1    PXB      X      NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0
GPU2    NODE    NODE     X      PXB     SYS     SYS     SYS     SYS     0-31,64-95      0
GPU3    NODE    NODE    PXB      X      SYS     SYS     SYS     SYS     0-31,64-95      0
GPU4    SYS     SYS     SYS     SYS      X      PXB     NODE    NODE    32-63,96-127    1
GPU5    SYS     SYS     SYS     SYS     PXB      X      NODE    NODE    32-63,96-127    1
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      PXB     32-63,96-127    1
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    PXB      X      32-63,96-127    1

---

Description

When running bidirectional Host-Device memory copy tests, I observe significant asymmetry in bandwidth:

• Unidirectional transfers (expected behavior):

  • H2D: ~25 GB/s ✅
  • D2H: ~25 GB/s ✅

• Bidirectional transfers (unexpected behavior):

  • D2H: ~24 GB/s ✅ (maintains performance)
  • H2D: ~11 GB/s ❌ (drops by ~56%)

---

Test Results

1. Unidirectional Host-to-Device (Baseline)

$ ./nvbandwidth -t host_to_device_memcpy_ce

Result: ~24.8 GB/s per GPU (SUM: 198.72 GB/s)

memcpy CE CPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     24.81     24.80     24.84     24.83     24.88     24.83     24.84     24.88

2. Unidirectional Device-to-Host (Baseline)

$ ./nvbandwidth -t device_to_host_memcpy_ce

Result: ~25.9 GB/s per GPU (SUM: 207.13 GB/s)

memcpy CE CPU(row) <- GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     25.89     25.88     25.89     25.89     25.89     25.89     25.89     25.90

3. Bidirectional Device-to-Host

$ ./nvbandwidth -t device_to_host_bidirectional_memcpy_ce

Result: ~24.2 GB/s per GPU (SUM: 194.07 GB/s) ✅ Maintains performance

memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     24.25     24.26     24.24     24.26     24.27     24.26     24.27     24.27

4. Bidirectional Host-to-Device ⚠️ ISSUE HERE

$ ./nvbandwidth -t host_to_device_bidirectional_memcpy_ce

Result: ~11 GB/s per GPU ❌ (56% performance drop)

(Note: The output appears to be truncated in the original test, but based on the pattern, H2D bandwidth drops significantly)

---

Expected Behavior

Since PCIe is full-duplex, I expect:

• Bidirectional H2D bandwidth: ~24-25 GB/s (similar to unidirectional)
• Bidirectional D2H bandwidth: ~24-25 GB/s (similar to unidirectional)

Both directions should maintain near-peak performance simultaneously.

---

Observations

1. D2H maintains performance in bidirectional mode (~24 GB/s)
2. H2D drops significantly in bidirectional mode (~11 GB/s, ~56% reduction)
3. Memory bandwidth is not saturated (verified with pcm-memory - DDR4 8-channel not fully utilized)
4. PCIe 4.0 x16 theoretical bandwidth: ~32 GB/s per direction (full-duplex)

---

Questions & Request for help

Could you please help clarify:

1. Whether this asymmetry is expected on this hardware configuration
2. What might be causing the H2D throttling in bidirectional mode
3. Any recommendations for investigation or mitigation

Thank you!

[deepakcu]

What is the CPU manufacturer/model? Does the system attach pairwise GPUs via PCIE switches? It is hard to tell whether asymmetry is expected for the hardware configuration since system topology, PCI configuration can affect bidirectional bandwidth.

[chanpapa]

@deepakcu Thanks for the reply! Here is the detailed hardware information:

1. CPU Information

• Model: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake-SP)
• Architecture: Dual-Socket system (NUMA 0 & NUMA 1)

2. PCIe Topology & Switches

Yes, the GPUs are attached pairwise via PCIe Switches.

Based on lspci -tv and the topology matrix, the system has 4 PCIe Root Complexes, and each Root Complex connects to 2 GPUs via a PCIe Switch hierarchy.

Topology Layout:

• Root Complex 1 (NUMA 0): Connects GPU 0 & GPU 1 (via PCIe Switch)
• Root Complex 2 (NUMA 0): Connects GPU 2 & GPU 3 (via PCIe Switch)
• Root Complex 3 (NUMA 1): Connects GPU 4 & GPU 5 (via PCIe Switch)
• Root Complex 4 (NUMA 1): Connects GPU 6 & GPU 7 (via PCIe Switch)

3. Additional Configuration & Hypothesis

1. MRRS Configuration: I have verified that the Max Read Request Size (MRRS) is set to the maximum value of 4096 bytes for all devices, so this shouldn't be the bottleneck.

2. Hypothesis regarding Root Complex Arbitration:
   Since D2H (GPU writing to System Memory) maintains high throughput while H2D (CPU writing to GPU Memory) suffers during bidirectional traffic, could this be an issue with the Root Complex (RC) Arbiter?

   Is it possible that when the RC is under heavy load, it prioritizes processing Inbound Posted Writes (D2H) to clear its internal buffers and prevent overflow, thereby starving the Outbound Posted Writes (H2D)? I suspect the RC might be throttling CPU-to-PCIe traffic to handle the incoming flood from the GPUs.

[deepakcu]

Can you try enabling PCI RelaxedOrdering
https://forums.developer.nvidia.com/t/how-to-enable-pcie-relaxed-ordering-for-gpu-direct-rdma-p2p-load-stores/326323/2

[chanpapa]

Hi @deepakcu,

Thank you so much for the quick response and the link!

Following your advice, I double-checked the PCIe configuration. It appears that PCI Relaxed Ordering is indeed already enabled (RlxdOrd+) on all the GPUs and the PCIe Bridges.

Here is the lspci output confirming the status:

$ sudo lspci -vv -d 10de: | grep -E "^[0-9a-f]{2}:[0-9a-f]{2}\.|RlxdOrd"
10:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
16:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
48:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
4c:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
53:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
54:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
55:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
56:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
57:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
58:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
8a:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
8f:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
c6:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
ca:00.0 3D controller: NVIDIA Corporation Device 20b2 (rev a1)
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-

Unfortunately, the Copy Engine (memcpy_ce) performance remains unchanged at around 11-12 GB/s:

Running host_to_device_bidirectional_memcpy_ce.
memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     11.89     11.31     11.83     12.04     11.62     11.71     11.64     11.70

SUM host_to_device_bidirectional_memcpy_ce 94.74

However, I would like to share an interesting observation that might be helpful. When I switch to using the SM for the copy (memcpy_sm option), the bidirectional bandwidth is much better and roughly symmetric (reaching ~21 GB/s).

Here is the memcpy_sm result:
D2H:

$ ./nvbandwidth -t device_to_host_bidirectional_memcpy_sm -b 2000
...
Running device_to_host_bidirectional_memcpy_sm.
memcpy SM CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     21.52     21.48     21.54     21.55     21.57     21.58     21.47     21.47

SUM device_to_host_bidirectional_memcpy_sm 172.18

H2D:

$ ./nvbandwidth -t host_to_device_bidirectional_memcpy_sm -b 2000
...
Running host_to_device_bidirectional_memcpy_sm.
memcpy SM CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     21.29     18.06     21.53     21.55     20.11     21.49     21.54     21.52

SUM host_to_device_bidirectional_memcpy_sm 167.09

Since the SM-based copy can achieve high throughput, it seems the physical link capability is sufficient. Do you have any insights on why the Copy Engine (CE) might be bottlenecked while the SM is not?

Thanks again for your time and support

[deepakcu]

RlxdOrd+ indicates RO is supported, can you enable it on the GPU side
sudo modprobe nvidia NVreg_SetPCIERelaxedOrdering=1

[chanpapa]

Hi @deepakcu,

Thanks for the suggestion. I checked the driver parameters, and you were right—EnablePCIERelaxedOrderingMode was indeed disabled (0) by default.

$ cat /proc/driver/nvidia/params | grep "Relaxed"
EnablePCIERelaxedOrderingMode: 0

I followed your advice and reloaded the driver to enable it:
sudo modprobe nvidia NVreg_EnablePCIERelaxedOrderingMode=1

I confirmed that it is now successfully enabled:

$ cat /proc/driver/nvidia/params | grep "Relaxed"
EnablePCIERelaxedOrderingMode: 1

However, the performance results remain virtually unchanged. The H2D Copy Engine bandwidth is still stuck around 11-12 GB/s, as shown below:

$ ./nvbandwidth -t host_to_device_bidirectional_memcpy_ce -b 2000
...
Running host_to_device_bidirectional_memcpy_ce.
memcpy CE CPU(row) <-> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0     11.76     11.66     11.75     11.68     11.74     11.67     11.69     11.67

Since enabling Relaxed Ordering didn't resolve the issue, are there any other factors that could specifically throttle the Copy Engine?

Thanks again.