OpenNIC + QDMA on CPM5: which PCIe BAR is QDMA config/CSR, and what config_bar should I use?

[nehalbaganal]

Hi,

I’m trying to bring up the QDMA IP in an OpenNIC shell design and access it either:

• From userspace via VFIO (mapping the QDMA CSR BAR and programming queue contexts), or

• Via the AMD QDMA PF kernel driver from dma_ip_drivers/QDMA/linux-kernel.

In both paths I hit issues that look like the QDMA config / CSR BAR is not where I think it is, or not wired as the driver expects.

I’d appreciate guidance on:

• Which PCIe BAR OpenNIC shell uses for QDMA config/CSR (AXI-lite)

• How to correctly set the config_bar for this design

What I tried

1) Direct CSR access via VFIO

I bound vfio-pci to 0000:e1:00.0 and mapped BAR0/BAR2, then tried to read the EQDMA CPM5 indirect context registers at offsets from eqdma_cpm5_reg.h:

EQDMA_CPM5_IND_CTXT_DATA_ADDR = 0x804

EQDMA_CPM5_IND_CTXT_MASK_ADDR = 0x824

EQDMA_CPM5_IND_CTXT_CMD_ADDR = 0x844

For both BAR 0 and BAR 2, reads always return 0xffffffff:

[initial] CTXT_CMD = 0xffffffff, CTXT_MASK = 0xffffffff, CTXT_DATA0 = 0xffffffff
So it looks like I’m not actually hitting the QDMA CSR block.

2) Using the PF driver with config_bar

With QDMA PF driver:

sudo insmod ./qdma-pf.ko config_bar="e1:0:0"
# also tried "e1:0:2", "225:0:0", "225:0:2"

dmesg shows:

qdma_pf:qdma_mod_init: Xilinx QDMA PF Reference Driver v2024.1.0.0.
qdma_pf:probe_one: 0000:e1:00.0: func 0x0, p/v 0/0...
qdma_pf:probe_one: Configuring 'e1:00:0' as master pf
qdma_pf:qdma_device_open: qdma-pf, e1:00.00, ... 0x10ee:0x903f.
qdma_is_config_bar: Invalid config bar, err:-4
qdma_hw_access_init: config bar passed is INVALID, err:-1
qdma-pf: probe of 0000:e1:00.0 failed with error -22

# same for 0000:e1:00.1

So the driver is rejecting my config_bar value as “Invalid config bar”.

Any pointers regarding these issues will be helpful.
Thank You