FW Desktop BIOS 3.04 PCI-E 4x slot failing most of the times to get passed boot

[stau]

When using a PCI-E 4x NVMe adapter WITH NVMe disk fails to get passed the BIOS most of the times.

Sanity checks

• Funnily enough, when there is no disk, we do get passed the BIOS.
• The NVMe disk works fine in the normal slots.
• The NVMe adapters tested work fine in other systems and can access the NVMe disk
  • Silverstone ECM22
  • Icybox IB-PCI208-HS
• Using the Arch Linux installer; when in the off-chance we manage to get passed boot, I managed to see the following errors nvme nvme2: I/O tag 24 (1018) QID 0 timeout, disable controller nvme nvme2: failed to read smart log (error -5) nvme 0000:c2:00.0: probe with drive nvme failed with error -5

The set-up with the Icybox IB-PCI208-HS booted fine (aka got passed the BIOS flash screen) with the BIOS 3.02, however had soft lock-ups when installing the [Proxmox VE 9.1 ](Proxmox VE 9.1 ISO Installer

I will attempt to downgrade the BIOS to 3.02; since I have successfully installed Proxmox VE 9.1 on BIOS 3.04 and report back. Just to note here, when I was on the BIOS 3.02 I got the following errors when trying to install Proxmox VE 9.1:

BUG: soft lockup - CPU#1 stuck for 26s

I tried setting various kernel parameters, but to no avail.

The failing to get passed BIOS was ESPECIALLY confusing and alarming, as the PCI-E adapter was connected during the BIOS update. Obviously coming back to a black screen after a BIOS update was not encouraging.

Edit#1:
Since I did not use fwupgrade to perform the BIOS, I could only downgrade to BIOS 3.03.

• With PCI-E expansion:
  • The behavior is that it takes a long time to get to the BIOS to load and then it fails to boot Proxmox being stuck at "Loading initial ramdisk".
• Without PCI-E expansion:
  • works

Edit#2:

• I cannot find where I can version 3.02 of the BIOS, so that's as far as it goes as getting that part to work. However the softlock issues which prompted me to update the BIOS were on 3.02...

[kiram9]

@stau can you see if changing the PCIE speed to GEN3 improves this issue?

[stau]

@kiram9 With these exact settings as above with the PCI E slot speed set to GEN3 has the same behavior.

Some extra information here, I have disabled quick boot; and it does not even reach the "memory check". Just a _ on the top left of a black screen. And enabling nor disabling quick boot has any effect (as expected). Just hoping to give more clarity at which part of the boot process it gets stuck.§
Very rarely, and I have not been able to figure out why, it passes the boot process and proceeds to boot the OS... just to get stuck at "Loading initial ram disk".