AMD Single GPU Passthrough

Hardware specifications at the point of inital writing

* Operating System: EndeavourOS
* DE: KDE Plasma
* Graphics Platform: Wayland
* GPU Drivers: Mesa/amdgpu
* Processors: AMD Ryzen 9 5900X
* Memory: 32 GB of RAM
* Graphics Processor: Sapphire Nitro R9 Fury
* Motherboard: ASUS Tuf Gaming X570 Plus

Current hardware specifications

* Operating System: CachyOS
* DE: KDE Plasma
* Graphics Platform: Wayland
* GPU Drivers: Mesa/amdgpu
* Processors: AMD Ryzen 9 9950X3D
* Memory: 64 GB of RAM
* Graphics Processor: AsRock 7900XTX Taichi
* Motherboard: AsRock X870E Taichi

Preparations
Preparing boot
Configuring Libvirt
Setting up Windows VM
Hook scripts
- Editing hooks
GPU not resetting properly
Export GPU ROM
Passthrough (virt-manager)
- USB controllers passthrough
- Integrated BT adapter passthrough
Final checks
Internet not available in the VM
Improving VM and CPU performance
EAC games detecting VM
Logging

Preparations

All scripts in this guide are written for Arch-based systems. If you are not on an Arch-based system, follow the scripts and execute commands appropriate for your OS.

To prepare, make sure you have virtualization features enabled in your BIOS.

For AMD enable
- IOMMU
- NX Mode
- SVM Mode
For Intel enable
- VT-d
- VT-x

Clone the repository:

 git clone https://github.com/stele95/AMD-Single-GPU-Passthrough && cd AMD-Single-GPU-Passthrough

Preparing boot

If you are using GRUB, run the appropriate script based on your cpu:

AMD: sudo ./grub_setup_amd.sh
Intel: sudo ./grub_setup_intel.sh

If you are using other boot systems (systemd-boot, etc), scripts will not work. You will have to manually add the following to the kernel parameters:

AMD: amd_iommu=on iommu=pt video=efifb:off
Intel: intel_iommu=on iommu=pt

Configuring Libvirt

To configure libvirt run the script which configures libvirt and QEMU by typing ./libvirt_configuration.sh. Restart the PC after the script finishes.

Setting up Windows VM

Open the virt-manager and prepare Windows iso, I used sata and qcow2 for the disk type. For Windows 11, you need to have over 54 GB of storage space.
You can use virtio disks for improved performance, but regular sata works fine for initial setup. You can read more about it here and you'll need drivers from virtio-win.iso.
Use the Q35 chipset and x64/OVMF_CODE.secboot.4m.fd bootloader.

For Windows 11 installation, add a TPM emulator in your xml file:

<tpm model="tpm-tis">
  <backend type="emulator" version="2.0"/>
</tpm>

Your VM settings should look similar to this before starting the installation. You can remove all unnecessary devices before starting the installation.
Disable memballoon in your xml file:
```
<memballoon model="none"/>
```

Add these to your XML for improved performance (not sure if this works for Intel). Check the win11.xml example file for proper placement of each section.

XML Configs

Enabling Hyper-V enlightenments (Windows only)

<hyperv mode='custom'>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
  <vpindex state='on'/>
  <runtime state='on'/>
  <synic state='on'/>
  <stimer state='on'/>
  <reset state='on'/>
  <vendor_id state='on' value='AuthenticAMD'/>  <!-- The value doesn't matter -->
  <frequencies state='on'/>
  <reenlightenment state='off'/>   <!-- We use only one guest. Not fully supported on KVM, disable it. -->
  <tlbflush state='on'/>
  <ipi state='on'/>
  <evmcs state='off'/> 		<!-- We do not use nested KVM in Hyper-v -->
</hyperv>

KVM features

Add this below </hyperv> tag

<kvm>
  <hidden state='on'/>
  <hint-dedicated state='on'/>
</kvm>
<vmport state='off'/>
<ioapic driver='kvm'/>

Passthrough mode and policy

<cpu mode='host-passthrough' check='none' migratable='on'>  <!-- Set the cpu mode to passthrough -->
  <topology sockets='1' dies='1' cores='8' threads='2'/>    <!-- Match the cpu topology. In my case 8c/16t, or 2 threads per each core -->
  <cache mode='passthrough'/>                     <!-- The real CPU cache data reported by the host CPU will be passed through to the virtual CPU -->
  <feature policy='require' name='topoext'/>  <!-- Required for the AMD CPUs -->
  <feature policy='require' name='svm'/>
  <feature policy='require' name='apic'/>         <!-- Enable various features improving behavior of guests running Microsoft Windows -->
  <feature policy='disable' name='hypervisor'/>
  <feature policy='require' name='invtsc'/>
</cpu>

Timers

<clock offset="localtime">
  <timer name="rtc" present="no" tickpolicy="catchup"/>
  <timer name="pit" present="no" tickpolicy="delay"/>
  <timer name="hpet" present="no"/>
  <timer name="kvmclock" present="no"/>
  <timer name="hypervclock" present="yes"/>
  <timer name="tsc" present="yes" mode="native"/>
</clock>

Additional libvirt attributes

<devices>
...
  <memballoon model='none'/>    <!-- Disable memory ballooning -->
  <panic model='hyperv'/>	<!-- Provides additional crash information when Windows crashes -->
</devices>

Additional QEMU agrs

You will have to modify virtual machine domain configuration to <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
  </devices>
  <qemu:commandline>
    <qemu:arg value='-overcommit'/>
    <qemu:arg value='cpu-pm=on'/>
  </qemu:commandline>
</domain>

Hook Scripts

There is an amazing hook script made by @risingprismtv on gitlab. What this script does is stop your display manager service and all of your running programs, and unhooks your graphics card off of Linux and rehooks it onto the Windows VM.

Clone Risngprism's single GPU passthrough gitlab page: git clone https://gitlab.com/risingprismtv/single-gpu-passthrough && cd single-gpu-passthrough.
Run the install script as sudo: sudo ./install_hooks.sh.
The scripts will successfully install into their required places without issue!

Editing hooks

Edit the hooks script located at /etc/libvirt/hooks/qemu
On the line with the if then statement change the name to the name of your VM.
Now you should be good to turn on your VM! On Windows drivers will auto install.

GPU not resetting properly

After setting everything up, you just get a black screen once you start your VM and the GPU fans speed up, or your VM shut down doesn't return you to the host login screen, check logs at /var/log/libvirt/qemu/{VM name}.log for a vfio: Cannot reset device {gpu device id}, no available reset mechanism.. If you have that error, add rtcwake -m mem -s 3 at the end of the /usr/local/bin/vfio-startup script and at the start of the /usr/local/bin/vfio-teardown script.

Export GPU ROM

This step is optional, try it if the VM is not starting or shutting down properly.

If you have Windows installed as dual boot

Extract it from Windows using GPU-Z, rename it to gpu.rom and copy that file to /var/lib/libvirt/vbios/.
If you don't have Windows installed as dual boot

Go to VGA bios collection by TechPowerUp and find your GPU. This should show all available VBIOS versions. If there are multiple VBIOS versions available, run cat /sys/class/drm/card*/device/vbios_version and check which VBIOS matches yours by going into details for all available VBIOS. The VBIOS version matching yours should be inside the BIOS Internals section. Download it, rename it to gpu.rom and copy that file to /var/lib/libvirt/vbios/.

In case you can't find it in the VGA bios collection, you can try these steps:
1. Find your GPU's device ID: lspci -vnn | grep '\[03'. You should see some output such as the following; the first bit (09:00.0 in this case) is the device ID.
```
 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Fiji [Radeon R9 FURY / NANO Series] [1002:7300] (rev cb)
```
1. Run find /sys/devices -name rom and ensure the device ID matches. For example looking at the case above, you'll want the last part before the /rom to be 09:00.0, so you might see something like this (the extra 0000: in front is fine):
```
 /sys/devices/pci0000:00/0000:00:03.1/0000:09:00.0/rom
```
1. For convenience's sake, let's call this PATH_TO_ROM. You can manually set this variable as well, by first becoming root (run sudo su) then running export PATH_TO_ROM=/sys/devices/pci0000:00/0000:00:03.1/0000:09:00.0/rom
2. Then, still as root, run the following commands:
```
 echo 1 > $PATH_TO_ROM
 mkdir -p /var/lib/libvirt/vbios/
 cat $PATH_TO_ROM > /var/lib/libvirt/vbios/gpu.rom
 echo 0 > $PATH_TO_ROM
```
1. Run exit to stop acting as root

Passthrough (virt-manager)

Open the virt-manager and remove DisplaySpice, VideoQXL and other serial devices from XML file. Also delete all spice related things and usb redirections.

 <!-- Remove Display Spice -->
 <graphics type="spice" port="-1" tlsPort="-1" autoport="yes">
   <image compression="off"/>
 </graphics>
 
 <!-- Remove USB Redirection -->
 <redirdev bus="usb" type="spicevmc"/>
 <redirdev bus="usb" type="spicevmc"/>
 
 <!-- Remove Video QXL -->
 <video>
   <model type="qxl"/>
 </video>
 
 <!-- Remove Tablet -->
 <input type="tablet" bus="usb"/>
 
 <!-- Remove console -->
 <console type="pty"/>

Add GPU and HDMI/DP Audio PCI Host devices. Make sure to add all of them from the same bus as the GPU
Add USB Host devices, like keyboard, mouse...
For sound: You can pass through the PCI HD Audio controller. BUT, be carefull. Ryzen 3000 and above apparently have problems when passing HD Audio Controller and USB controller that's on the same PCI bus as audio controller. For more info, look at this. If you are having issues with audio controller passthrough, you can set your audio up differently. Use this for PulseAudio or this for PipeWire.

If Virtual Network Interface is not present (NIC :xx:xx:xx), add it through Add hardware button

Network source: Virtual network 'default' : NAT
Device model: e1000e (could be different name for you, it was first option)

If you need to export your GPU ROM, don't forget to add gpu.rom file inside the win11.xml (or whatever your VM's name is) for the GPU and HDMI host PCI devices, example:
```
  ...
  </source>
  <rom file='/var/lib/libvirt/vbios/gpu.rom'/>  
  <address/>
  ...
```
Also, for 7900XTX GPU, make sure to disable Resize BAR in your BIOS and in the Virtual Manager (ROM BAR option on the PCI Device)

USB controllers passthrough

For a proper passthrough of USB controllers, make sure that you only select controllers that are in isolated IOMMU groups (there are no other controllers/devices in the same group, PCI bridge devices in the same group don't count as devices/controllers). To check your IOMMU groups, follow this guide. If you want to make sure that nothing important is attached to the controllers that you want to passthrough (BIOS LED controller or something similar), use the addresses from the IOMMU groups to check if there are some USB devices attached to those addresses by running sudo dmesg | grep -i usb

Integrated BT adapter passthrough

The best way to pass through a BT adapter is to add the USB Controller, which has the BT adapter attached to it, to the VM as a PCI device. But sometimes that is not possible because the USB controller is in the IOMMU group that is not viable for a passthrough. In that case, for a proper passthrough of an integrated BT adapter (motherboards with integrated WiFi and BT), you can add the BT adapter as an usb device with some additional tweaks to the VM XML.

Add additional qemu capabilities after </devices>, but before </domain>

  </devices>
  <qemu:capabilities>
    <qemu:del capability='usb-host.hostdevice'/>
  </qemu:capabilities>
</domain>

Just to be sure, edit startup and teardown hooks to unload and reload BT
- Edit /usr/local/bin/vfio-startup and add this before ## Load VFIO-PCI driver ##
```
systemctl disable --now bluetooth
modeprobe -r btusb
```
- Edit /usr/local/bin/vfio-teardown and add this before echo "$DATE End of Teardown!"
```
modeprobe btusb
systemctl enable --now bluetooth
```

Final checks

You might need to start the default network manually:

sudo virsh net-start default
sudo virsh net-autostart default

Don't forget to edit /etc/libvirt/qemu.conf:
```
user = "yourusername"
group = "kvm"
```

/etc/libvirt/libvirtd.conf should be set up like this:

unix_sock_group = "libvirt"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
log_filters = "2:libvirt.domain 1:qemu"
log_outputs = "1:file:/var/log/libvirt/libvirtd.log"

Check if CPU and RAM configurations are properly set

Internet not available in the VM

If you are having problems with the internet access inside the VM where Windows says something like "Unidentified network" and "No internet", check this link.

Improving VM and CPU performance

CPU pinning

It is a general recommendation to leave core 0 from all CCXs to the host.
Since I have a 5900x with 12c/24t, I will be passing 10c/20t with a setup of 5c/10t from the same CCX to the VM, so it will be two CCXs with 5c/10t. The rest will be pinned to the host. If you have a different CPU, this config will not apply to you, but you can check for a more detailed information on how to set this up here and here. If using lstopo and you have PU# and P# for threads, look at the P# value for the thread id.

You can also use virsh capabilities and look for a <cache> part, this will tell you how your cores/threads are separated per L3 cache. It should look something like this:

 <cache>
   <bank id='0' level='3' type='both' size='32' unit='MiB' cpus='0-5,12-17'/>
   <bank id='1' level='3' type='both' size='32' unit='MiB' cpus='6-11,18-23'/>
 </cache>

Try to match the L3 cache core assignments by adding fake cores that won't be enabled. Take a look at my code bellow and pay attention to vcpus with enabled="no". Those are fake cores that will be disabled, but are present so the assignment of cores per L3 cache is correct. For this, you will need to use Coreinfo inside the VM and figure out how many fake cores do you need and where do you need to put them.

The code for my setup

 <vcpu placement="static" current="20">26</vcpu>
 <vcpus>
 	<vcpu id="0" enabled="yes" hotpluggable="no"/>
 	<vcpu id="1" enabled="yes" hotpluggable="no"/>
 	<vcpu id="2" enabled="yes" hotpluggable="no"/>
 	<vcpu id="3" enabled="yes" hotpluggable="no"/>
 	<vcpu id="4" enabled="yes" hotpluggable="no"/>
 	<vcpu id="5" enabled="yes" hotpluggable="no"/>
 	<vcpu id="6" enabled="yes" hotpluggable="no"/>
 	<vcpu id="7" enabled="yes" hotpluggable="no"/>
 	<vcpu id="8" enabled="yes" hotpluggable="no"/>
 	<vcpu id="9" enabled="yes" hotpluggable="no"/>
 	<vcpu id="10" enabled="no" hotpluggable="yes"/>
 	<vcpu id="11" enabled="no" hotpluggable="yes"/>
 	<vcpu id="12" enabled="no" hotpluggable="yes"/>
 	<vcpu id="13" enabled="no" hotpluggable="yes"/>
 	<vcpu id="14" enabled="no" hotpluggable="yes"/>
 	<vcpu id="15" enabled="no" hotpluggable="yes"/>
 	<vcpu id="16" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="17" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="18" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="19" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="20" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="21" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="22" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="23" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="24" enabled="yes" hotpluggable="yes"/>
 	<vcpu id="25" enabled="yes" hotpluggable="yes"/>
 </vcpus>
 <cputune>
 	<vcpupin vcpu="0" cpuset="1"/>
 	<vcpupin vcpu="1" cpuset="13"/>
 	<vcpupin vcpu="2" cpuset="2"/>
 	<vcpupin vcpu="3" cpuset="14"/>
 	<vcpupin vcpu="4" cpuset="3"/>
 	<vcpupin vcpu="5" cpuset="15"/>
 	<vcpupin vcpu="6" cpuset="4"/>
 	<vcpupin vcpu="7" cpuset="16"/>
 	<vcpupin vcpu="8" cpuset="5"/>
 	<vcpupin vcpu="9" cpuset="17"/>
 	<vcpupin vcpu="16" cpuset="7"/>
 	<vcpupin vcpu="17" cpuset="19"/>
 	<vcpupin vcpu="18" cpuset="8"/>
 	<vcpupin vcpu="19" cpuset="20"/>
 	<vcpupin vcpu="20" cpuset="9"/>
 	<vcpupin vcpu="21" cpuset="21"/>
 	<vcpupin vcpu="22" cpuset="10"/>
 	<vcpupin vcpu="23" cpuset="22"/>
 	<vcpupin vcpu="24" cpuset="11"/>
 	<vcpupin vcpu="25" cpuset="23"/>
 	<emulatorpin cpuset="0,6,12,18"/>
 </cputune>

Make sure to update the <cpu> topology to match the number of cores and threads you are passing to the VM. For my setup, it looks like this:

 <cpu mode='host-passthrough' check='none' migratable='on'>  <!-- Set the cpu mode to passthrough -->
 	<!-- 13c/26t because 3c/6t are disabled and 10c/20t are used to match the proper L3 cache placement -->
 	<topology sockets='1' dies='1' cores='13' threads='2'/>
 	<cache mode='passthrough'/>                     <!-- The real CPU cache data reported by the host CPU will be passed through to the virtual CPU -->
 	<feature policy='require' name='topoext'/>  <!-- Required for the AMD CPUs -->
 	<feature policy='require' name='svm'/>
 	<feature policy='require' name='apic'/>         <!-- Enable various features improving behavior of guests running Microsoft Windows -->
 	<feature policy='disable' name='hypervisor'/>
 	<feature policy='require' name='invtsc'/>
 </cpu>

CPU Governor

This tweak takes advantage of the CPU frequency scaling governor.

My CPU uses powersave as the default governor. Please check which is the default for your CPU by running the following command in the terminal:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Update the cpu_mode_release.sh located in hooks/win11/release/end and replace powersave with your default governor. Also rename the win11 folder to the the name of your VM and then run:

sudo ./setup_cpu_governor_hooks.sh

The file tree should look similar to this now:

$ tree /etc/libvirt/hooks/
/etc/libvirt/hooks/
├── qemu
└── {name of your VM}
	├── prepare
	│   └── begin
	│       ├── ...
	│       └── cpu_mode_performance.sh
	└── release
		└── end
			├── ...
			└── cpu_mode_release.sh

CPU passthrough mode

This might not work as expected, as for me the latency improved but the L3 cache size changed from 32MB to 16MB, so I ended up not using this.
To check the cache latency, use AIDA64 Memory & Cache benchmarks (Tools->Memory & Cache benchmarks).
Also check your cache topology and size inside the VM using Coreinfo
This does not necessarily improve your performance, so please benchmark before and after to see which is better.
We can improve cache latency by changing from <cpu mode="host-passthrough"> to a custom mode that better matches your CPU.
1. To get a detailed info about your CPU, run virsh capabilities inside your terminal, look for <arch>x86_64</arch> and under that arch look for <model>. This is the model we are going to use inside our VM setup.
2. Go to VM settings, CPU, uncheck the Copy host CPU configuration and select the model you got from the previous step in the drop down menu.
3. You will have to remove the <cache mode="passthrough"/> option from the <cpu> inside your XML for this to work. You can try <cache level="3" mode="emulate"/> and see if that improves the performance over having no cache option. For me, it didn't make a difference in latency benchmarks so I removed it, but that might not be the case for you, so benchmark it.

Line-Based vs. Message Signaled-Based Interrupts (MSI)

This can sometimes help with audio stutters and cracks.
TL/DR: With this you can switch from Line-Based to MSI for improved interrupts handling which should improve audio stutters and cracks and some potential VM crashes related to interrupts.
Take a look at this detailed guide. I used MSI Utility V3 from the link in the post to switch to MSI

Internet improvements

Follow this link to possibly improve internet performance.
TL/DR: Set the network device as in the following picture. You will need the NetKVM driver for the ethernet controller inside the VM found in virtio-win.iso file.

EAC games detecting VM

Some games that use EAC might detect that Windows is running inside a VM. To try and fix that, enter the following line inside the <os> section in your XML config file for the VM:

<smbios mode="host"/>

Make sure you have dmidecode installed.

If that doesn't work, change <smbios mode="host"/> to <smbios mode="sysinfo"/> and add the following before the <os> tag:

<sysinfo type="smbios">
        <bios>
            <entry name="vendor">American Megatrends Inc.</entry>
            <entry name="version">3.50</entry>
            <entry name="date">09/18/2025</entry>
        </bios>
        <system>
            <entry name="manufacturer">ASRock</entry>
            <entry name="product">X870E Taichi</entry>
            <entry name="version">Default string</entry>
            <entry name="serial">Default string</entry>
            <entry name="uuid">43d5b292-a525-4a67-9cf0-a97f210de3f8</entry>
            <entry name="family">X870E</entry>
        </system>
    </sysinfo>

Make sure that <entry name="uuid"> value matches the uuid value of the VM, found inside the <uuid> tag at the start of the XML file. The rest should not matter, but you can match it with the info from running sudo dmidecode in the terminal.

Logging

Check all hook logs with sudo cat /dev/kmsg | grep libvirt-qemu
Check all libvirt logs in /var/log/libvirt/libvirtd.log file
Check all qemu logs in /var/log/libvirt/qemu/ directory

Credits

BigAnteater for easy guide and scripts for setting up GRUB, libvirt and qemu: https://github.com/BigAnteater/KVM-GPU-Passthrough
RisingPrismTV for amazing hooks scripts: https://gitlab.com/risingprismtv/single-gpu-passthrough
Zile995 for the really detailed guide: https://github.com/Zile995/PinnacleRidge-Polaris-GPU-Passthrough
Bryansteiner for details on CPU pinning and governor: https://github.com/bryansteiner/gpu-passthrough-tutorial

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

AMD Single GPU Passthrough

Table of contents

Preparations

Preparing boot

Configuring Libvirt

Setting up Windows VM

Hook Scripts

Editing hooks

GPU not resetting properly

Export GPU ROM

Passthrough (virt-manager)

USB controllers passthrough

Integrated BT adapter passthrough

Final checks

Internet not available in the VM

Improving VM and CPU performance

CPU pinning

CPU Governor

CPU passthrough mode

Line-Based vs. Message Signaled-Based Interrupts (MSI)

Internet improvements

EAC games detecting VM

Logging

Credits

About

Uh oh!

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 110 Commits
hooks/win11		hooks/win11
images		images
7900XTX-9950X3D-win10.xml		7900XTX-9950X3D-win10.xml
README.md		README.md
grub_setup_amd.sh		grub_setup_amd.sh
grub_setup_intel.sh		grub_setup_intel.sh
libvirt_configuration.sh		libvirt_configuration.sh
libvirtd.conf		libvirtd.conf
qemu.conf		qemu.conf
qemu_cpu_governor_hooks		qemu_cpu_governor_hooks
setup_cpu_governor_hooks.sh		setup_cpu_governor_hooks.sh
win10-7900xtx.xml		win10-7900xtx.xml
win11.xml		win11.xml

stele95/AMD-Single-GPU-Passthrough

Folders and files

Latest commit

History

Repository files navigation

AMD Single GPU Passthrough

Table of contents

Preparations

Preparing boot

Configuring Libvirt

Setting up Windows VM

Hook Scripts

Editing hooks

GPU not resetting properly

Export GPU ROM

Passthrough (virt-manager)

USB controllers passthrough

Integrated BT adapter passthrough

Final checks

Internet not available in the VM

Improving VM and CPU performance

CPU pinning

CPU Governor

CPU passthrough mode

Line-Based vs. Message Signaled-Based Interrupts (MSI)

Internet improvements

EAC games detecting VM

Logging

Credits

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages