Update September 14, 2021: This is a complete revamp, adding new, more robust methods and dropping outdated ones.
Update November 17, 2022: Kernel 6 seems to break the grub method.
When running a VM with GPU passthrough, that GPU should be bound to the VFIO driver. To make this happen, we need to prevent the regular graphics driver from binding to the passthrough GPU and instead bind the vfio-pci driver.
In the past we used to blacklist the graphics driver. This worked in most cases, but what if you need the graphics driver for another GPU, e.g. the host GPU?
Today there are much better ways to make the GPU use the vfio-pci driver. The most robust and versatile option is the “driver override feature” offered by the kernel. But recent changes in the kernel and the use of the systemd service have let to the development of a dedicated driver control utility called driverctl that could to be the ticket to success.
Note: This tutorial uses the nano editor together with sudo for editing files as root. If you prefer, you can use the xed editor found in Linux Mint Mate like this:
xed admin:///path/to/file
Table of Contents
Determine the PCI IDs of the passthrough devices
First we need to find the PCI ID(s) of the graphics card and perhaps other devices we want to pass through to the VM. To determine the PCI bus number and PCI IDs, enter:
lspci | grep VGA
Here is the output on my system:
01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation Device 13c2 (rev a1)
The first card under 01:00.0 is the Quadro 2000 I use for the Linux host. The other card under 02:00.0 I want to pass through to the guest.
Let’s find its PCI IDs:
lspci -nn | grep 02:00.
Substitute “02:00.” with the bus number of the graphics card you wish to pass through, without the trailing “0”. This way we get the PCI IDs for both the graphics device and the audio device residing on the graphics card. Here is the output on my computer:
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)
Write down the bus numbers (in our example 02:00.0 and 02:00.1), as well as the PCI IDs (10de:13c2 and 10de:0fbb). We need them both in the next part.
Below I describe 4 methods to bind the GPU to the vfio-pci driver. My preferred methods are option 4 (grub) and option 3 “Using the “driver_override” feature”.
Using the driverctl utility
driverctl is a relatively new Linux device driver control utility that was developed for recent Fedora or RHEL 7 distributions, but is also available under Linux Mint 20 (and most likely all new Ubuntu 20.04 based distros). This utility integrates with udev and makes driver assignment a breeze.
To install, use:
sudo apt install driverctl
A description of this utility (as well as the sources) can be found here:
https://gitlab.com/driverctl/driverctl
Using the PCI bus numbers we determined above, enter the following commands in a root shell to bind the graphics card to the vfio-pci driver:
driverctl set-override 0000:02:00.0 vfio-pci
driverctl set-override 0000:02:00.1 vfio-pci
To unbind, use the following command:
driverctl unset-override 0000:02:00.0
driverctl unset-override 0000:02:00.1
Please refer to the link above and the examples given in the README for more information.
Using a “module alias”
This method may be somewhat outdated. I keep it here for reference.
Run the following command:
cat /sys/bus/pci/devices/0000:02:00.0/modalias
where “02:00.0” is the PCI bus number of your graphics card, preceded by “0000:”. The output will look something like:
pci:v000010DEd000013C2sv00001458sd00003679bc03sc00i00
Repeat above command with the PCI bus number of the audio part:
cat /sys/bus/pci/devices/0000:02:00.1/modalias
where 0000:02:00.1 is the PCI bus number of your graphics card’ audio device.
Open or create /etc/modprobe.d/local.conf:
sudo nano /etc/modprobe.d/local.conf
and copy and paste the results from the two cat /sys/… commands above. Then precede the lines with “alias” and append the lines with “vfio-pci”, as shown below:
alias pci:v000010DEd000013C2sv00001458sd00003679bc03sc00i00 vfio-pci
alias pci:v000010DEd00000FBBsv00001458sd00003679bc04sc03i00 vfio-pci
At the end of that file, add the following line:
options vfio-pci ids=10de:13c2,10de:0fbb
where 10de:13c2 and 10de:0fbb are the PCI IDs for your graphics card’ VGA and audio part.
When using a 4.1 or newer kernel and boot the guest using UEFI (OVMF), you can also add the following option below the options vfio-pci entry:
options vfio-pci disable_vga=1
The above entry helps prevent VGA arbitration from interfering with host devices.
Save the file and exit the editor.
Update the initramfs by running:
sudo update-initramfs -u
Reboot your PC.
Using the “driver_override” feature
Note: Newer distribution releases (for example Linux Mint 20, Ubuntu 20.04, etc.) and kernels like kernel 5.11 may not work with the driver_override method described below. In that case you can use the driverctl device driver control utility described above.
Note 2: As of kernel 6, this method may work again. See “Linux Kernel 6 seems to be incompatible with the vfio_pci module needed for PCI passthrough” for more.
If you have identical graphics cards for both host and passthrough guest, use the driver_override feature as described below.
Create the driver-override script file:
sudo nano /sbin/vfio-pci-override-vga.sh
and copy/paste the following script:
#!/bin/sh
DEVS="0000:02:00.0 0000:02:00.1"
for DEV in $DEVS; do
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci
where “0000:02:00.0 0000:02:00.1” are the PCI bus numbers (both graphics and audio part) of the graphics card you wish to pass through.
Make the script executable:
sudo chmod 755 /sbin/vfio-pci-override-vga.sh
Open or create /etc/modprobe.d/local.conf:
sudo nano /etc/modprobe.d/local.conf
and paste the following line:
install vfio-pci /sbin/vfio-pci-override-vga.sh
Update the initramfs by running:
sudo update-initramfs -u
Reboot your PC.
For more information, see VFIO GPU How To series, part 3 – Host configuration and how to get two identical GPUs sorted for KVM using vfio-pci. All credit goes to Alex Williamson and Linux Mint forum member O_Shovah!
Binding to vfio-pci using the grub bootloader
An alternative to the above methods is binding the graphics driver to the vfio-pci driver in the /etc/default/grub file. This is the easiest and perhaps the most common method, but it has a few downsides:
- It doesn’t work if you have two identical graphics cards for host and VM.
- The graphics card will be bound to the vfio-pci driver when the host boots, making it inaccessible to the host (that should normally be no issue).
- Kernel 6 seems to break this method, as of now.
I use this method on my Manjaro host, but it works equally well on Ubuntu and derivatives, Fedora, and most other Linux distributions. A variation of this method can be used with kernelstub on Pop!_OS, see here.
Edit the /etc/default/grub file:
sudo nano /etc/default/grub
The entry we are looking for is:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
We are using the PCI vendor ID and model ID given in square brackets like [10de:13c2] that we wrote down in the first step “Determine the PCI IDs of the passthrough devices“. 10de
is the vendor ID for Nvidia, the 13c2
identifies a specific model, in this case a GTX 970 GPU.
Note that some modern GPUs have more than two devices, for example USB and serial in addition to graphics and audio. Make sure to list all of them.
With this information in hand, we can update the bootloader.
Important: Windows 10 release 1803 or newer require the kvm.ignore_msrs=1
option. However, there are reports that starting with kernel 5.9.1 this option causes issues and should be removed. I have used the kvm.ignore_msrs=1
option with kernel 5.10, 5.11, 5.12, and now 5.14 with no issues on my Manjaro PC. My recommendation: Start with the kvm.ignore_msrs option and remove it if it causes issues.
Here is what we add at the end of GRUB_CMDLINE_LINUX_DEFAULT, just before the closing “:
vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1
Make sure to replace the ids with the ones you determined! Here is how the entire line may look like:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1"
Note: The above is one single line without line break!
Save the modified file, then run:
sudo update-grub
Reboot and check if it worked:
lspci -kn | grep -A 2 02:00.
where 02:00. is the PCI bus ID you wrote down in paragraph “Determine the PCI IDs of the passthrough devices“.
Go through the output and look for the graphics card you wish to pass through. The output should be similar to this:
02:00.0 0300: 10de:13c2 (rev a1)
Subsystem: 1458:3679
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 0403: 10de:0fbb (rev a1)
Subsystem: 1458:3679
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
Notice the lines “Kernel driver in use: vfio-pci”. If it shows a different driver, something went wrong (be sure to check the right graphics card!).
Note: My Nvidia RTX 2700 Super has 4 sub-devices. Three of them – graphics, audio, and serial – bind to the vfio-pci driver. The USB driver doesn’t, which works fine.
If this post has been helpful, click the “Like” button below. Don’t forget to share this page with your friends.
The easiest way to prevent the graphics driver from binding to the passthrough GPU is to specify GPU vendor ID:model ID in the /etc/default/grub file:
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash amd_iommu=on amd_iommu=pt hugepages=8192 vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1″
Just add the vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1 to the end of the GRUB_CMDLINE_LINUX_DEFAULT= line, before the quotation marks (“). You should also specify the boot option kvm.ignore_msrs=1. Replace the vendor:model IDs with the ones for your GPU.
After editing the /etc/default/grub file, run:
sudo update-grub
There is a new Linux device driver control utility called driverctl that is available in the latest Linux Mint/Ubuntu and Fedora/Redhat repositories. The driverctl script integrates with udev and systemd service and can change bindings to survive reboots.
For more information see https://gitlab.com/driverctl/driverctl
Note that you can install the package via package manager in newer Ubuntu etc. releases.
Found this just in time. Confirming that driverctl is available for Debian 11 Bullseye as well and I’ve used it to swap my GPU between host and vfio-pci and back successfully.
Thanks 🙂
For openSUSE Tumbleweed none of the above works – since the naming / method is a bit different. Found this SLES article which when following does the job: https://doc.opensuse.org/documentation/leap/virtualization/html/book-virtualization/app-gpu-passthru.html
Thanks for the link! I’m sure this will be helpful to others.
This post helped me to create my own solution which allows using vfio-pci on kernel 6.X with only few tweaks. https://hardcoded.info/post/2023/03/04/pci-passthrough-using-vfio-pci-for-linux-kernel-version-6-solution/