Update September 14, 2021: This is a complete revamp, adding new, more robust methods and dropping outdated ones.

Update November 17, 2022: Kernel 6 seems to break the grub method.

When running a VM with GPU passthrough, that GPU should be bound to the VFIO driver. To make this happen, we need to prevent the regular graphics driver from binding to the passthrough GPU and instead bind the vfio-pci driver.

In the past we used to blacklist the graphics driver. This worked in most cases, but what if you need the graphics driver for another GPU, e.g. the host GPU?

Today there are much better ways to make the GPU use the vfio-pci driver. The most robust and versatile option is the “driver override feature” offered by the kernel. But recent changes in the kernel and the use of the systemd service have let to the development of a dedicated driver control utility called driverctl that could to be the ticket to success.

Note: This tutorial uses the nano editor together with sudo for editing files as root. If you prefer, you can use the xed editor found in Linux Mint Mate like this:

xed admin:///path/to/file

Determine the PCI IDs of the passthrough devices

First we need to find the PCI ID(s) of the graphics card and perhaps other devices we want to pass through to the VM. To determine the PCI bus number and PCI IDs, enter:

lspci | grep VGA

Here is the output on my system:

01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation Device 13c2 (rev a1)

The first card under 01:00.0 is the Quadro 2000 I use for the Linux host. The other card under 02:00.0 I want to pass through to the guest.

Let’s find its PCI IDs:

lspci -nn | grep 02:00.

Substitute “02:00.” with the bus number of the graphics card you wish to pass through, without the trailing “0”. This way we get the PCI IDs for both the graphics device and the audio device residing on the graphics card. Here is the output on my computer:

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

Write down the bus numbers (in our example 02:00.0 and 02:00.1), as well as the PCI IDs (10de:13c2 and 10de:0fbb). We need them both in the next part.

Below I describe 4 methods to bind the GPU to the vfio-pci driver. My preferred methods are option 4 (grub) and option 3 “Using the “driver_override” feature”.

Using the driverctl utility

driverctl is a relatively new Linux device driver control utility that was developed for recent Fedora or RHEL 7 distributions, but is also available under Linux Mint 20 (and most likely all new Ubuntu 20.04 based distros). This utility integrates with udev and makes driver assignment a breeze.

To install, use:

sudo apt install driverctl

A description of this utility (as well as the sources) can be found here:

https://gitlab.com/driverctl/driverctl

Using the PCI bus numbers we determined above, enter the following commands in a root shell to bind the graphics card to the vfio-pci driver:

driverctl set-override 0000:02:00.0 vfio-pci
driverctl set-override 0000:02:00.1 vfio-pci

To unbind, use the following command:

driverctl unset-override 0000:02:00.0
driverctl unset-override 0000:02:00.1

Please refer to the link above and the examples given in the README for more information.

Using a “module alias”

This method may be somewhat outdated. I keep it here for reference.

Run the following command:

cat /sys/bus/pci/devices/0000:02:00.0/modalias

where “02:00.0” is the PCI bus number of your graphics card, preceded by “0000:”. The output will look something like:

pci:v000010DEd000013C2sv00001458sd00003679bc03sc00i00

Repeat above command with the PCI bus number of the audio part:

cat /sys/bus/pci/devices/0000:02:00.1/modalias

where 0000:02:00.1 is the PCI bus number of your graphics card’ audio device.

Open or create /etc/modprobe.d/local.conf:

sudo nano /etc/modprobe.d/local.conf

and copy and paste the results from the two cat /sys/… commands above. Then precede the lines with “alias” and append the lines with “vfio-pci”, as shown below:

alias pci:v000010DEd000013C2sv00001458sd00003679bc03sc00i00 vfio-pci
alias pci:v000010DEd00000FBBsv00001458sd00003679bc04sc03i00 vfio-pci

At the end of that file, add the following line:

options vfio-pci ids=10de:13c2,10de:0fbb

where 10de:13c2 and 10de:0fbb are the PCI IDs for your graphics card’ VGA and audio part.

When using a 4.1 or newer kernel and boot the guest using UEFI (OVMF), you can also add the following option below the options vfio-pci entry:

options vfio-pci disable_vga=1

The above entry helps prevent VGA arbitration from interfering with host devices.

Save the file and exit the editor.

Update the initramfs by running:

sudo update-initramfs -u

Reboot your PC.

Using  the “driver_override” feature

Note: Newer distribution releases (for example Linux Mint 20, Ubuntu 20.04, etc.) and kernels like kernel 5.11 may not work with the driver_override method described below. In that case you can use the driverctl device driver control utility described above.

Note 2: As of kernel 6, this method may work again. See “Linux Kernel 6 seems to be incompatible with the vfio_pci module needed for PCI passthrough” for more.

If you have identical graphics cards for both host and passthrough guest, use the driver_override feature as described below.

Create the driver-override script file:

sudo nano /sbin/vfio-pci-override-vga.sh

and copy/paste the following script:

#!/bin/sh
DEVS="0000:02:00.0 0000:02:00.1"
for DEV in $DEVS; do
   echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci

where “0000:02:00.0 0000:02:00.1” are the PCI bus numbers (both graphics and audio part) of the graphics card you wish to pass through.

Make the script executable:

sudo chmod 755 /sbin/vfio-pci-override-vga.sh

Open or create /etc/modprobe.d/local.conf:

sudo nano /etc/modprobe.d/local.conf

and paste the following line:

install vfio-pci /sbin/vfio-pci-override-vga.sh

Update the initramfs by running:

sudo update-initramfs -u

Reboot your PC.

For more information, see VFIO GPU How To series, part 3 – Host configuration and  how to get two identical GPUs sorted for KVM using vfio-pci. All credit goes to Alex Williamson and Linux Mint forum member O_Shovah!

Binding to vfio-pci using the grub bootloader

An alternative to the above methods is binding the graphics driver to the vfio-pci driver in the /etc/default/grub file. This is the easiest and perhaps the most common method, but it has a few downsides:

• It doesn’t work if you have two identical graphics cards for host and VM.
• The graphics card will be bound to the vfio-pci driver when the host boots, making it inaccessible to the host (that should normally be no issue).
• Kernel 6 seems to break this method, as of now.

I use this method on my Manjaro host, but it works equally well on Ubuntu and derivatives, Fedora, and most other Linux distributions. A variation of this method can be used with kernelstub on Pop!_OS, see here.

Edit the /etc/default/grub file:

sudo nano /etc/default/grub

The entry we are looking for is:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

We are using the PCI vendor ID and model ID given in square brackets like [10de:13c2] that we wrote down in the first step “Determine the PCI IDs of the passthrough devices“. 10de is the vendor ID for Nvidia, the 13c2 identifies a specific model, in this case a GTX 970 GPU.

Note that some modern GPUs have more than two devices, for example USB and serial in addition to graphics and audio. Make sure to list all of them.

With this information in hand, we can update the bootloader.

Important: Windows 10 release 1803 or newer require the kvm.ignore_msrs=1 option. However, there are reports that starting with kernel 5.9.1 this option causes issues and should be removed. I have used the kvm.ignore_msrs=1 option with kernel 5.10, 5.11, 5.12, and now 5.14 with no issues on my Manjaro PC. My recommendation: Start with the kvm.ignore_msrs option and remove it if it causes issues.

Here is what we add at the end of GRUB_CMDLINE_LINUX_DEFAULT, just before the closing “:

vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1

Make sure to replace the ids with the ones you determined! Here is how the entire line may look like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_pci.ids=10de:13c2,10de:0fbb kvm.ignore_msrs=1"

Note: The above is one single line without line break!

Save the modified file, then run:

sudo update-grub

Reboot and check if it worked:

lspci -kn | grep -A 2 02:00.

where 02:00. is the PCI bus ID you wrote down in paragraph “Determine the PCI IDs of the passthrough devices“.

Go through the output and look for the graphics card you wish to pass through. The output should be similar to this:

02:00.0 0300: 10de:13c2 (rev a1)
Subsystem: 1458:3679
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 0403: 10de:0fbb (rev a1)
Subsystem: 1458:3679
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

Notice the lines “Kernel driver in use: vfio-pci”. If it shows a different driver, something went wrong (be sure to check the right graphics card!).

Note: My Nvidia RTX 2700 Super has 4 sub-devices. Three of them – graphics, audio, and serial – bind to the vfio-pci driver. The USB driver doesn’t, which works fine.

If this post has been helpful, click the “Like” button below. Don’t forget to share this page with your friends.