Last update: September 14, 2021

Starting with the GeForce 1000 series, vfio passthrough of a Nvidia GPU has become a little more complicated. If, when starting the VM, you get a black screen, chances are you need to pass along a VBIOS file to the VM so the GPU can properly initialize.

This post is about passing through a Nvidia RTX 2070 Super GPU or any other modern Nvidia GPU to a Windows 10 guest.

Glossary

Here is a list of the most common terms in this tutorial that might need explanation:

Device: In the context of this tutorial, device usually refers to a PCI hardware device such as a graphics card.

GPU passthrough – synonymous with vfio, VGA passthrough, or simply passthrough: a framework or technology that allows the virtual machine to communicate with the “passthrough GPU” using native drivers.

Guest: Synonym for “virtual machine”.

Host: The computer hardware and operating system that hosts our virtual machines. In our case we use Linux as the host operating system.

IOMMU: The input-output management unit that controls and isolates devices to allow for example virtual machines to have direct access to hardware devices. Hardware support for IOMMU is quintessential for GPU passthrough to work. For more details see IOMMU Groups – What You Need to Consider.

kvm/QEMU: kvm or “kernel virtual machine” is a Linux technology that allows the creation of virtual machines that make efficient use of host resources. QEMU or “Quick Emulator” is a virtualizer and emulator that can emulate hardware. QEMU and kvm are used together as a versatile and powerful framework for creating virtual machines.

Native driver: A driver written specifically for the operating system. If we run Windows 10 in a virtual machine, a native driver for the Nvidia GPU would be the Windows driver available on the Nvidia website.

Passthrough GPU: The GPU that is being passed through to the virtual machine. The passthrough GPU is controlled by the virtual machine using native drivers.

Sub-device: Multifunction devices such as graphics cards feature more than one “device”. For example, practically all modern graphics cards have – in addition to their graphics device – an audio device located on their circuit board. The correct term for these is “function” (that’s where the name “multifunction device” derives from). I call them sub-devices.

VBIOS: The video BIOS that is contained on the graphics card inside non-volatile memory (usually an EPROM). The instructions in that VBIOS initialize the graphics card and allow it to display text and graphics on a VGA screen. It is used for example to display the BIOS screen at boot. In this tutorial we use a truncated copy of that VBIOS to initialize the passthrough GPU when starting the virtual machine.

VFIO: The framework, technology and driver that are employed to allow virtual machines to have access to the PC hardware in a save way.

Virtual machine: Abbreviated “VM“. A complete operating system, drivers and application software running isolated on top of the host operating system. For example Microsoft Windows 10 as a VM on a host that uses Linux. Both Windows and Linux are running concurrently on the same hardware.

System Specifications

This tutorial should work for Debian and Ubuntu-based distributions, as well as for Manjaro / Arch Linux ( it might be necessary to adapt some commands).

The tutorial is specifically written for Nvidia graphics cards, though it doesn’t have to be a RTX 2070. The instructions should likewise work with any recent series 1000, series 2000, or series 3000 Nvidia GPU.

Hardware

Here is my hardware:

• AMD Ryzen 3900X CPU
• Gigabyte Aorus X570 Pro motherboard
• 1st PCIe slot: Gigabyte RTX 2070 Super Windforce OC 3x 8G GPU (guest)
• 3rd PCIe slot (2nd long slot): PNY Quadro 2000 GPU (host)
• Asus Xonar STX sound card (host)
• 64GB RAM
• 2x Samsung NVMe plus an assortment of HDD and SSD

Software

Here are my software specs at the time of writing, for reference:

Host:

• Manjaro Linux
• Kernel 5.8.18-1-MANJARO
• QEMU emulator version 5.1.0

Guest:

• Windows 10 Professional release 20H2
• Nvidia studio driver release 460.89
• Virtio drivers virtio-win-0.1.173.iso

VFIO Graphics Passthrough

I already had a working VFIO graphics passthrough using the Nvidia GTX 970. In fact, passing through the older Nvidia GPUs is a breeze. No modified VBIOS needed, just declaring the PCI bus IDs.

Modern Nvidia graphics cards are often reported to black-screen after passthrough. This can happen when the passthrough graphics card gets initialized before it is bound to the vfio-pci driver. A solution to this problem is to pass along the VBIOS of the graphics card. So lets go step by step.

Initial configuration

I have covered the initial setup in two different tutorials: Creating a Windows 10 VM on the AMD Ryzen 9 3900X using Qemu 4.0 and VGA Passthrough and Running Windows 10 on Linux using KVM with VGA Passthrough.

In summary the initial configuration steps are:

1. Enable virtualization, in particular IOMMU / VTd, using the motherboard BIOS screen.
2. Install the required packages and download the Virtio drivers ISO.

The above steps are described in detail in the above tutorials. If you own a modern AMD Ryzen CPU, follow this link.

Continue with the next step after you enabled IOMMU and downloaded/installed the required packages.

Obtain the correct VBIOS

Unfortunately neither Nvidia nor the graphics card vendor (in my case Gigabyte) offer downloadable VBIOS images. Lucky for us there are other ways to obtain the VBIOS. I’ll describe both of them.

If you have a working Windows installation that uses the graphics card you wish to pass through:

1. Inside Windows, download the GPU-Z utility from TechPowerUp.
   
2. Open GPU-Z.

3. To save the graphic card VBIOS, click the grey right arrow underneath the NVIDIA logo. Then select “Save to file…” to save the VBIOS. (Note: The screenshot above was taken inside my Windows VM, which prevented me from saving the VBIOS.)

---

As an alternative, if you are using Linux, you can download a VBIOS from the TechPowerUp VBIOS database:

1. Open a terminal window and enter:

lspci -knn | grep -i -A 2 vga

Look for the graphics card you wish to pass through in the output. My NVIDIA 2070 Super GPU has the subsystem device ID 1458:4001. The subsystem ID specifies the GPU manufacturer and model, which is what you want. Write down the subsystem ID (make sure to get the one from the graphics card you wish to pass through, NOT the one your host uses).

Note: Provided your host boots with the correct GPU, you can use the following command line script to get the info for your passthrough GPU:

for i in $(find /sys/devices/pci* -name boot_vga); do [[ $(cat $i) -eq 0 ]] && lspci -knn -s $(echo $i | rev | cut -d/ -f2 | rev); done

2. To get the VBIOS version on your GPU, enter:

cat /proc/driver/nvidia/gpus/*/information

and look for the information pertaining to your passthrough GPU, in this example the RTX 2070 Super:

Video BIOS: 90.04.86.00.D0

Write down the Video BIOS version.

3. Open your browser and write / paste the following into your DuckDuckGo (or Google) search window:

techpowerup 1458:4001

Replace the numbers 1458:4001 with the subsystem ID you determined. Follow the link to the TechPowerUp VBIOS database site. You will get a list similar to the one below:

4. There may be different VBIOS files to choose from. For a starter, it’s best to select the VBIOS version you wrote down in step 2.

You can click “Details” to learn more about the files and the corresponding graphics card. Make sure you get the right one that matches your GPU vendor, model, and revision.

Download the VBIOS.

Edit VBIOS file using a hex editor

In recent years NVIDIA has packed additional code into the VBIOS. This additional code prevents QEMU to properly load the VBIOS and initialize the graphics card.

We are going to use a hex editor to edit the VBIOS file and strip off the unnecessary code at the beginning of the file. Sounds difficult? It’s not.

1. Install a hexadecimal (hex) editor of your choice. Using Linux, I found “bless” to be a suitable choice.
   
2. Open the VBIOS file using the hex editor.

3. Enlarge the hex editor window. Click the search icon and search for “55AA” hexadecimal. If necessary, click the “Find Next” button until you see the word “VIDEO” appear in the right (text) window (see below).

This is the beginning of the actual VBIOS, starting with 55AA hexadezimal, some letters or symbols and then the word “VIDEO” and some more text.

Note: The hexadecimal sequence 55AA may appear one or more times before the actual beginning of the BIOS code. So always look for the word “VIDEO”!

4. Mark and delete everything BEFORE the part beginning with 55 AA. For faster editing, use the editor in full screen mode. The result should look like below.

The file now starts with the hexadecimal sequence 55AA.

5. Save the VBIOS file under a different name, for example:

2070S-VBIOS-edited.rom

Close the editor.

Bind passthrough GPU to VFIO driver

When booting the Linux host, one of the first things that get initialized is the graphics card. This allows us to see boot messages etc. (if we chose to). We must prevent the Nouveau or Nvidia graphics driver from initializing and binding to the passthrough GPU.

This is done by binding the GPU to the vfio-pci driver which then allows the guest to use its own drivers for the GPU. (For the curious, you can find technical details on the kernel.org website.)

Binding the passthrough GPU to the VFIO driver involves several steps:

1. List the GPUs in your system:

lspci | grep VGA

[heiko@woody ~]$ lspci | grep VGA
0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)
0c:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)

I selected the Nvidia GeForce RTX 2070 Super for my Windows VM, which has the PCI bus ID 0b:00.0. The .0 in the bus ID denotes the sub-device called “function” in PCI slang. Most likely the GPU has several sub-devices such as graphics, audio, USB, etc. A GPU is therefore a “multifunction” device (we’ll get to that later).

Note on syntax: PCI devices on a PCI(e) bus are identified using the BDF notation, which stands for Bus, Device, and Function. In our example – 0b:00.0 – “0b” is the bus, “00” the device, and “0” the function. Often a 4-digit domain is added in front of the BDF ID, such as 0000:0b:00.0, where 0000 specifies the domain.

2. To determine all sub-devices, substitute the “VGA” in the previous command with the PCI bus ID, but remove the trailing “0“:

lspci -nn | grep 0b:00.

[heiko@woody ~]$ lspci -nn | grep 0b:00.
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1)
0b:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
0b:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)

Make sure to replace the PCI bus ID with the one you determined!

My Nvidia RTX 2070 Super has 4 sub-devices in total:

• 0b:00.0 – graphics
• 0b:00.1 – audio
• 0b:00.2 – USB controller
• 0b:00.3 – serial bus controller

All of them must be passed through to the Windows VM.

3. Next we check the IOMMU grouping. For VFIO passthrough to work, the configuration needs to fulfill the following two requirements:

• All PCI (sub-) devices inside a specific IOMMU group must be passed to the VM (exception: PCI root ports/controllers);
• All PCI sub-devices of the passthrough GPU must be passed to the VM.

Sounds more complicated than it actually is.

Let’s get a sorted list of all IOMMU groups and their PCI devices:

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort

[heiko@woody ~]$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
...
/sys/kernel/iommu_groups/25/devices/0000:07:00.0
/sys/kernel/iommu_groups/26/devices/0000:0b:00.0
/sys/kernel/iommu_groups/26/devices/0000:0b:00.1
/sys/kernel/iommu_groups/26/devices/0000:0b:00.2
/sys/kernel/iommu_groups/26/devices/0000:0b:00.3
/sys/kernel/iommu_groups/27/devices/0000:0c:00.0
/sys/kernel/iommu_groups/27/devices/0000:0c:00.1
...

As you can see, my Nvidia RTX GPU with PCI bus IDs 0b:00.0 to 0b:00.3 is within IOMMU group 26. There are no additional devices in that IOMMU group. Perfect!

4. Now you need to determine the method you’re going to use to bind the passthrough GPU to the vfio-pci driver. Typical with Linux, there are several ways to bind the GPU (see my post “Blacklisting Graphics Driver” for more options). Use the following to decide:

• You have two non-identical GPUs in your system: use the bootloader  (grub) method. This is perhaps the easiest and most robust method.
• You have two identical GPUs for host and guest: you must use the driver override method. Caveat: Newer distro releases such as Ubuntu 20.04, Linux Mint 20 but also the latest Fedora releases have somehow changed the procedure. In that case, use the driverctl utility.
• If you have only one GPU that you use for the host and wish to pass this GPU to your VM, then you are reading the wrong tutorial. Check Yuri Alek’s or Joe Knock’s tutorials. Don’t worry – what you’ve done so far is still valid. You may even want to read through this entire tutorial to get a better understanding of the process.

5. As a precaution, before you bind a GPU to the vfio-pci driver, you should change its Linux graphics driver to the open source version. Look for your driver management application and select the open-source Nouveau driver for the Nvidia card you wish to pass through.

On a Manjaro Linux system, open “Manjaro Settings Manager”, then double-click on “Hardware Configuration”:

On Linux Mint, open “Control Center”, then click “Driver Manager”:

Make sure to install and select the nouveau open-source driver for the GPU you wish to pass through.

After changing the driver, reboot to see everything still works.

Bind Graphics Card using the Bootloader

To bind the graphics card to the vfio-pci driver using the bootloader method, do the following:

1. Enter again the following command, replacing the 0b:00. with the PCI ID of your passthrough GPU:

lspci -nn | grep 0b:00.

[heiko@woody ~]$ lspci -nn | grep 0b:00.
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1)
0b:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
0b:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)

Notice the hexadecimal numbers (numbers and letters) inside square brackets.

2. Write down or better copy/paste the vendor and model IDs into a file. The vendor:model IDs for all the sub-devices of my RTX 2070S graphics card are:

10de:1e84 – graphics
10de:10f8 – audio
10de:1ad8 – USB
10de:1ad9 – serial bus

3. Open the /etc/default/grub file with root permissions, for example using:

sudo nano /etc/default/grub

4. Look for the line that starts with “GRUB_CMDLINE_LINUX_DEFAULT=“.

At the end of that line, before the double quote, append the options

kvm.ignore_msrs=1 vfio-pci.ids=

followed by your vendor:model IDs, separated by commas.

Here is an example:

kvm.ignore_msrs=1 vfio-pci.ids=10de:1e84,10de:10f8,10de:1ad8,10de:1ad9

The above has to be all in one line (don’t press Enter). The kvm.ignore_msrs=1 option is required for all newer releases of Windows 10. You can also add

kvm report_ignored_msrs=0

to prevent your logs clogging up with “ignored rdmsr” messages.

Important: It’s been reported that from kernel 5.9.1 on this option is either not required, or may even lead to issues loading the VM. See this VFIO thread on Reddit. On my Manjaro machine kvm.ignore_msrs=1 works with kernels 5.10, 5.11, and 5.12.

5. Make sure the options after the “GRUB_CMDLINE_LINUX_DEFAULT=” are enclosed in double quotes. Below is an example /etc/default/grub file for reference:

Save and close the file.

6. To update the grub boot loader, execute:

sudo update-grub

7. Reboot the PC.

8. Before we continue with creating or editing the VM configuration, let’s check that the graphics card has been bound to the vfio-pci driver:

lspci -kn | grep -A 2 0b:00.

Replace 0b:00. with the PCI device ID for your GPU, without function (.0).

[heiko@woody ~]$ lspci -kn | grep -A 2 0b:00.
0b:00.0 0300: 10de:1e84 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci
--
0b:00.1 0403: 10de:10f8 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci
--
0b:00.2 0c03: 10de:1ad8 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: xhci_hcd
--
0b:00.3 0c80: 10de:1ad9 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci

All but the USB devices use the vfio-pci kernel driver. Success!

Bind Graphics Card using the driver-override Method

If you chose the driver override method, please follow the instructions in my “Blacklisting Graphics Driver” tutorial under “Using  the “driver_override” feature”.

Create/Modify the XML VM configuration file

The process of creating a Windows 10 VM using the Virtual Machine Manager GUI is described in detail under “Create Windows 10 VM Configuration using the Virtual Machine Manager GUI” followed by “Additional XML Configurations“. If you are creating a new Windows VM, follow those instructions and return back here.

Booting the virtual machine using a VBIOS / ROM file requires us to modify the XML configuration we have created for the VM:

1. Open the Virtual Machine Manager GUI.

2. Select the PCI device ID of the GPU, in my case it is PCI 0000:0b:00.0:

Note the device description.

3. Select the XML tab. You must have XML editing enabled in the Virtual Machine Manager settings.

4. Insert the following line below the “</source>” tag:

<rom file="/home/heiko/.bin/Gigabyte_RTX2070Super_8192_191021_edit.rom"/>

Replace the path with the path to your .rom file!

5. Modify the “<address type=>” line so that it reads “multifunction='on'” at the end before the closing tag, like here:

<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0" multifunction="on"/>

Make sure to click “Apply” after you have finished with the changes!

Here some explanation: The Virtual Machine Manager isn’t too smart and cannot determine multifunction devices. As a result, it creates a separate device for each sub-device or function on the graphics card and puts each function in a separate “virtual” PCIe slot. The Windows guest then sees the graphics device in one slot, the audio device in another slot, the USB device in yet another slot and so on. As strange as this may sound, it usually works.

However, a Windows 10 or graphics driver update has the potential of breaking things. Moreover, if you run Windows 10 sometimes on bare metal using “dual-boot” and sometimes as a virtual machine inside kvm, you better make sure that the differences that Windows perceives are as small as possible.

This and the following steps create a virtual PCIe slot that contains the graphics card and its sub-devices as functions on that slot (ergo the name “multifunction”).

While we are at it, let’s decipher the syntax of the XML entry:

<hostdev mode="subsystem" type="pci" managed="yes"

The above line defines a subsystem of type PCI that is managed by Virtual Machine Manager / libvirt.

<source>
  <address domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
</source>

is the physical location of the hardware device in your PC. We have seen this before when listing the IOMMU groups, but in a different notation: 0000:0b:00.0

<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0" multifunction="on"/>

is the virtual PCI bus that Windows will see. In addition, this line tells QEMU that it’s a multifunction device. Write down the bus and slot numbers!

6. We have to modify the XML configuration for each sub-device/function on the graphics card. Select the PCI (sub-)device that’s next, PCI 0000:0b:00.1 in my case, and edit as follows:

<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x1"/>

Change the bus number (if different) and slot number to the ones you wrote down above. Then change the function from 0 to 1, like:

function="0x1"

Click “Apply“.

Note: All sub-devices (functions) of the GPU use the same domain, bus, and slot. The only number that should change is for the function, i.e. 0, 1, 2, …

7. Edit the next PCI sub-device, 0000:0b:00.2 in my example:

<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x2"/>

Make sure you got the same bus and slot as before, and increment the function=0x2.

Click “Apply“.

8. Repeat one more time (for all 4 functions of the GPU).

Click “Apply” when done!

This should complete the configuration for the Nvidia graphics card.

9. Before you boot the VM, make sure memballoon is turned off. (Unfortunately Virtual Machine Manager turns this option on by default.)

Click “Overview” on the top left Virtual Machine Manager VM configuration screen, then select the “XML” tab.

Scroll down to near the end of the XML configuration and look for the “memballoon” entry. Change that entry as follows:

<memballoon model="none"/>

Click “Apply“.

Boot the Windows VM

Now is the time to buckle up and boot your Windows VM. Start the Windows VM from within Virtual Machine Manager. If it’s a new installation, follow the Windows 10 instructions in my other tutorial.

If you already have a Windows 10 VM and upgraded your graphics card, Windows will search for a new driver and install it. You should go to the Nvidia driver download website to get the latest driver.

After each Nvidia driver installation or upgrade, Nvidia will revert to line based interrupts, which is less efficient and may cause audio issues. To fix that, see MSI based interrupts in my tuning section.

I hope this has been helpful. Let me know how it went by posting a comment. Likewise if you had issues.

Tuning and trouble-shooting

If you run into troubles, you can find a lot more information on this site – see my other two tutorials. Those tutorials also contain a long list of references, links, other tutorials and whatever I felt could be of help.

There is also a very active VFIO Reddit where you can ask for help or report issues.

Finally the best: Check out the comprehensive VFIO channel on Discord with concise, up-to-date instructions and helpful experts.

If this article has been helpful, click the “Like” button below. Don’t forget to share this page with your friends.