Passing Through a Nvidia RTX 2070 Super GPU

Tutorial for passing through a Nvidia RTX graphics card to a Windows 10 virtual machine using a modified VBIOS

Recently I replaced my old NEC MultiSync 2690 HD screen for a HP DreamColor wide gamut screen with a 2560 x 1440 pixel resolution. The Nvidia GTX 970 GPU I used, though still perfectly capable at HD resolution, is likely to get challenged by higher resolutions. Moreover, the new Nvidia Studio driver doesn’t support models before the Geforce 1000 series, and I do want the 30 bpp (bit-per-pixel) support that this driver provides. Time for an upgrade – the Gigabyte RTX 2070 Super GPU.

This post is about passing through a Nvidia RTX 2070 Super GPU or any other modern Nvidia GPU to a Windows 10 guest.

System Specifications

This tutorial should work for Debian and Ubuntu-based distributions, as well as Manjaro / Arch Linux ( it might be necessary to adapt some commands).

The tutorial is specifically written for Nvidia graphics cards, though it doesn’t have to be a RTX 2070. The instructions should likewise work with any recent series 1000, series 2000, or series 3000 Nvidia GPU.

CPU specific instructions use the syntax for AMD CPUs, but they can easily be adapted to work on Intel systems. See my tutorial “Running Windows 10 on Linux using KVM with VGA Passthrough” for references.

Hardware

    • AMD Ryzen 3900X CPU
    • Gigabyte Aorus X570 Pro motherboard
    • 1st PCIe slot: Gigabyte RTX 2070 Super Windforce OC 3x 8G GPU (guest)
    • 3rd PCIe slot (2nd long slot): PNY Quadro 2000 GPU (host)
    • Asus Xonar STX sound card (host)
    • 64GB RAM
    • 2x Samsung NVMe plus an assortment of HDD and SSD

Software

Host:

    • Manjaro Linux
    • Kernel 5.8.18-1-MANJARO
    • QEMU emulator version 5.1.0

Guest:

    • Windows 10 Professional release 20H2
    • Nvidia studio driver release 460.89
    • Virtio drivers virtio-win-0.1.173.iso

VFIO Graphics Passthrough

I already had a working VFIO graphics passthrough using the Nvidia GTX 970. In fact, passing through the older Nvidia GPUs is a breeze. No VBIOS needed, just declaring the PCI bus IDs.

Modern Nvidia graphics cards are often reported to black-screen after passthrough. This can happen when the passthrough graphics card gets initialized before it is bound to the vfio-pci driver. A solution to this problem is to pass along the VBIOS of the graphics card. So lets go step by step.

Initial configuration

I have covered the initial setup in two different tutorials: Creating a Windows 10 VM on the AMD Ryzen 9 3900X using Qemu 4.0 and VGA Passthrough and Running Windows 10 on Linux using KVM with VGA Passthrough.

In summary the initial configuration steps are:

    1. Enable virtualization, in particular IOMMU / VTd, using the motherboard BIOS screen.
    2. Install the required packages and download the Virtio drivers ISO.

The above steps are described in detail in the above tutorials. If you own a modern AMD Ryzen CPU, follow this link.

Continue with the next step after you enabled IOMMU and downloaded/installed the required packages.

Obtain the correct VBIOS

Unfortunately neither Nvidia nor the graphics card vendor (in my case Gigabyte) offer downloadable VBIOS images. Lucky for us there are other ways to obtain the VBIOS. I’ll describe both of them.

If you have a working Windows installation that uses the graphics card you wish to pass through:

    1. Inside Windows, download the GPU-Z utility from TechPowerUp.
    2. Open GPU-Z.
      GPU-Z
      GPU-Z application

      To save the graphic card VBIOS, click the grey right arrow underneath the NVIDIA logo. Then select “Save to file…” to save the VBIOS. (Note: The screenshot above was taken inside my Windows VM, which prevented me to save the VBIOS.)

Alternatively, download a VBIOS from the TechPowerUp VBIOS database:

    1. Open a terminal window and enter:lspci -knn
      lspci -knn to obtain device ID of GPU

      Look for your graphics card in the output. My NVIDIA 2070 Super GPU has the subsystem device ID 1458:4001. The subsystem ID specifies the GPU manufacturer and model, which is what you want. Write down the subsystem ID (make sure to get the one from the graphics card you wish to pass through, NOT the one your host uses).

    2. Open your browser and write / paste the following into your DuckDuckGo (or Google) search window:

      techpowerup 1458:4001

      Replace the numbers 1458:4001 with the subsystem ID you determined. Follow the link to the TechPowerUp VBIOS database site. You will get a list similar to the one below:

      VBIOS options for device ID 1458:4001
    3. There may be different VBIOS files to choose from. Click “Details” to learn more about the files and the corresponding graphics card. Make sure you get the right one that matches your GPU vendor, model, and revision.
      Download the VBIOS.

Edit VBIOS file using a hex editor

In recent years NVIDIA has packed additional code into the VBIOS. This additional code prevents qemu to properly load the VBIOS and initialize the graphics card.

We are going to use a hex editor to edit the VBIOS file and strip off the unnecessary code at the beginning of the file. Sounds difficult? It’s not.

    1. Install a hexadecimal (hex) editor of your choice. Using Linux, I found “bless” to be a suitable choice.
    2. Open the VBIOS file using the hex editor.
      bless hex editor with VBIOS file
    3. Enlarge the hex editor window. Click the search icon and search for “55AA” hexadecimal. If necessary, click the “Find Next” button until you see the word “VIDEO” appear in the right (text) window (see below).
      Beginning of VBIOS code

      This is the beginning of the actual VBIOS, starting with 55AA hexadezimal, some letters or symbols and then the word “VIDEO” and some more text.

      Note: The hexadecimal sequence 55AA may appear one or more times before the actual beginning of the BIOS code. So always look for the word “VIDEO”!

    4. Mark and delete everything BEFORE the part beginning with 55 AA. For faster editing, use the editor in full screen mode. The result should look like below.
      Edited VBIOS file

      The file now starts with the hexadecimal sequence 55AA.

    5. Save the VBIOS file under a different name, for example:

      2070S-VBIOS-edited.rom

      Close the editor.

Bind passthrough GPU to VFIO driver

In this step, we

    1. list the GPUs installed in your system, together with their PCI ID;
    2. determine the PCI device IDs for the GPU sub-devices. Note that GPUs often have multiple sub-devices, all of which must be passed to the Windows VM;
    3. check the IOMMU group(s) for the GPU and other PCI devices we want to pass through to the Windows VM;
    4. decide on which method to use to bind the GPU to the vfio-pci driver;
    5. determine the vendor and model IDs for the devices to pass through, if required (depends on the method we chose);
    6. bind the passthrough GPU to the vfio-pci driver using the method we determined to best suit the configuration.

The steps together with some background information are listed in my previous tutorial here. For clarity, I am describing the steps below.

List the GPUs in your system:

lspci | grep VGA

[[email protected] ~]$ lspci | grep VGA
0b:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)
0c:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)

I selected the Nvidia GeForce RTX 2070 Super for my Windows VM, which has the PCI bus ID 0b:00.0. The .0 in the bus ID denotes the sub-device called “function” in PCI slang. Most likely the GPU has several sub-devices such as graphics, audio, USB, etc. A GPU is therefore a “multifunction” device (we’ll get to that later).

To determine all sub-devices, remove the trailing “0” in the grep command as follows:

lspci -nn | grep 0b:00.

[[email protected] ~]$ lspci -nn | grep 0b:00.
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1)
0b:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
0b:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)

Make sure to replace the PCI bus ID with the one you determined!

My Nvidia RTX 2070 Super has 4 sub-devices in total:

    • 0b:00.0 – graphics
    • 0b:00.1 – audio
    • 0b:00.2 – USB controller
    • 0b:00.3 – serial bus controller

All of them must be passed through to the Windows VM.

Next we check the IOMMU grouping. For VGA passthrough to work, the configuration needs to fulfill the following two requirements:

    1. All PCI (sub-) devices inside a specific IOMMU group must be passed to the VM (exception: PCI root ports/controllers);
    2. All PCI sub-devices of the passthrough GPU must be passed to the VM.

To list all IOMMU groups and their PCI devices, use:

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort

[[email protected] ~]$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort –version-sort
/sys/kernel/iommu_groups/0/devices/0000:00:01.0

/sys/kernel/iommu_groups/25/devices/0000:07:00.0
/sys/kernel/iommu_groups/26/devices/0000:0b:00.0
/sys/kernel/iommu_groups/26/devices/0000:0b:00.1
/sys/kernel/iommu_groups/26/devices/0000:0b:00.2
/sys/kernel/iommu_groups/26/devices/0000:0b:00.3
/sys/kernel/iommu_groups/27/devices/0000:0c:00.0
/sys/kernel/iommu_groups/27/devices/0000:0c:00.1

GPU vfio
PCI bus IDs and IOMMU group for my Nvidia RTX 2070 Super GPU

As you can see, my Nvidia RTX GPU with PCI bus IDs 0b:00.0 to 0b:00.3 is within IOMMU group 26. There are no additional devices in that IOMMU group. Perfect!

Now we need to determine the method we’re going to use to bind the passthrough GPU to the vfio-pci driver. Typical with Linux, there are several ways to bind the GPU. So here is how we decide:

    1. You have two non-identical GPUs in your system: use the boot manager  (grub) method (this works also with Debian or Ubuntu 20.04 based distros and latest kernels).
    2. You have two identical GPUs for host and guest: you must use the driver override method *. Caveat: Debian or Ubuntu 20.04 based distros, together with kernel 5.4. and later, have introduced a bug that prevents the built-in driver to attach to the GPU.
    3. If you have only one GPU that you use for the host and wish to pass this GPU to your VM, then you are reading the wrong tutorial. Check Yuri Alek’s or Joe Knock’s tutorials. Don’t worry – what you’ve done so far is still valid.

(* There is also a pciback driver that could be used instead of vfio-pci, but the latter has significant improvements over the former. I mention this so you know that there is yet another option, in case you need it.)

As a precaution, before you bind the GPU to the vfio-pci driver, you should change your Linux graphics driver to the open source version. Look for your driver management application and select the open-source Nouveau driver for the Nvidia card you wish to pass through.

On a Manjaro Linux system, open “Manjaro Settings Manager”, then double-click on “Hardware Configuration”:

vfio GPU passthrough
Manjaro Settings Manager – Hardware Configuration screen

On Linux Mint, open “Control Center”, then click “Driver Manager”:

vfio GPU passthrough
Control Center – Driver Manager GUI (example)

Make sure to install and select the nouveau open-source driver for the GPU you wish to pass through.

After changing the driver, reboot to see everything still works.

Let’s bind the graphics card to the vfio-pci driver. Using the boot manager method, do the following:

    1. Enter again the following command:lspci -nn | grep 0b:00.[[email protected] ~]$ lspci -nn | grep 0b:00.
      0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1)
      0b:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
      0b:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
      0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)Write down or copy/paste the vendor and model IDs into a file. The vendor:model IDs for all the sub-devices listed above are:

      10de:1e84
      – graphics
      10de:10f8 – audio
      10de:1ad8 – USB
      10de:1ad9 – serial bus
    2. Open the /etc/default/grub file with root permissions, for example using:sudo nano /etc/default/grub
    3. Look for the line that starts with “GRUB_CMDLINE_LINUX_DEFAULT=”.At the end of that line, before the quote, append the options “kvm.ignore_msrs=1 vfio-pci.ids=” followed by your vendor:model IDs, separated by commas. Here is an example:kvm.ignore_msrs=1 vfio-pci.ids=10de:1e84,10de:10f8,10de:1ad8,10de:1ad9The above has to be all in one line (don’t press Enter). The kvm.ignore_msrs=1 option is required for all newer releases of Windows 10.
    4. Make sure the options after the “GRUB_CMDLINE_LINUX_DEFAULT=” are enclosed in double quotes. Below is an example /etc/default/grub file for reference:
      vfio GPU passthrough
      /etc/default/grub file for vfio GPU passthrough

      Save and close the file.

    5. To update the grub boot loader, execute:sudo update-grub
    6. Reboot the PC.

If you chose the driver override method, please follow the instructions under “Bind Passthrough GPU to VFIO Driver“, section  Pop!_OS 19.10 / Ubuntu 19.10 / pre-5.4 kernel“.

Before we continue with creating or editing the VM configuration, let’s check that the graphics card has been bound to the vfio-pci driver:

lspci -kn | grep -A 2 0b:00.

with 0b:00. being the PCI device ID for your GPU, without function (.0).

[[email protected] ~]$ lspci -kn | grep -A 2 0b:00.
0b:00.0 0300: 10de:1e84 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci

0b:00.1 0403: 10de:10f8 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci

0b:00.2 0c03: 10de:1ad8 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: xhci_hcd

0b:00.3 0c80: 10de:1ad9 (rev a1)
Subsystem: 1458:4001
Kernel driver in use: vfio-pci

All but the USB devices use the vfio-pci kernel driver. Success!

Create/Modify the XML VM configuration file

The process of creating a Windows 10 VM using the Virtual Machine Manager GUI is described in detail under “Create Windows 10 VM Configuration using the Virtual Machine Manager GUI” followed by “Additional XML Configurations“. If you are creating a new Windows VM, follow those instructions and return back here.

Booting the virtual machine using a VBIOS / ROM file requires us to modify the XML configuration we have created for the VM:

    1. Open the Virtual Machine Manager GUI.
    2. Select graphics card PCI device, in my case it is PCI 0000:0b:00.0:
      vfio GPU passthrough
      VFIO passthrough GPU in Virtual Machine Manager

      Note the device description.

    3. Select the XML tab. You must have XML editing enabled in the Virtual Machine Manager settings.
    4. Insert the following line below the “</source>” tag:<rom file=”/home/heiko/.bin/Gigabyte_RTX2070Super_8192_191021_edit.rom”/>Replace the path with the path to your .rom file!
    5. Modify the “<address type=>” line so that it reads “multifunction=’on'” at the end before the closing tag, like here:<address type=”pci” domain=”0x0000″ bus=”0x04″ slot=”0x00″ function=”0x0″ multifunction=”on”/>
      loading VBIOS into passthrough VM
      Loading a VBIOS .rom file into the VM

      Make sure to click “Apply” after you have finished with the changes!

      Here some explanation: The Virtual Machine Manager isn’t too smart and cannot determine multifunction devices. As a result, it creates a separate device for each sub-device or function on the graphics card and puts each function in a separate “virtual” PCIe slot. The Windows guest then sees the graphics device in one slot, the audio device in another slot, the USB device in yet another slot and so on. As strange as this may sound, it usually works.

      However, a Windows 10 or graphics driver update has the potential of breaking things. Moreover, if you run Windows 10 sometimes on bare metal using “dual-boot” and sometimes as a virtual machine inside kvm, you better make sure that the differences that Windows perceives are as small as possible.

      This and the following steps create a virtual PCIe slot that contains the graphics card and its sub-devices as functions on that slot (ergo the name “multifunction”).

      While we are at it, let’s decipher the syntax of the XML entry:

      <hostdev mode=”subsystem” type=”pci” managed=”yes”>

      means that this defines a subsystem of type PCI that is managed by Virtual Machine Manager / libvirt.

      <source>
        <address domain=”0x0000″ bus=”0x0b” slot=”0x00″ function=”0x0″/>
      </source>

      is the physical location of the hardware device in your PC. We have seen this before when listing the IOMMU groups, but in a different notation: 0000:0b:00.0

      <address type=”pci” domain=”0x0000″ bus=”0x04″ slot=”0x00″ function=”0x0″ multifunction=”on”/>

      is the virtual PCI bus that Windows will see. In addition, this line tells QEMU that it’s a multifunction device. Write down the bus and slot numbers!

    6. We have to modify the XML configuration for each sub-device/function on the graphics card. Select the PCI (sub-)device that follows, PCI 0000:0b:00.1 in my case, and edit as follows:<address type=”pci” domain=”0x0000″ bus=”0x04″ slot=”0x00″ function=”0x1″/>Change the bus number (if different) and slot number to the ones you wrote above. Then change the function from 0 to 1, like:function=”0x1″

      Click “Apply”.

      multifunction devices
      Domain, bus and slot numbers are identical for all sub-devices of a multifunction device, just the function changes!

      Note: All sub-devices (functions) of the GPU use the same domain, bus, and slot. The only number that should change is for the function, i.e. 0, 1, 2, …

    7. Edit the next PCI sub-device, 0000:0b:00.2 in my example:<address type=”pci” domain=”0x0000″ bus=”0x04″ slot=”0x00″ function=”0x2″/>Make sure you got the same bus and slot as before, and increment the function=0x2.
    8. Repeat one more time (for all 4 functions of the GPU).

This should complete the configuration for the Nvidia graphics card. Before you boot the VM, make sure memballoon is turned off.

Click “Overview” on the top left Virtual Machine Manager VM configuration screen, then select the “XML” tab.

Scroll down to near the end of the XML configuration and look for the “memballoon” entry. Change that entry as follows:

<memballoon model=”none”/>

Click “Apply”.

Boot the Windows VM

Now is the time to buckle up and boot your Windows VM. Start the Windows VM from within Virtual Machine Manager. If it’s a new installation, follow the Windows 10 instructions in my other tutorial.

If you already have a Windows 10 VM and upgraded your graphics card, Windows will search for a new driver and install it. You should still go to the Nvidia driver download website to get the latest driver.

After each Nvidia driver installation or upgrade, Nvidia will revert to line based interrupts, which is less efficient and may cause audio issues. To fix that, see MSI based interrupts in my tuning section.

I hope this has been helpful. Let me know how it went by posting a comment. Likewise if you had issues.

Tuning and trouble-shooting

If you run into troubles, you can find a lot more information on this site – see my other two tutorials. Those tutorials also contain a long list of references, links, other tutorials and whatever I felt could be of help.

There is also a very active VFIO reddit that you can check out.

Finally the best: Check out the comprehensive VFIO channel on Discord with concise, up-to-date instructions and helpful experts.

If this article has been helpful, click the “Like” button below. Don’t forget to share this page with your friends.

This site uses Akismet to reduce spam. Learn how your comment data is processed.