GPU Passthrough with Low-End Intel i3-6100 CPU

For some time I wanted to run a kvm virtual machine with GPU passthrough on a low-end Asus H110M-K D3 motherboard with an i3-6100 CPU and an Nvidia GTX 1060 GPU, but never found the time. Now I finally had a chance to give it a try. While the preparations were easy, I ran into a problem when starting the Windows 10 VM:

Jan 22 15:21:24 alon-i3 kernel: [  108.514992] vfio_pin_pages_remote: RLIMIT_MEMLOCK (16777216) exceeded

A search on the web didn’t produce useful results so I posted in the Reddit VFIO group. Thanks to Alex Williamson who gave me the solution. It was an issue with memory usage restrictions that was easy to resolve.

Hardware & Software

Hardware configuration:

• Intel i3-6100 CPU with 2 cores and hyper-threading (supports VT-d), IGPU
• Asus H110M-K D3 motherboard, BIOS version 2604
• 8GB DDR3 RAM
• Nvidia GeForce GTX 1060 with 6GB memory
• Samsung SSD 850 EVO 250GB

Host configuration:

• Linux Mint 19.1 Mate 64bit
• Kernel 4.18
• Qemu 2.11
• LVM formatted root partition
• i3-6100 IGPU for graphics

Guest configuration:

• Windows 10 (1803) 64 bit
• 4GB memory using hugepages
• 100GB RAW image file, fully pre-allocated
• Nvidia GeForce GTX 1060 graphics

Creating a Windows 10 Virtual Machine

I followed my tutorial on Running Windows 10 on Linux using KVM with VGA Passthrough:

• Update motherboard BIOS to version 2604, the latest as of this writing
• BIOS: Enable Virtualization under CPU settings
• BIOS: Enable VT-d under the Advanced tab
• BIOS: In the Graphics settings, select CPU internal graphics
• Follow the tutorial up until and including “Check configuration”

The following command creates the virtual raw disk image for Windows:

qemu-img create -f raw -o preallocation=full vmdisk.img 100G

Everything went smooth until I actually ran the VM start script. Every time I tried to run it, the VM crashed with a core dump. Note that I run my VMs using the -runas myusername option, meaning in user mode.

Looking at /var/log/kern.log provided the following clue:

Jan 22 15:21:21 alon-i3 kernel: [  106.204809] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Jan 22 15:21:21 alon-i3 kernel: [  106.205021] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jan 22 15:21:24 alon-i3 kernel: [  108.514992] vfio_pin_pages_remote: RLIMIT_MEMLOCK (16777216) exceeded
Jan 22 15:21:24 alon-i3 kernel: [  108.632857] bridge0: port 2(vmtap0) entered disabled state
Jan 22 15:21:24 alon-i3 kernel: [  108.633266] device vmtap0 left promiscuous mode
Jan 22 15:21:24 alon-i3 kernel: [  108.633270] bridge0: port 2(vmtap0) entered disabled state

vfio_pin_pages_remote: RLIMIT_MEMLOCK (16777216) exceeded means that the Windows guest exceeds its memory limit, which is 16 MB.

The solution is to increase the memory limit, or simply specify “unlimited”. Open /etc/security/limits.conf and add the following lines:

@kvm            soft    memlock         unlimited
@kvm            hard    memlock         unlimited

@kvm means the kvm group, and both soft and hard memlock are set to unlimited. You can also enter a limit (for example 4294967296 for 4 GB), just make sure it’s big enough. Make sure your Windows VM user is a member of the kvm group:

sudo usermod -a -G kvm myusername

See Run Windows VM in user mode (non-root) for a complete description on how to run the Virtual Machine in user mode.

Here is the script I used for the Windows 10 VM:

#!/bin/bash
vmname="windows10vm"
# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native
cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd
chown alon:kvm /tmp/my_vars.fd
qemu-system-x86_64 \
-name $vmname,process=$vmname \
-runas alon \
-enable-kvm \
-machine type=q35,accel=kvm \
-cpu host,kvm=off,hv_vendor_id=1234567890ab,hv_vapic,hv_time,hv_relaxed,hv_spinlocks=0x1fff,l3-cache=on,-hypervisor,migratable=no,+invtsc \
-smp 4,sockets=1,cores=2,threads=2 \
-mem-path /dev/hugepages \
-mem-prealloc \
-m 4G \
-balloon none \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-no-hpet \
-serial none \
-parallel none \
-soundhw hda \
-usb \
-device usb-host,vendorid=0x045e,productid=0x076c \
-device usb-host,vendorid=0x045e,productid=0x07b9 \
-device pcie-root-port,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-boot order=dc \
-object iothread,id=io1 \
-device virtio-blk-pci,drive=disk0,iothread=io1 \
-drive if=none,id=disk0,cache=none,format=raw,aio=threads,file=/home/alon/vmdisk.img \
-drive file=/home/alon/Downloads/win10home.img,index=1,media=cdrom \
-drive file=/home/alon/Downloads/virtio-win-0.1.160.iso,index=2,media=cdrom \
-netdev type=tap,id=net0,ifname=vmtap0,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:02
exit 0

Once the “RLIMIT_MEMLOCK exceeded” problem was solved, the VM started as expected and the installation of Windows 10 was easy.

Note that the above script may leave room for improvement. I did enable MSI for the graphics card and its audio part inside the Windows registry, but the crackling sound didn’t disappear.

Windows needed some help in identifying the C: drive as SSD. Typing winsat formal in a Windows administrator shell did the trick.

Unfortunately I had to restore the PC to its original state and return it to its owner, so no more tweaking for now.

Summary

Except for running into the memlock issue which was a first time for me, creating a Windows virtual gaming PC on a low-end platform was surprisingly easy.

Considering the base-line CPU and motherboard, the PC provides reasonable performance. The only issue that remains is the crackling sound, but I hadn’t given it any attention.

Here some benchmarks:

UserBenchmarks: 
 CPU: Intel Core i3-6100 - 64.1%
 GPU: Nvidia GTX 1060-6GB - 74.5%
 HDD: Red Hat VirtIO 107GB - 167.6%
 RAM: QEMU 1x4GB - 58.7%
 MBD: QEMU Standard PC (Q35 + ICH9, 2009)

Note that UserBenchmark did not correctly identify the SSD.

I’m sure that with a little effort the crackling sound issue could have been solved.

Other Thoughts

The crackling sound phenomena has been one of the most obnoxious issues with VFIO GPU passthrough. For gamers, the lack of good audio support is a deal breaker.
There are a number of remedies on can try and I have listed or linked to them in my passthrough tutorial, but the most promising one is implemented in the new qemu 3 release. (It seems that the pulse audio driver patch that Reddit user Spheenix wrote made it into the new qemu 3.0+ releases.)

Sadly the current version of Linux Mint (release 19.1) only comes with qemu 2.11, as it’s based on Ubuntu 18.04. If you chose Ubuntu 19.04, you’ll be able to enjoy qemu 3.1. There is also the option to compile qemu from source.

For some time I’ve been trying out Manjaro Linux, a distribution based on Arch Linux. It uses a “rolling release” model and is therefore up-to-date. I have no experience as to its robustness (Linux Mint has been rock solid for me, so far), as only time will tell. Using the latest qemu release, it will be my next candidate for the i3-6100 based GPU passthrough attempt.