Upgrading my PC to an AMD Ryzen 9 3900X System

Introduction

I’ve been contemplating a PC upgrade for more than a year (see my post here). At first I considered staying with Intel and getting an i9-9900K CPU with integrated GPU on a Z390 motherboard.

Along came the AMD Ryzen 9 3900X that topped the benchmarks, including the Adobe Lightroom and Photoshop benchmarks (to be precise, it ranked #4 in the Adobe Lightroom benchmark, and a narrow #1 in the Adobe Photoshop benchmark). These good news about the AMD Ryzen 3900X were soon followed by reports about BIOS issues and VFIO incompatibility. At the very least, it looked like VGA passthrough was more challenging.

Then I read Bryan Steiner’s GPU passthrough tutorial for the AMD Ryzen 9 3900X and an Ubuntu-based Pop!_OS Linux host. Several users on the VFIO Reddit forum reported successful VFIO VGA passthrough with the AMD Ryzen 9.

Thanks to the side effects of the Corona virus, I also found the time for this new build. So I went ahead and bought the PC parts.

Hardware

Instead of purchasing all the parts necessary to build a new PC, I will use most of my existing PC components.

Here is the list of the new PC parts I purchased:

• AMD Ryzen 9 3900X CPU
• Gigabyte X570 Aorus Pro motherboard
• Noctua NH-D15S CPU air cooler
• Corsair Vengeance LPX 2x32GB 3200 MHz DDR4 system memory (64 GB RAM)
• Samsung 970 EVO Plus 1 TB NVMe M.2 SSD (for Windows VM)
• Samsung 970 EVO Plus 500 GB (for Linux host)

Below are the existing components that will be used:

• Gigabyte GeForce GTX 970 graphics card for the VM
• PNY Quadro 2000 graphics card for the Linux host
• Samsung SSD 850 EVO 1TB (currently holding some VMs, but will be repurposed)
• 4 HDD drives of various makes, totaling 11 TB
• Corsair Carbide 500R chassis
• SeaSonic 660W Gold X modular power supply

I was going to get perhaps an Nvidia GT 1030 low-end graphics card as a replacement for the Quadro 2000, but I hope to postpone any further upgrade for now.

Which Host OS?

I’ve been using Linux Mint ever since it came out, or shortly thereafter. Despite some criticism and security concerns, it has been extremely stable for me. I liked it so much that when the first VGA passthrough tutorials came out for Fedora and later Arch Linux, I stuck to Linux Mint and wrote my own Xen-based passthrough tutorial to support this platform. In the meantime I switched to Qemu/kvm and again it was easy to setup on my Intel i7 3930K based system.

The AMD Ryzen 9 platform, however, is a different story. It’s new and demands the latest kernel releases as well as a modern, updated version of Qemu. The latter has built-in improvements for audio and other “goodies”.

Linux Mint 19.3 ships with Qemu 2.11, which can be updated to Qemu 2.12 through PPA. Compare that to running Qemu 4.0 on Pop!_OS or Manjaro.

I’ve been testing Manjaro for about half a year and I was going to install it as my new host OS. Based on Arch Linux, it’s a rolling distribution meaning it’s constantly updated as new package releases emerge. This ensures that the host is getting the latest improvements and compatibility updates, especially important when using newly released hardware.

But Manjaro has also its downsides: Lots of updates with some chance that an update can break things. The other downside is that it comes with tons of software, a lot of which I consider unnecessary.

Reading the above mentioned tutorial by Bryan Steiner, I started to take a closer look at Pop!_OS. I haven’t been a fan of Ubuntu’s (not so) new Gnome desktop, as it’s different to what I’m used to. But using Pop!_OS with the Ubuntu desktop for a week or so made me more comfortable with the concept.

Pop!_OS comes in two versions:

• Intel/AMD
• Nvidia

I chose the Nvidia version to test in a VM with the Nvidia GTX 970 and everything was working fine.

Edit: Pop!_OS turned out to be a real disappointment (see below)!

Migration Plan

I have a lot of data that I want/need to keep. The PC will be mostly used for work (accounting, web, email, processing photos, etc.), so downtime during migration should be minimal.

I will be reusing my existing storage, except for an old Sandisk 120 GB SSD that contains the Linux Mint host and some data that I want to transfer to the new 500 GB NVMe SSD.

Here are the preliminary steps before touching the PC hardware:

1. Backup /home/user on the Linux host to external media and remote server – 2 copies
2. Backup Windows VM to backup media and remote server – 2 copies
3. Within the Windows VM, copy all Adobe Lightroom and Photoshop folders under /Pictures to non-system NTFS drive – I plan to completely reinstall Windows 10 and the Adobe CC suite
4. Within the Windows VM, copy all utilities I want to preserve to non-system NTFS drive
5. Create bootable USB sticks for:
   1. Linux host OS – this time I plan to use Pop!_OS
   2. Windows 10 – to install Windows directly on the NVMe drive, if necessary
   3. Linux Mint and/or Manjaro – in case I need a fallback host OS that I’m more familiar with, or for fixing things
   4. Test each of the above USB sticks to verify that they boot
6. Create a USB stick with the following content:
   1. User space scripts to create VMs, to mount VM partitions on the host, etc.
   2. Content of /root that includes Luckybackup configuration and backup/restore scripts for host and Windows VM
   3. ssh keys and configurations
   4. Samba configuration, fstab, etc.
   5. VFIO / Qemu configuration files and scripts
   6. If using libvirt / virt-manager, copy the xml and config files
7. Make a checklist of all the applications needed to run on the new host – if necessary, find and test possible replacements
8. “Dry run” the new host system in a VM
9. Optional – for my non-UEFI PNY Quadro 2000 card:
   1. Download patched vBIOS from Techpowerup to support UEFI for my PNY Quadro 2000 – the vBIOS can be found here
   2. Download Linux version of Nvidia Nvram BIOS flashing software from Techpowerup NVIDIA NVFlash
   3. Only if you want/need to upgrade your GPU BIOS: flash the vBIOS of your graphics card following the GUIDE – NVIDIA GPU VBIOS Flashing with HiveOS. Disclaimer: there is a good chance that this will brick your graphics card – don’t try unless you really know what you are doing! You have been warned.

After all of the above steps were performed successfully, I proceeded with disassembling and reassembling the PC with the new components.

Hardware Installation

That was the easiest part. Remove the GPUs and PCIe cards, take out the old m/b with CPU and cooler.

Remove the original SSD boot disk holding my old Linux host system and make it available via external docking station (it holds a small boot partition, the rest is LVM volumes).

Place the AMD CPU on the motherboard and install the new cooler, install the motherboard in the chassis, then place the PCIe cards. Hook up the internal and external cables and done.

Pop!_OS Installation

I had prepared a Pop!_OS boot disk with the Nvidia driver, since I got two Nvidia cards. In the BIOS, I chose to use the Quadro 2000 in the second PCIe slot as the primary GPU. The GTX 970 sitting in the first slot is going to be passed through to the Windows VM.

The Nvidia version came with a 440.44 Nvidia driver that supports my passthrough GTX 970. However, it doesn’t support the Nvidia Quadro 2000 and when booting the host with the Pop!_OS Nvidia USB stick the system got stuck at the “starting firmware daemon” message 🙁 .

Flashing an UEFI BIOS to the Quadro 2000 didn’t make a difference. This is an old card and the latest driver supporting this GPU is a Nvidia 390 driver. Luckily this driver is available in the Pop!_OS / Ubuntu repository, but I had to use the Intel/AMD version of the Pop!_OS installation ISO to install the system on the host. So I prepared another USB stick, this time without the proprietary Nvidia driver. That went fine.

The Pop!_OS installer is a bit of a disappointment. It doesn’t allow for LVM formatting, but recognizes it once I designated a partition as LVM partition using gparted (which has to be installed manually).

Coming from Linux Mint and having used a Mate desktop for years, Pop!_OS is quite a challenge.

First the choice of desktop – come on, I’m not fiddling around with a stupid tablet, but have a real screen. Yet somehow the makers of Pop!_OS think that touchscreens are universal and who am I to question that wisdom?

Mouse travel is unacceptable, so is the need to jump from mouse to keyboard shortcuts back to mouse. Having to click several times to get to an application – without the ability to create a desktop / panel / whatever shortcut – is simply a crime. I had RSI, I don’t want to be there again.

I nearly threw Pop!_OS out the window, and if it wasn’t relatively up-to-date without being bleeding edge, I would have done so.

I tried Mate on Pop!_OS which looked terrible out of the box and was cluttered with useless applications. So I removed Mate.

I decided to keep going with Pop!_OS – there must be a reason why people like it?

The first light on the horizon was this post. I followed the steps described there, did some more reading, and learned how to customize the desktop. So finally I was pleased on the desktop / UI side. Until…

a Gnome plugin updates and breaks 🙁 , which happens about once day. As I write this, I’m fighting with “Dash to Panel” to go/stay at the bottom, and for “Horizontal Workspaces” to uninstall as all of a sudden it places itself above the panel and won’t disappear.

I removed Gnome extensions, I reinstalled them, nothing helps. Every time I boot there comes a point where the desktop gets garbled.

Back to installation: I made some changes (load modules, hugepages, etc.) and ran update-initramfs. Next boot the changes weren’t implemented – great!

Turns out update-initramfs -u updates only the initrd.img in /boot, but not the one in /boot/efi/EFI/Pop_OS-… where the UEFI bootloader looks for the kernel and initial ramdisk. Another 1/2-1 hour wasted in debugging. At least I filed a bug report, in the hope somebody fixes this nuisance. For now, whenever you need update-initramfs, use the following combination to fix the problem:

sudo update-initramfs -u

sudo kernelstub

The second command does what the first one should have done after creating the initrd image.

Ah, which reminds me – they threw out the familiar grub2 which has worked and still works in so many other distributions. That’s why you need kernelstub in the first place.

I’ve spent literally days in getting the host up and running, installing the software I need and making it accessible.

Talking about package installation. Pop!_Shop may look nice at first, but is nothing more than a severely crippled package manager. Screenshots are usually unavailable. There are no package description, no release notes, no nothing. The search function finds everything just not the package I’m looking for. And the software updater collects most updates under the vague term “Operating System Updates” without further clues. It’s almost as bad as Microsoft Windows!

I could fill pages listing more bugs and flaws. It’s almost as if Pop!_OS wants to tell me: “Leave me alone”.

Eventually I got a somewhat working system and created a Windows 10 VM using Virtual Machine Manager and some XML hacking. You can find the tutorial under Creating a Windows 10 VM on the AMD Ryzen 9 3900X using Qemu 4.0 and VGA Passthrough.

After Pop!_OS had been kind to me for a couple of days, I suddenly wasn’t able to print from a Macbook (it did work before). No matter what I tried, nothing helped. I searched for a support forum and only found the Pop!_OS reddit. My call for help remained unanswered. The Pop!_OS website itself offers some decent instructions for common issues, but it can’t cover everything.

The printer problem as well as the lack of response from the community and distro maintainer settled it: I threw out Pop!_OS and installed Manjaro. Finally back to sanity.