I was so busy studying, writing, and processing photos that I didn’t touch my Manjaro Linux host for a long long time. Then one day I decided to upgrade my PC from a 5.4 kernel to the most recent LTS or “Long Term Support” Manjaro Linux kernel 5.10.
Installing a New Kernel in Manjaro
Open the Applications menu (in the bottom left corner) and search for “settings”. Then open the Manjaro Settings Manager and double-click “Kernel”. Choose the kernel you wish to install and click “Install”. Before doing so, have a look at the “Changelog”. Is it worth the upgrade?
Don’t get carried away with promises of improved IO, reduced memory latency, faster you-know-what etc. Improvements are often, if any, minimal. A good reason to try the latest and greatest kernel is new hardware that has no or only limited support in older kernels.
For the sake of convenience and peace of mind, I only consider LTS releases. They are released about once a year and receive security updates and bug fixes over many years. Unless you love to experiment and don’t mind fixing broken systems or debugging, choose the latest LTS kernel.
After installing the new kernel, you need to reboot to activate it. Don’t remove the old kernel, yet. It’s always good to keep a known, working kernel version installed.
Running into Problems
So I installed Manjaro Kernel 5.10 and rebooted. Everything went fine until I tried to start my Windows VM: /dev/media/photo_raw not available.
I’m using LVM (Logical Volumes) for everything except the /boot/efi partition (which must be vfat on a native partition). Here are my volume groups (VG):
> sudo vgs
VG #PV #LV #SN Attr VSize VFree
host 1 4 0 wz--n- <465.26g <169.26g
media 2 2 0 wz--n- <7.28t 1.13t
photos 1 2 0 wz--n- 10.91t <4.40t
vmvg 2 14 1 wz--n- <1.82t <772.25g
“photo_raw is” a logical volume (LV) in volume group “media”. This LV is a virtual drive under Windows 1o where I store a second copy of all my imported photos. What happened to it?
> sudo lvscan
inactive '/dev/media/movies' [<3.42 TiB] inherit
inactive '/dev/media/photo_raw' [<2.73 TiB] inherit
ACTIVE '/dev/vmvg/win10' [270.00 GiB] inherit
ACTIVE '/dev/vmvg/workdrive' [400.00 GiB] inherit
ACTIVE '/dev/vmvg/centos8' [50.00 GiB] inherit
ACTIVE '/dev/vmvg/linuxmint' [50.00 GiB] inherit
ACTIVE '/dev/vmvg/opensuse15.1' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/worker2' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/worker1' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/worker3' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/worker4' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/ubuntu1804' [40.00 GiB] inherit
ACTIVE '/dev/vmvg/pop_os20' [25.00 GiB] inherit
ACTIVE '/dev/vmvg/debian10' [20.00 GiB] inherit
ACTIVE Original '/dev/vmvg/ubuntu' [20.00 GiB] inherit
ACTIVE Snapshot '/dev/vmvg/ubuntusnap' [<15.77 GiB] inherit
ACTIVE '/dev/host/swap' [16.00 GiB] inherit
ACTIVE '/dev/host/root' [30.00 GiB] inherit
ACTIVE '/dev/host/home' [150.00 GiB] inherit
ACTIVE '/dev/host/data' [100.00 GiB] inherit
ACTIVE '/dev/photos/photo_stripe' [5.73 TiB] inherit
ACTIVE '/dev/photos/tmp_stripe' [800.00 GiB] inherit
Both LVs in VG “media” were not activated. Activating them manually worked fine:
sudo vgchange -a y media
But after a reboot they were again not activated. I could go into the nasty details of creating a systemd service to automate the activation, but it was in vain. Whatever I tried, the two logical volumes under volume group media were not activated after a reboot.
The Culprit
Searching for a solution, I found this thread: Does mkinitcpio.conf survive new kernel? Aha, when upgrading the Linux kernel, Manjaro does not update the configuration file that determines the initramfs image to boot with. It also does not ask the user to edit or replace this file (I believe Linux Mint and Ubuntu do).
Here is an excerpt of my old /etc/mkinitcpio.conf file, without comments:
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net"
FILES="/usr/local/bin/vfio-pci-override.sh"
HOOKS="base systemd autodetect modconf block keyboard keymap lvm2 filesystems"
Newer kernels have vfio-pci integrated, so the modules don’t need to be loaded.
Note: The Arch Linux Wiki still advises to load the modules – there is no harm in doing so. Just keep the “MODULES” section above as is.
I had forgotten to remove the override script from the FILES section. Instead of the override script I use now the /etc/default/grub file to specify the PCI IDs for the vfio-pci driver:
GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor udev.log_priority=3 vfio-pci.ids=10de:1e84,10de:10f8,10de:1ad8,10de:1ad9"
The culprit, however, was most likely the HOOKS section. This has changed significantly. The new section in /etc/mkinitcpio.conf.pacnew looks like this:
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)
There is one item missing, though: lvm2. My revised mkinitcpio.conf file has now the following entry under HOOKS:
HOOKS=(base udev autodetect modconf block lvm2 filesystems keyboard fsck)
Following my changes in the mkinitcpio.conf file, I ran:
sudo mkinitcpio -p linux510
This will create a new initramfs for the Manjaro Linux 5.10 kernel only.
After a reboot, all LVM devices were properly activated.
Summary
Manjaro and the Linux kernel undergo constant changes. When upgrading to a newer kernel, it is advisable to check and compare the newly installed /etc/mkinitcpio.conf.pacnew file with the existing /etc/mkinitcpio.conf file.
If you select to use the /etc/mkinitcpio.conf.pacnew as your new /etc/mkinitcpio.conf file, make sure to add missing options like lvm2 in my case (though chances are it works without that option).
Useful Tools to Create the initramfs
The command mkinitcpio provides a list of useful options. To list all available hooks, use:
mkinitcpio -L
For help on a specific hook, use:
mkinitcpio -H hookname
For a dryrun, use the command without options:
mkinitcpio
To list the content of a specified initramfs (here for kernel 5.10), use:
lsinitcpio /boot/initramfs-5.10-x86_64.img
or more verbose:
lsinitcpio -v /boot/initramfs-5.10-x86_64.img
Once you built your initramfs (using the mkinitcpio), and before reboot, you should most definitely analyze it using:
lsinitcpio -a /boot/initramfs-5.10-x86_64.img
Here the output for my initial ramdisk and kernel 5.10:
==> Image: /boot/initramfs-5.10-x86_64.img ==> Created with mkinitcpio 30 ==> Kernel: 5.10.52-1-MANJARO ==> Size: 17.34 MiB ==> Compressed with: gzip -> Uncompressed size: 45.81 MiB (.378 ratio) -> Estimated decompression time: 0.265s ==> Included modules: cdrom dm-mod hid-microsoft crc16 dm-persistent-data jbd2 crc32c_generic dm-region-hash libcrc32c crc32c-intel dm-snapshot mbcache crc-itu-t dm-thin-pool mmc_block dm-bio-prison ext4 mmc_core dm-bufio ff-memless sr_mod dm-cache firewire-core usbhid dm-cache-smq firewire-sbp2 usb-storage dm-log hid-logitech-dj virtio_blk dm-mirror hid-logitech-hidpp xhci-pci ==> Included binaries: blkid fsck.ext4 pdata_tools busybox kmod switch_root dmsetup lvm systemd-tmpfiles fsck mount udevadm ==> Early hook run order: udev ==> Hook run order: udev ==> Cleanup hook run order: udev
Hope this helped.