At sometime you may need to install a new Mainline or Upstream Linux kernel to test something. I recently stumpled across a bug with mouse cursor due to a kernel issue. So I needed to be able to install / remove kernels to test the fix.


Status: Mostly done and helpful if you need to do this. Could provide a utility to help with grub configuration.


Over the last 12 months I have done quite a bit of work on collectd and also moved from using VMWare to run MacOS VMs to QEMU/KVM.

Due to combination of bugs and feature needs this drove me off the LTS Ubuntu release path through a pretty regular sucession of releases: 16.04 (LTS) -> 17.10 (netplan bump) -> 18.04 (MacOS VMs, but collectd build bug) -> 18.10 (getting better) -> 19.04 (collectd build train issues) -> 19.10 (console cursor bug).

This is consistent with the CI/CD "alway's roll / fail forward" principle, though the time frames are well outside the ambitions of this.

The result was that I needed to do some testing against recent Linux kernels to verify the source of "mgag200" invisible cursor bug and whether an update fixed this.

The bug appeared in Linux kernel 5.3 (used by Ubuntu 19.10) but is fixed in Linux kernel 5.4 (target for Ubuntu 20.04 when released).


Using Ubuntu Mainline

For my first test I decided that best approach was to use install a 5.4 Kernel via Ubuntu Mainline ppa.

The documented approach to this is not very clear. Having never done standalone Linux kernel update before (as the kernel is just a small part of overall distribution), I looked for a tool to make it easy.

I found ukuu == Ubuntu Kernel Update Utility.

Installation of this was via build of a forked variation (see References & Links below)

This tool does a scan of the kernel version on your machine (installed) or available via mainline ppa:

$ ukuu --list
ukuu v18.9.3
Distribution: Ubuntu 19.10
Architecture: amd64
Running kernel: 5.3.0-26-generic
Kernel version: 5.3.0.26.28
Cache: /home/jbh/.cache/ukuu
Temp: /tmp/ukuu/WyIqzo20
aria2c version: 1.34.0
Fetching index from kernel.ubuntu.com...
OK

Fetching index...

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100 % 

----------------------------------------------------------------------
Found installed: 5.3.0-26.28
Found installed: 5.3.0.24.28
Found installed: 5.3.0-23.25
Found installed: 5.3.0-24.26
----------------------------------------------------------------------

======================================================================
Available Kernels
======================================================================
v5.4.13                        5.4.13                    
v5.4.12                        5.4.12                    
v5.4.11                        5.4.11                    
v5.4.10                        5.4.10                    
v5.4.9                         5.4.9                     
v5.4.8                         5.4.8                     
v5.4.7                         5.4.7                     
v5.4.6                         5.4.6                     
v5.4.5                         5.4.5                     
v5.4.4                         5.4.4                     
v5.4.3                         5.4.3                     
v5.4.2                         5.4.2                     
v5.4.1                         5.4.1                     
v5.4                           5.4                       
v5.3.18                        5.3.18                    
v5.3.17                        5.3.17                    
v5.3.16                        5.3.16                    
v5.3.15                        5.3.15                    
v5.3.14                        5.3.14                    
v5.3.13                        5.3.13                    
v5.3.12                        5.3.12                    
v5.3.11                        5.3.11                    
v5.3.10                        5.3.10                    
v5.3.9                         5.3.9                     
v5.3.8                         5.3.8                     
v5.3.7                         5.3.7                     
v5.3.6                         5.3.6                     
v5.3.5                         5.3.5                     
v5.3.4                         5.3.4                     
v5.3.3                         5.3.3                     
v5.3.2                         5.3.2                     
v5.3.1                         5.3.1                     
v5.3                           5.3                                         
...
...
...
v4.0.6-wily                    4.0.6                     
v4.0.5-wily                    4.0.5                     
v4.0.4-wily                    4.0.4                     
v4.0.3-wily                    4.0.3                     
v4.0.2-wily                    4.0.2                     
v4.0.1-wily                    4.0.1 

Having got the list of available kernels it is pretty trival to install a particular version:

$ ukuu --install 5.3.0-24.26

The next step is just a reboot..


Removing the Mainline Kernel

Having tested it is likely that you will will want to remove the kernel.

Even with ukuu installed this is a little more complicated as before you can remove the kernel you need to first get your machine to boot from a differnt kernel than the one you want to remove.

If you have physical access to the machine, this can be achieved by selecting a different Kernel from the grub boot menu, which is a simple menu selection.

If you do not have physical access then you need to change the default kernel that grub will boot.

This is messy as it is requires you to edit the grub defaults:

  • /etc/default/grub and
  • run update-grub to build the grub config
  • reboot
  • use ukuu to remove the installed test kernel
  • remove /etc/default/grub changes
  • run update-grub again
  • reboot

So pretty messy...

The worst part is that the grub defaults are set by using a concatenation of two uuid based identifiers. The details are described here.


What is really going on...

Having dones this once with ukuu and removal by editing of grub configurations and rebooting, next I had to do a test of some pre-release patched kernel builds...

As these were not Mainline but custom built kernels ukuu was no help.

The process is simpler then expected.

First the Ubuntu Linux kernels are delivered as a set of five Debian packages:

$ ls mgag200-fixed-kernel
linux-headers-5.3.0-26_5.3.0-26.28_all.deb
linux-headers-5.3.0-26-generic_5.3.0-26.28_amd64.deb
linux-image-unsigned-5.3.0-26-generic_5.3.0-26.28_amd64.deb
linux-modules-5.3.0-26-generic_5.3.0-26.28_amd64.deb
linux-modules-extra-5.3.0-26-generic_5.3.0-26.28_amd64.deb

So simply put the packages into a single directory and installation is a simple Debian package install:

$ sudo dpkg -i linux-*.deb

The Debian install does the update-grub as part of install and so you can just now reboot.

Removal is still messy... (as above).

Once the rebooted just use apt to remove the kernel:

$ sudo apt remove linux-image-unsigned-5.4.6-050406-generic

So in summary while ukuu is helpful to start it does not really help with the messy part of getting machine back booting with the grub default kernel configuation.

A utility to help with this would be more useful.

Sometime cleanup up is harder than building ;-)


References & Links:

Ubuntu Mainline Kernels - Ubuntu distribution provides kernel updates as part of is regular patch mechanism but there is also a CI/CD pipeline does regular pull of upstream kernels for testing and validation, this is the mainline, which has its own ppa (personal package archive) of Ubuntu / Debian packaged kernels for easy installation.

Change the grub default kernel - this post provides details on how to change the default kernel that is booted

Ubuntu Kernel Update Utility (ukuu) - a useful utility to install and manage upstream kernel installation. This is available as maintained paid software or via prior open source github DoY. There a number of forked versions of original. I used this free-ukuu variant successfully via git build and it seems this renamed "mainline" variant is getting some community traction.

"alway's roll / fail forward" - origin questionable (note 2015 blog date), principle strongly debated back in 2011 blogs... but it makes sense sometimes, which means that it is really what is preferred rather than alway's done.