Xen Part 10: Compiling a Custom DomU Kernel

I upgraded the kernel on the Debian dom0, and switched domU to the newer kernel. Suddenly things started to break, including PCI passthrough.

Xen uses PV kernels residing on the dom0, but the modules obviously need to reside on the domU. In this case, it looks like module compatibility broke when I upgraded the kernel for domU without replacing its modules.

In my opinion, this separation between the kernel on dom0 and the modules on domU is rather odd and inconvenient. It seems like one should mount the domU partition in ro mode and load the kernel from there; in that way, the domU would be in near-complete control of its own kernel.

However, I've never seen a method like that mentioned. The other obvious workaround is to compile a domU kernel without any modules, but that sounded like a terrible idea. So I figured I'd try a compile-on-dom0 solution first.

Remember, do not use root for kernel compilation. The example below is for 3.2.13.

Get the kernel sources for the version you want from kernel.org.

$ VERSION=3.2.13
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-$VERSION.tar.bz2
$ tar xjf linux-$VERSION.tar.bz2
$ cd linux-$VERSION

Here I'm basing the Ubuntu domU kernel config on my Debian Wheezy dom0 kernel config; whether that's wise is another question.

$ make oldconfig
Unless you have special requirements, just hold down enter to accept the defaults
$ make menuconfig
Make any customisations you require
$ vim .config
Further customisations; e.g., search for XEN and check everything you require is enabled ('y' or 'm')
$ make -j 10 
$ make -j 10 modules
# VERSION=3.2.13
# cp arch/x86_64/boot/bzImage /boot/bzImage-$VERSION
# make modules_install
# update-initramfs -c -k $VERSION 

Mount your domU partition/image (you'd be well advised to stop your domU first, otherwise... well, you can figure it out)

# xen shutdown ace2x1
# mount /dev/xendomu/ace2x1-disk /mnt

Copy the modules and source to domU. The second command in particular may take a while.

# make INSTALL_MOD_PATH=/mnt modules_install
# cp -ru . /mnt/usr/src/linux-$VERSION
# cp .config /mnt/boot/config-$VERSION
# umount /mnt

Use the compiled kernel for the domU

# vim /etc/xen/ace2x1
Set the kernel and ramdisk variables to /boot/bzImage-3.2.13 and /boot/initrd.img-3.2.13 respectively 

Back on the domU, just finish tidying up after the hacks.

# xen create /etc/xen/ace2x1
# xen console ace2x1 / $ ssh root@ace2x1
root@ace2x1# cd /lib/modules/3.2.13
root@ace2x1# rm build source
root@ace2x1# ln -s /usr/src/linux-3.2.13 build

root@ace2x1# ln -s /usr/src/linux-3.2.13 source


Next > ATi Graphics drivers on the domU