Sunday, 18 March 2012

Xen Part 8: Guest Installation (Take #2)

My first attempt to get a guest running under Debian Squeeze (see Part 6) failed in a mess of problems. The solution I adopted was to migrate to Debian Wheezy (see Part 7), which gave me a much newer kernel OOTB. In this section, I try the process of creating an Ubuntu Oneiric guest again - with far fewer problems and much more success than before.

XM vs XL

Xen upstream now uses the new 'xl' scripts for xen domain management. Debian still chooses to use the 'xm' scripts by default. They aren't as good, so we should switch over before proceeding.

# vim /etc/default/xen


Either reboot now, or wait until you need to use the xl/xm command.

Guest Network Configuration

I'm going to use Xen's default network scripts for simplicity. 

Guest Installation Methods

There are different ways to install the guest system. Here's a few methods:
  • Manually burn an installation CD, reboot the system and install it into the LV. Of course, no seasoned Linux user would consider a solution involving a reboot. Reboots are those things performed daily by users of Microsoft Windows.
  • Manually install Debian with debootstrap, or CentOS with rpmstrap, etc.
Either of those manual methods would then require you to manually create a config file for the installation, and issue an xm create -f /path/to/config command.  This is unnecessarily laborious, so here's some simpler methods:
  • Use a GUI: virt-manager is a very cool solution. It's based on libvirt, a virtualisation abstraction layer which can sit on top of either KVM or Xen. I played with this under Fedora. The problem was, I found it to be quite buggy. Most of the bugs can be worked around, but in some senses I was left wondering if it wasn't simpler to use the CLI in the first place
  • Use the complementary CLI tool, virt-install. It's developed by the same team as virt-manager, and uses the same backend, so can be seen as virt-manager, with neither the GUI nor the bugs. 
  • Use xen-create-image, part of xen-tools. This is what we'll be doing below.

First, make sure the prerequisites are installed:

# apt-get install xen-tools debootstrap

xen-create-image can be run with a mass of command line arguments, which define all the aspects of our domu. A more sensible approach is to amend its configuration file, which defines all the default parameters for the command. When we finally run the command, it will use the configuration file for its settings, and any parameters supplied as overrides. 

xen-create-image Configuration

# vim /etc/xen-tools/xen-tools.conf

1) Storage type. Tell the script that we're using LVM storage and provide the VG name:

lvm = xendomu

Note that EVMS is supported. I haven't tried this yet, but it sounds like a superb way of managing your storage. 

2) Installation method. The installation method section gives a number of options:
  • debootstrap (for Debian systems)
  • rpmstrap (for CentOS, Fedora etc.)
  • rinse (an alternative for CentOS, Fedora etc.) 
  • from an existing installation directory
  • from an existing tar file
install-method = debootstrap

3) Disk. The tool is going to create a Logical Volume (LV) on the VG we specified of the specified size, i.e. the below will result in xen-create-image running a command like: lvcreate -L 50G -n <hostname>-disk xendomu.

size = 50Gb

The filesystem is also required; some people have encountered difficulty using ext4 for the guest, but since it represents such a large improvement over ext3, I'm willing to take the risk.

fs = ext4

A sparse image type is one which starts using no disk space, and grows its size as required. I'm not aware of any compelling reasons to opt for full allocation over sparse (aside from simpler disk space accounting). 

image = sparse

4) Memory. Choose how much RAM to allocate to the domu. This can be changed later. Note that swap space is assigned by creating a dedicated LV.

memory = 8Gb
swap = 4Gb
noswap = 0

5) Distribution. Pick the distribution to install. This field takes the name of the distribution version, e.g. Oneiric for Ubuntu 11.10 Oneiric Ocelot. Other examples are centos5 for CentOS 5 and squeeze for Debian 6 stable. You can find the distributions supported by your copy of xen-tools in the hook script directory: /usr/lib/xen-tools

dist = oneiric

6) Network. Pick your settings as appropriate for your network.

dhcp = 1
mac = 00:11:22:33:44:55 

7) Root password. By default, a generated root password is used: if, like me, you'd rather set your own at installation time, use the following option.

passwd = 1

8) Architecture. I doubt this is necessary, but just in case I'm setting this explicitly. If you don't have a 64-bit system & dom0 OS, ignore this.

arch = amd64

9) Kernel. The kernel image and corresponding initrd to use for the guest. We will be using the same kernel as dom0.

kernel = /boot/vmlinuz-`uname -r`
initrd = /boot/initrd.img-`uname -r`

10) Mirror. You might find you need to explicitly specify Ubuntu's mirror location like so:

mirror =

Create the Guest

# xen-create-image --hostname=ace2x1 --vcpus=7

Its configuration file will be placed under /etc/xen. Out of habit I still remove the .cfg from the end of its filename, since it made things easier when I was using xm...

# mv /etc/xen/ace2x1.cfg /etc/xen/ace2x1

Test the Guest

# xl create /etc/xen/ace2x1

Using config file "/etc/xen/ace2x1".
Started domain ace2x1 (id=1)

# xl list

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  7658     8     r----- 1389731.9
ace2x1                                       1  8192     7     -b----      2.1

So far so good. Time to connect to the guest (note that the only user we've created is root).

# xl console ace2x1

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.0-1-amd64 (Debian 3.2.6-1) ( (gcc version 4.6.2 (Debian 4.6.2-14) ) #1 SMP Fri Feb 17 05:17:36 UTC 2012
[    0.000000] Command line: root=/dev/xvda2 ro ip=: 
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Released 0 pages of unused memory
[    0.000000] Set 0 page(s) to 1-1 mapping
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000200800000 (usable)
[    0.000000] NX (Execute Disable) protection: active

... etc ...

Ubuntu 11.10 ace2x1 hvc0

ace2x1 login: root
Welcome to Ubuntu 11.10 (GNU/Linux 3.2.0-1-amd64 x86_64)

 * Documentation:

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@ace2x1:~# ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_req=1 ttl=57 time=24.7 ms
64 bytes from ( icmp_req=2 ttl=57 time=23.8 ms
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 23.887/24.314/24.742/0.455 ms

We can log in and have network access, but we haven't got much of a set of applications yet (debootstrap installs a very barebones system). Let's get a few essentials.

root@ace2x1:~# apt-get update && apt-get dist-upgrade
root@ace2x1:~# apt-get install pciutils vim man

Now we can check the guest's view of the hardware exposed to it by Xen.

root@ace2x1:~# lspci -vv

Chances are, you see no output either. This is usual (apparently), unless you passthrough some devices to the guest. I'd be interested to hear if your experience is different.

Finally, we should create a non-root user.

root@ace2x1:~# adduser <username>