Installing the Proprietary ATI Driver

"Proprietary drivers on Debian". That just sounds wrong.

Even so, it would be nice if AMD could spare 30 minutes to polish their proprietary Linux driver installation. Perhaps they're too busy resting on their Eyefinity laurels.

To start, we'll remove the open source drivers and clear any remnants.

# apt-get remove --purge fglrx*
# apt-get remove --purge xserver-xorg-video-ati xserver-xorg-video-radeon
# update-alternatives --remove-all x86_64-linux-gnu_gl_conf
# apt-get install --reinstall libgl1-mesa-glx:amd64 libgl1-mesa-dri:amd64 libgl1-mesa-glx:i386 libgl1-mesa-dri:i386 
# apt-get install --reinstall xserver-xorg-core

AMD offers two installation options: direct installation and generation of a distribution package. Opting for the former can result in fewer potential pitfalls initially.

If you opt to generate a distribution package, you can bypass the irritating graphical prompts by using --buildpkg. For example: 


# ./amd-driver-installer-12-3-x86.x86_64.run --buildpkg Ubuntu/oneiric

There are a few things to watch out for. For starters, ATI omits to tell you that you'll need various binaries for this to work, and if you don't have them, generation will failf. To save you some time trawling through the fglrx-install log file, this is what you'll need under Ubuntu or other Debian-based distros:

# apt-get install devscripts execstack dh-make dh-modaliases

If you tried to install the driver directly before trying to generate a distribution package, you may encounter errors like this:

objdump: debian/fglrx/usr/lib/fglrx/alt_ld.so.conf: File truncated
objdump: debian/fglrx/usr/lib/fglrx/ld.so.conf: File format not recognized
objdump: debian/fglrx/usr/lib/pxpress/alt_ld.so.conf: File truncated
objdump: debian/fglrx/usr/lib/pxpress/ld.so.conf: File format not recognized
   debian/rules override_dh_shlibdeps
make[1]: Entering directory `/tmp/fglrx.NszX9x'
dh_shlibdeps -l/tmp/fglrx.NszX9x/debian/fglrx/usr/lib/fglrx:/tmp/fglrx.NszX9x/debian/fglrx/usr/lib32/fglrx -Xlib32
dpkg-shlibdeps: warning: debian/fglrx/usr/lib/fglrx/libAMDXvBA.so.1.0 contains an unresolvable reference to symbol dlsym: it's probably a plugin.
dpkg-shlibdeps: warning: 21 other similar warnings have been skipped (use -v to see them all).
dpkg-shlibdeps: warning: debian/fglrx/usr/lib/fglrx/bin/atieventsd contains an unresolvable reference to symbol XauFileName: it's probably a plugin.
dpkg-shlibdeps: warning: debian/fglrx/usr/lib/fglrx/libGL.so.1.2 contains an unresolvable reference to symbol XOpenDisplay: it's probably a plugin.
dpkg-shlibdeps: warning: 31 other similar warnings have been skipped (use -v to see them all).
dpkg-shlibdeps: error: no dependency information found for /usr/share/ati/lib64/libQtCore.so.4 (used by debian/fglrx/usr/lib/fglrx/bin/amdnotifyui).

[...]

 debian/fglrx/usr/lib/fglrx/libaticalcl.so debian/fglrx/usr/lib/fglrx/libaticaldd.so returned exit code 2

make[1]: *** [override_dh_shlibdeps] Error 2

make[1]: Leaving directory `/tmp/fglrx.NszX9x'

make: *** [binary-arch] Error 2

dpkg-buildpackage: error: debian/rules binary gave error exit status 2

The solution which worked for me was to completely remove remnants of previous installations. I ran:

# sh /usr/share/ati/fglrx-uninstall.sh
# apt-get remove --purge fglrx* xserver-xorg-video-ati xserver-xorg-video-radeon
# shutdown -r now

Successful generation of packages will dump three .deb files (or .rpms for RH, etc) to the current directory.


Package /home/ace/Downloads/fglrx_8.951-0ubuntu1_amd64.deb has been successfully generated
Package /home/ace/Downloads/fglrx-dev_8.951-0ubuntu1_amd64.deb has been successfully generated
Package /home/ace/Downloads/fglrx-amdcccle_8.951-0ubuntu1_amd64.deb has been successfully generated


You then need to install them manually:

# apt-get install dkms
# dpkg -i fglrx*8.951*.deb


Load the module


To load it automatically, do the usual and add fglrx to /etc/modules. To load it manually, modprobe it, and verify it's loaded with lsmod. Please use the verbose flag with modprobe as shown below, because useful error messages are typically suppressed.


# modprobe -v fglrx
# lsmod | grep fglrx


If you get an error at this stage, you may find more details in dmesg.


# dmesg | tail -n 50

Xen Part 9: PCI Passthrough

One of my requirements was to expose a dedicated graphics card to the guest OS. Before you start this, make sure you have switched over to using xl instead of xm (or you may encounter some surprises with pci-list-assignable-devices).

This is how pci passthrough is going to work. The xen-pciback driver is loaded and bound to the dedicated graphics card. xen-pciback exposes the graphics card to Xen, and we instruct Xen to offer it to a guest OS.

Loading PCIBack

Debian Wheezy's 3.2.0-1 kernel includes xen-pciback as a module, which means we have to do a little work to load the module and to bind the graphics card to it. First, we load the pciback module:

# modprobe xen_pciback

# echo "xen_pciback" >> /etc/modules

Binding the Graphics Card

Now we have to bind the graphics card to the pciback driver. 

To do this, we have to know the pci ID of the card in BDF notation. We can find this easily via lspci:

# lspci | grep VGA

00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)

01:00.0 VGA compatible controller: ATI Technologies Inc Cayman XT [Radeon HD 6970]

In the above example, 01:00.0 is the ID of the graphics card I'm going to passthrough to the guest, and to convert this to BDF notation, all I need to do is add 0000: to the front. Now we have the card's BDF ID, we can bind it to xen-pciback. 

This binding code needs to be added to a startup script and should ideally run before the other xen init scripts. In the following example I have chosen to piggyback off of the /etc/init.d/xencommons init script, but you could just as easily create your own init script and use update-rc.d if you prefer.

# vim /etc/init.d/xencommons

Add this line at the top of the do_start() function:

init_passthrough

so it looks like this:

do_start () {

        init_passthrough

        echo do_start

        local time=0

        ...

Add this function in some clear space, e.g. after the end of the do_stop() function:

init_passthrough () {

        BDF=0000:01:00.0

        # Unbind a PCI function from its driver as necessary

        [ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \

                echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind

        # Add a new slot to the PCI Backend's list

        echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot

        # Now that the backend is watching for the slot, bind to it

        echo -n $BDF > /sys/bus/pci/drivers/pciback/bind

}

so it looks like this:

        echo WARNING: Not stopping xenstored, as it cannot be restarted.

}

init_passthrough () {

        BDF=0000:01:00.0

        ...

Don't forget to change the BDF= line so it uses your BDF ID.

Now that's done, reboot your machine, and check the card is showing as available for passthrough:

# xl pci-list-assignable-devices

0000:01:00.0

Hopefully you got a line of output which matches your BDF ID, like the above. If not, check the pciback module is loaded: 

# lsmod | grep pciback && echo loaded || echo "not loaded"

double-check it is actually working:

# [ -e /sys/bus/pci/drivers/pciback/ ] && echo working || echo "not working"

and check your graphics card is bound:

# BDF=0000:XX:XX.X    <- enter your BDF ID

# [ -e /sys/bus/pci/drivers/pciback/$BDF ] && echo bound || echo "not bound"

Cede the Device to the Guest

All that remains is to expose the device to the guest.

# xl create /etc/xen/ace2x1

# xl console ace2x1

root@ace2x1:~# lspci

root@ace2x1:~#

On dom0, issue the passthrough command with your BDF ID:

# xl pci-attach ace2x1 0000:01:00.0


See if dom0 thinks passthrough worked:

# cat /var/log/messages | grep pciback | tail -n 1

Mar 18 22:44:26 ace2 kernel: [ 3703.267343] xen-pciback: vpci: 0000:01:00.0: assign to virtual slot 0

# xl pci-list ace2x1
Vdev Device
00.0 0000:01:00.0

Back on the guest, verify that the device can be seen:

root@ace2x1:~# lspci

00:00.0 VGA compatible controller: ATI Technologies Inc Cayman XT [AMD Radeon HD 6900 Series]

root@ace2x1:~#


If you see errors like these in dmesg, as per this bug:

pci 0000:00:00.0: address space collision: [mem
0xfa8f8000-0xfa8fffff 64bit] conflicts with System RAM [mem
0x00100000-0xffffffff]

pcifront pci-0: Could not claim resource 0000:00:01.0/4! Device
offline. Try giving less than 4GB to domain.


Then adjust the RAM assigned to the domU to < 4GB. I haven't had time to work out a workaround for this just yet.

Finishing Off


Once you're happy that everything's working, set things up permanently by adding the BDF ID (this time without the 0000: prefix) to domU's config file.


# vim /etc/xen/ace2x1
pci = [ '01:00.0' ]


I recommend you shutdown the guest and start it again. Verify that your device shows under lspci and that the dmesg logs look healthy.


root@ace2x1:~# lspci
00:00.0 VGA compatible controller: ATI Technologies Inc Cayman XT [AMD Radeon HD 6900 Series]

root@ace2x1:~# dmesg | grep -E "(pci.*0000|pcifront)"
[    6.003747] pcifront pci-0: Installing PCI frontend
[    6.003861] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    6.004351] pci 0000:00:00.0: [1002:6718] type 0 class 0x000300
[    6.004577] pci 0000:00:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    6.004744] pci 0000:00:00.0: reg 18: [mem 0xfbe20000-0xfbe3ffff 64bit]
[    6.004860] pci 0000:00:00.0: reg 20: [io  0xe000-0xe0ff]
[    6.005522] pci 0000:00:00.0: supports D1 D2
[    6.007309] pcifront pci-0: claiming resource 0000:00:00.0/0
[    6.007313] pcifront pci-0: claiming resource 0000:00:00.0/2
[    6.007316] pcifront pci-0: claiming resource 0000:00:00.0/4

Next > Compiling a Custom DomU Kernel

Xen Part 8: Guest Installation (Take #2)

My first attempt to get a guest running under Debian Squeeze (see Part 6) failed in a mess of problems. The solution I adopted was to migrate to Debian Wheezy (see Part 7), which gave me a much newer kernel OOTB. In this section, I try the process of creating an Ubuntu Oneiric guest again - with far fewer problems and much more success than before.

XM vs XL

Xen upstream now uses the new 'xl' scripts for xen domain management. Debian still chooses to use the 'xm' scripts by default. They aren't as good, so we should switch over before proceeding.

# vim /etc/default/xen

TOOLSTACK=xl 

Either reboot now, or wait until you need to use the xl/xm command.


Guest Network Configuration

I'm going to use Xen's default network scripts for simplicity. 

Guest Installation Methods

There are different ways to install the guest system. Here's a few methods:

• Manually burn an installation CD, reboot the system and install it into the LV. Of course, no seasoned Linux user would consider a solution involving a reboot. Reboots are those things performed daily by users of Microsoft Windows.
• Manually install Debian with debootstrap, or CentOS with rpmstrap, etc.

Either of those manual methods would then require you to manually create a config file for the installation, and issue an xm create -f /path/to/config command.  This is unnecessarily laborious, so here's some simpler methods:

• Use a GUI: virt-manager is a very cool solution. It's based on libvirt, a virtualisation abstraction layer which can sit on top of either KVM or Xen. I played with this under Fedora. The problem was, I found it to be quite buggy. Most of the bugs can be worked around, but in some senses I was left wondering if it wasn't simpler to use the CLI in the first place
• Use the complementary CLI tool, virt-install. It's developed by the same team as virt-manager, and uses the same backend, so can be seen as virt-manager, with neither the GUI nor the bugs. 
• Use xen-create-image, part of xen-tools. This is what we'll be doing below.

First, make sure the prerequisites are installed:

# apt-get install xen-tools debootstrap

xen-create-image can be run with a mass of command line arguments, which define all the aspects of our domu. A more sensible approach is to amend its configuration file, which defines all the default parameters for the command. When we finally run the command, it will use the configuration file for its settings, and any parameters supplied as overrides. 

xen-create-image Configuration

# vim /etc/xen-tools/xen-tools.conf

1) Storage type. Tell the script that we're using LVM storage and provide the VG name:

lvm = xendomu

Note that EVMS is supported. I haven't tried this yet, but it sounds like a superb way of managing your storage. 

2) Installation method. The installation method section gives a number of options:

• debootstrap (for Debian systems)
• rpmstrap (for CentOS, Fedora etc.)
• rinse (an alternative for CentOS, Fedora etc.) 
• from an existing installation directory
• from an existing tar file

install-method = debootstrap

3) Disk. The tool is going to create a Logical Volume (LV) on the VG we specified of the specified size, i.e. the below will result in xen-create-image running a command like: lvcreate -L 50G -n <hostname>-disk xendomu.

size = 50Gb

The filesystem is also required; some people have encountered difficulty using ext4 for the guest, but since it represents such a large improvement over ext3, I'm willing to take the risk.

fs = ext4

A sparse image type is one which starts using no disk space, and grows its size as required. I'm not aware of any compelling reasons to opt for full allocation over sparse (aside from simpler disk space accounting). 

image = sparse

4) Memory. Choose how much RAM to allocate to the domu. This can be changed later. Note that swap space is assigned by creating a dedicated LV.

memory = 8Gb

swap = 4Gb

noswap = 0

5) Distribution. Pick the distribution to install. This field takes the name of the distribution version, e.g. Oneiric for Ubuntu 11.10 Oneiric Ocelot. Other examples are centos5 for CentOS 5 and squeeze for Debian 6 stable. You can find the distributions supported by your copy of xen-tools in the hook script directory: /usr/lib/xen-tools

dist = oneiric

6) Network. Pick your settings as appropriate for your network.

dhcp = 1

mac = 00:11:22:33:44:55 

7) Root password. By default, a generated root password is used: if, like me, you'd rather set your own at installation time, use the following option.

passwd = 1

8) Architecture. I doubt this is necessary, but just in case I'm setting this explicitly. If you don't have a 64-bit system & dom0 OS, ignore this.

arch = amd64

9) Kernel. The kernel image and corresponding initrd to use for the guest. We will be using the same kernel as dom0.

kernel = /boot/vmlinuz-`uname -r`

initrd = /boot/initrd.img-`uname -r`

10) Mirror. You might find you need to explicitly specify Ubuntu's mirror location like so:

mirror = http://gb.archive.ubuntu.com/ubuntu/

Create the Guest

# xen-create-image --hostname=ace2x1 --vcpus=7

Its configuration file will be placed under /etc/xen. Out of habit I still remove the .cfg from the end of its filename, since it made things easier when I was using xm...

# mv /etc/xen/ace2x1.cfg /etc/xen/ace2x1

Test the Guest

# xl create /etc/xen/ace2x1

Using config file "/etc/xen/ace2x1".
Started domain ace2x1 (id=1)

# xl list

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  7658     8     r----- 1389731.9
ace2x1                                       1  8192     7     -b----      2.1

So far so good. Time to connect to the guest (note that the only user we've created is root).

# xl console ace2x1

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Linux version 3.2.0-1-amd64 (Debian 3.2.6-1) (ben@decadent.org.uk) (gcc version 4.6.2 (Debian 4.6.2-14) ) #1 SMP Fri Feb 17 05:17:36 UTC 2012

[    0.000000] Command line: root=/dev/xvda2 ro ip=:127.0.255.255::::eth0:dhcp 

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] Released 0 pages of unused memory

[    0.000000] Set 0 page(s) to 1-1 mapping

[    0.000000] BIOS-provided physical RAM map:

[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)

[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)

[    0.000000]  Xen: 0000000000100000 - 0000000200800000 (usable)

[    0.000000] NX (Execute Disable) protection: active

... etc ...

Ubuntu 11.10 ace2x1 hvc0

ace2x1 login: root

Password: 

Welcome to Ubuntu 11.10 (GNU/Linux 3.2.0-1-amd64 x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by

applicable law.

root@ace2x1:~# ping google.com

PING google.com (173.194.41.69) 56(84) bytes of data.

64 bytes from lhr08s01-in-f5.1e100.net (173.194.41.69): icmp_req=1 ttl=57 time=24.7 ms

64 bytes from lhr08s01-in-f5.1e100.net (173.194.41.69): icmp_req=2 ttl=57 time=23.8 ms

^C

--- google.com ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1001ms

rtt min/avg/max/mdev = 23.887/24.314/24.742/0.455 ms

root@ace2x1:~#

We can log in and have network access, but we haven't got much of a set of applications yet (debootstrap installs a very barebones system). Let's get a few essentials.


root@ace2x1:~# apt-get update && apt-get dist-upgrade
root@ace2x1:~# apt-get install pciutils vim man


Now we can check the guest's view of the hardware exposed to it by Xen.


root@ace2x1:~# lspci -vv

Chances are, you see no output either. This is usual (apparently), unless you passthrough some devices to the guest. I'd be interested to hear if your experience is different.

Finally, we should create a non-root user.

root@ace2x1:~# adduser <username>

Next > PCI Passthrough

Xen Part 7: The Move to Wheezy

The last time I tried installing guests on Xen, it didn't really pan out. I ended up with more problems than I could reasonably expect to fix in my spare time, and a lot of them were related to Debian. Yes, you heard right: problems related to Debian. For whilst Debian is a superbly stable distro (and, it just so happens, my favourite), it achieves its trademark stability through relatively lengthy testing periods. In other words, if you're running Debian squeeze, you're running relatively old software - and pertinent to this case - a relatively old kernel.

No, scrap that; it's not relatively old - it's ancient. Kernel 2.6.32 was released in December 2009, some two-and-a-quarter years ago. A few of points to note:

1. A lot of Xen changes have been made since 2.6.32.
2. Ubuntu Oneiric (which I'm trying to install as a guest) currently boasts kernel 3.0.0, released in July 2011. I'm rather lazily getting my dom0 kernel to double as my guest kernel, and I don't know why I expected 2.6.32 (2009) to run smoothly with Oneiric (2011-2012).
3. The rather annoying DPMS problem I encountered (which you may recall from Part 5: Dom0 X Instability) was resolved in one of the kernel updates. 

Clearly, I needed a more recent kernel.

To understand the options available, we first need to understand the Debian distribution structure. If you're running Debian and don't know otherwise, you'll be running Debian stable (Toy Story codename: Squeeze). There are a couple of other distributions you could be running: Debian unstable and Debian testing. 

A new package version is first uploaded to unstable, where most of the big problems are ironed out. When various criteria are reached, the version is promoted to the testing distribution. This process of uploading to unstable and subsequent promotion to testing continues, until the number of bugs in testing reduces to acceptable levels. Then the release manager will promote testing to stable, and a new Debian release is announced to an expectant world.

If you want to run a more recent package version than is available in stable, you have a few options. You can:

• Compile the package version of your choice, and handle any dependencies yourself
• Run a mixed stable and unstable/testing distribution
• Run an unstable/testing distribution

So, I tried the first option, compiled the latest kernel from kernel.org, spun it up under squeeze... and ran headfirst into a stone wall of problems. I also tried a few other kernels back down to 3.0, and encountered the same sorts of issues. 

That was when I decided that enough was enough: I would be moving my dom0 to Debian testing (Toy Story codename: Wheezy).

Should you also choose to do so, I ask that you to do your research first. Testing does not bear the "stable" moniker for a reason. You're likely to encounter some problems before you've even got it fully installed. Security updates aren't rolled out to testing as promptly as they are to stable. Finally, I don't know of any reasonable way to revert back to stable if you decide that the experience is proving too problematic.

That said, moving to Wheezy is elementary. First, you need to edit your /etc/apt/sources.list file to point to the Wheezy software repositories - something like this:

# Wheezy (Testing)

deb http://security.debian.org/ wheezy/updates main non-free contrib

deb-src http://security.debian.org/ wheezy/updates main non-free contrib

deb http://ftp.uk.debian.org/debian/ wheezy main non-free contrib

deb-src http://ftp.uk.debian.org/debian/ wheezy main non-free contrib

deb http://www.debian-multimedia.org/ wheezy main non-free

Then you need to upgrade your packages to the latest in Wheezy. You'll want to use dist-upgrade for this, to allow new packages to be installed and old packages to be removed (after this process, I had to re-install a number of my packages).

# apt-get update && apt-get dist-upgrade

And if that runs smoothly without a hiccup, you just got very lucky. Be ready with apt-get -f install, and be prepared to spend some time fixing some dependency breaks. You may have to download some packages and install them manually with dpkg -i --force-overwrite and such.

Note that there are better and worse times to switch to Wheezy (or unstable, Toy Story codename: Sid). You can get a feel for weather* now is a good time at Debian's weather page.

* Awful pun intended

Update 24th Oct 2012:

It's also worth pointing out a handy pre-upgrade step:

# apt-get -s dist-upgrade 

This runs a simulation of the upgrade without actually upgrading anything (it can be invaluable to know if there are any terrible dependency breaks before you actually commit to the dist-upgrade). 

Next > Guest Installation (Take #2)

Raspberry Pi - It's a Date!

A week ago, Farnell furnished me with an estimated delivery date for my Raspberry Pi model B. I'm not going to be one of the first, but at least I know I have a place reserved in the queue.

As I've written elsewhere, the Raspberry Pi's release day on the 29th February was a bit of a shambolic affair. The recent announcement of a manufacturing blip, which will add a few days to the wait, hasn't exactly helped either. However, from speaking to others waiting for their Pi, I get a sense that people have shifted from worrying about getting their Pi, to planning and scheming about projects to start when it arrives.

Of course, you don't need to wait. There are many sets of instructions for how to set up a development environment for the Pi. I haven't found the time yet, but clearly many others have.

The software options for the Pi are already pretty broad. So far, the list of supported OSs stands at Debian, Fedora, Arch and OpenELEC (an OS designed specifically to run the XBMC media centre). I'm expecting many more distros to provide ISOs for the Pi in due course.

Before you pick an OS though, you need to know what you intend to use it for (it all boils down to not trying to hammer a nail with a screwdriver). What should you build first? When will the Gertboards be available (or should you build one yourself)? How many Pis will you ultimately want/need? When will you get the time to put all these plans into action? And just what sort of Pandora's box has the Raspberry Pi team opened here?

It's an exciting device with oodles of potential, but the most exciting thing of all will be seeing all the wonderfully inventive contraptions everybody else builds with theirs.