Chapter 3: Provisioning domUs

From PrgmrWiki
Revision as of 04:41, 19 June 2010 by Nyck (talk | contribs) (NOTE)
You can suck Linux right out of the air, as it were, by downloading the right files and
putting them in the right places, but there probably are not more than a few hundred
people in the world who could create a functioning Linux system in that way.
—Neal Stephenson, In the Beginning Was the Command Line

Up until now, we’ve focused on administering the dom0, leaving the specifics of domU creation up to the virt-install tool. However, you’ll probably need to build a domU image from scratch on occasion. There are plenty of good reasons for this—perhaps you want an absolutely minimal Linux environment to use as a base for virtual private server (VPS) hosting setups. Maybe you’re deploying some custom application—a server appliance— using Xen. It might just seem like a good way to keep systems patched. Possibly you need to create Xen instances without the benefit of a network connection.

Just as there are many reasons to want custom filesystem images, there are many ways to make the images. We’ll give detailed instructions for some that we use frequently, and briefly mention some others, but it would be impossible to provide an exhaustive list (and very boring besides). The goal of this chapter is to give you an idea of the range of options you have in provisioning domU filesystems, a working knowledge of the principles, and just enough step-by-step instruction to get familiar with the processes.

A Basic DomU Configuration

All of the examples that we’re presenting here should work with a basic—in fact, downright skeletal—domU config file. Something along the lines of this should work:

kernel = /boot/vmlinuz-2.6-xen.gz
vif = ['']
disk = ['phy:/dev/targetvg/lv,sda,w']

This specifies a kernel, a network interface, and a disk, and lets Xen use defaults for everything else. Tailor the variables, such as volume group and kernel name, to your site. As we mention elsewhere, we recommend including other variables, such as a MAC and IP address, but we’ll omit them during this chapter for clarity so we can focus on creating domU images.


This doesn’t include a ramdisk. Either add a ramdisk= line or include xenblk (and

xennet if you plan on accessing the network before modules are available) in your kernel. When we compile our own kernels, we usually include the xenblk and xennet drivers directly in the kernel. We only use a ramdisk to satisfy the requirements of the distro


If you’re using a modular kernel, which is very likely, you’ll also need to ensure that the kernel has a matching set of modules that it can load from the domU filesystem. If you’re booting the domU using the same kernel as the dom0, you can copy over the modules like this (if the domU image is mounted on /mnt):

# mkdir -p /mnt/lib/modules
# cp -a /lib/modules/`uname -r` /mnt

Note that this command only works if the domU kernel is the same as the dom0 kernel! Some install procedures will install the correct modules automatically; others won’t. No matter how you create the domU, remember that modules need to be accessible from the domU, even if the kernel lives in the dom0. If you have trouble, make sure that the kernel and module versions match, either by booting from a different kernel or copying in different modules.

Selecting a Kernel

Traditionally, one boots a domU image using a kernel stored in the dom0 filesystem, as in the sample config file in the last section. In this case, it’s common to use the same kernel for domUs and the dom0. However, this can lead to trouble—one distro’s kernels may be too specialized to work properly with another distro. We recommend either using the proper distro kernel, copying it into the dom0 filesystem so the domain builder can find it, or compiling your own generic kernel.

Another possible choice is to download Xen’s binary distribution, which includes precompiled domU kernels, and extracting an appropriate domU kernel from that.

Alternatively (and this is the option that we usually use when dealing with distros that ship Xen-aware kernels), you can bypass the entire problem of kernel selection and use PyGRUB to boot the distro’s own kernel from within the domU filesystem. For more details on PyGRUB, see Chapter 7. PyGRUB also makes it more intuitive to match modules to kernels by keeping both the domU kernel and its corresponding modules in the domU.

Quick-and-Dirty Install via tar

Let’s start by considering the most basic install method possible, just to get an idea of the principles involved. We’ll generate a root filesystem by copying files out of the dom0 (or an entirely separate physical machine) and into the domU. This approach copies out a filesystem known to work, requires no special tools, and is easy to debug. However, it’s also likely to pollute the domU with a lot of unnecessary stuff from the source system and is kind of a lot of work.

A good set of commands for this “cowboy” approach might be:

# xm block-attach 0 duncan.img /dev/xvda1 w 0
# mke2fs -j /dev/xvda1
# mount /dev/xvda1 /mnt
# cd /
# tar -c -f - --exclude /home --exclude /mnt --exclude /tmp --exclude \
  /proc --exclude /sys --exclude /var | ( cd /mnt/ ; tar xf - )
# mkdir /mnt/sys
# mkdir /mnt/proc

Do all this as root.

These commands, in order, map the backing file to a virtual device in the dom0, create a filesystem on that device, mount the filesystem, and tar up the dom0 root directory while omitting /home, /mnt, /tmp, /proc, /sys, and /var. The output from this tar command then goes to a complementary tar used to extract the file in /mnt. Finally, we make some directories that the domU will need after it boots. At the end of this process, we have a self-contained domU in duncan.img.

Why This Is Not the Best Idea

The biggest problem with the cowboy approach, apart from its basic inelegance, is that it copies a lot of unnecessary stuff with no easy way to clear it out. When the domU is booted, you could use the package manager to remove things or just delete files by hand. But that’s work, and we are all about avoiding work.

Stuff to Watch Out For

There are some things to note:

  • You must mkdir /sys and /proc or else things will not work properly.
The issue here is that the Linux startup process uses /sys and /proc to discover and configure hardware—if, say, /proc/mounts doesn’t exist, the boot scripts will become extremely annoyed.
  • You may need to mknod /dev/xvda b 220 0.
/dev/xvd is the standard name for Xen virtual disks, by analogy with the hd and sd device nodes. The first virtual disk is /dev/xvda, which can be partitioned into /dev/xvda1, and so on. The command
# /mknod /dev/xvda b 220 0
creates the node /dev/xvda as a block device (b) with major number 220 (the number reserved for Xen VBDs) and minor number 0 (because it’s xvda—the first such device in the system).

On most modern Linux systems, udev makes this unnecessary.

  • You may need to edit /etc/inittab and /etc/securettys so that /dev/xvc0 works as the console and has a proper getty.
We’ve noticed this problem only with Red Hat’s kernels: for regular XenSource kernels (at least through 3.1) the default getty on tty0 should work without further action on your part. If it doesn’t, read on!
The term console is something of a holdover from the days of giant time-sharing machines, when the system operator sat at a dedicated terminal called the system console. Nowadays, the console is a device that receives system administration messages—usually a graphics device, sometimes a serial console.
In the Xen case, all output goes to the Xen virtual console, xvc0. The xm console command attaches to this device with help from xenconsoled. To log in to it, Xen’s virtual console must be added to /etc/inittab so that init knows to attach a getty.1 Do this by adding a line like the following:
xvc:2345:respawn:/sbin/agetty -L xvc0
(As with all examples in books, don’t take this construction too literally! If you have a differently named getty binary, for example, you will definitely want to use that instead.)
You might also, depending on your policy regarding root logins, want to add /dev/xvc0 to /etc/securetty so that root will be able to log in on it. Simply append a line containing the device name, xvc0, to the file.

Using the Package Management System with an Alternate Root

Red Hat, CentOS, and Other RPM-Based Distros

Debootstrap with Debian and Ubuntu

QEMU Install

virt-install—Red Hat’s One-Step DomU Installer

Converting VMware Disk Images

Mass Deployment

Manual Deployment

===QEMU and Your Existing Infrastructure === ===Installing pypxeboot === ===Automated Installs the Red Hat Way ===

And Then . . .