August 2008 Archives

Damn, blast, etc.  Looks like a few days' worth of changes didn't get saved somehow.  In fact, I've got no evidence they were ever made at all.  I mean, I remember making them, but that doesn't really do me much good.

*sigh* Next book (if, not when) is going to stay in plain text until it hits the prepress dudes.

So Solaris 2008.05 integrates a version of Xen capable of 32-on-64-bit operation, which makes one section of our chapter on non-Linux systems obsolete.  (That being the section where we advise users to download the Xen binary distribution for a convenient 64-bit kernel image.)

Personally I'm glad of it.  One can see, very clearly, Xen making its way from a loose blob of bits to an actual product.  Eventually the struggles we had will be not worth mentioning, the sort of barbarism one doesn't bring up in polite company.

Or, as Luke is fond of pointing out, "Hey, Xen works now!  Let's play with KVM."

Pygrub will complain and exit if you try to use an unfamiliar terminal type.  The symptoms are that, rather than showing the familiar GRUB-ish menu, Pygrub will exit immediately with errors like these:

Using config file "/etc/xen/test".
Error opening terminal: rxvt-unicode.
No handlers could be found for logger "xend"
Error: Boot loader didn't return any data!
Usage: xm create <ConfigFile> [options] [vars]

And, of course, the domain won't start.

One solution is to symlink rxvt to /usr/share/terminfo/r/rxvt-unicode, thus:

ln -s rxvt /usr/share/terminfo/r/rxvt-unicode

Alternately you could just copy your own rxvt-unicode terminfo file to the remote machine, e. g.:

scp /usr/share/terminfo/r/rxvt-unicode user@remotehost:.terminfo/r/rxvt-unicode

New error message for the index:

Error: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')

The unformatted output is truly striking.

Anyway, this error occurred when I tried to load a kernel that wasn't Xen-aware -- I was just curious to see what the exact message would be.

Okay, this is actually pretty cool.  You can provision a few distros pretty easily via Pygrub -- here's the relevant config (bootloader and disk line) that I've been using lately:

bootloader = 'pygrub'
disk = [

'tap:aio:/opt/xen/opensuse-10.3.iso,xvdb:cdrom,r',
'tap:aio:/opt/xen/domain.img,xvda,w'

]

NOTE THAT THIS EXAMPLE IS ENTIRELY MADE UP!  OpenSUSE 10.3 does not support this install method, as far as I can tell.

You can see that I'm defining two devices, one a CDROM and the other an ordinary Xen disk.  Pygrub automatically mounts the ISO, pulls a kernel off of it, and boots normally -- and since the initrd tells it to go into install mode, that's what it does.

It even works with Solaris -- we just have to give the kernel and initrd explicitly:

disk = [

'tap:aio:/opt/xen/opensolaris-2008-05.iso,6:cdrom,r'
'tap:aio:/opt/xen/domain.img,0,w'

]
kernel = '/platform/i86xpv/kernel/unix'
ramdisk = 'boot/x86.microroot'

Note the Solaris-style device numbers.

Obviously this won't work with distros that don't ship a Xen-aware kernel on their installation media, but that's it's a nice trick for those that do.  (I think that's .  (Apparently this is one of the standard ways of doing things now!  I'm so excited!)

Have I mentioned I don't really like Ubuntu?  For some reason Xen's network-bridge script kills my networking, even though I'm connected just fine without.  First let's discuss how I fixed it (since I like to cut to the happy ending whenever possible).

What fixed it eventually was adding to /etc/network/interfaces:

iface eth0 inet dhcp

This let ifup handle eth0.  I'm not sure why the non-Xen kernel was able to deal with the absence of this line, or indeed why the Xen kernel wasn't.  Perhaps it is a red herring.

Then I restarted network-bridge (with sh -x) and noted that its output was a bit more sensible, but I still wasn't connected.  Eventually I noticed that I had two default routes, one via peth0 and one via eth0.  So I ran:

ip route del default dev peth0

And I'm connected.  Adding that last bit to a script, or hacking network-bridge's transfer_routes() function to figure out why it's leaving two routes and fixing it, would be pretty trivial.

Network-bridge, the default Xen network backend, doesn't care what sort of data it's sending -- as far as it's concerned, they're Ethernet frames.  There's a special case for IPv4 and the antispoof rules, but that's it.  Other protocols, like IPv6, will "just work," and there's no current provision for Xen to inspect packets.  (Although it wouldn't be that hard to add, building on the IPv4 support.)

I need to write a blurb about paravirt_ops and what that means for upstream Linux Xen support.  I don't think there are any administrator-visible changes (other than possibly better distro packaging,) but I'm not entirely certain.  (I also have no idea where it'd go, but that's another issue entirely.)

Part of it is just that any term thrown around on the mailing list so much has got to be important, right?  Maybe someone should do a concordance script to find common and significant terms in mailing lists.

I think Xen's PCI passthrough support has been dramatically updated since we wrote about it previously.  Accordingly, we too will update dramatically!

(A review: the PCI passthrough allows the administrator to forward a PCI device to a domU.  Once upon a time it was necessary to boot with the PCI device hidden from the dom0 -- this may no longer be the case.  Hard to say.)

Of course, if we had some VT-d hardware we could test that, too.  *sigh*

Our discussion of the "hotplug scripts not working" error makes me extremely unhappy.  Now that we're running into the bug, I really want to narrow down the cause and fix it.

Of course, this is "really want" in the sense of "except that I can't seem to concentrate on anything at all, ever.  It's very disheartening.

Even so, I've been working on debugging the problem.  Initially I thought that the hotplug subsystem wasn't working.  Some work with udevmonitor convinced me otherwise.  Now I'm leaning toward the idea that Xen isn't firing the right events for some reason.  The machine's been up for a while -- domain IDs are in the 300s -- so that's a plausible cause.

Also, what's with that device number?

Checked out the xen-3.2-testing.hg repo.  Was considering xen-unstable, but it's too unstable for me.  I wonder what the Xen uses openSSL and PAM for.  It's probably related to the remote management and XenAPI stuff -- but it's hard to say.

Working on compiling a kernel the "ubuntu way."  Just realized I need some kind of baseline, so I've also installed the Ubuntu Xen package.  Time to reboot.

I'm so over you, management frontends.

I mean, really.  None of this stuff works.  My feelings on convirt and enomalism are mostly unpublishable.  I'm going to shelve the entire problem and work on compiling Xen from source instead.  Maybe write about attacking the "Xen frontend" problem from the perspective of network monitoring software, which sounds like a dandy way to do it.

Instead, my plan for the weekend is to revisit my compilation instructions from "tips", save appropriate dumps to use as figures and such, and then work on the debugging instructions.  I'll also take an excursion to Ubuntu (which I dislike, but which also represents the desktop state-of-the-art.  Also we've been too RH-focused.)

Looks like they've finally got a version of the i810 driver that works with Xen.  I cannot tell you how glad I am to hear it.  (To tell the truth, they could have released it any time in the last year, and I wouldn't have known.  It's still welcome news, though.)

So.  Let's talk a bit about Enomalism.

As I've mentioned before, I think of the frontends chapter as a chance to show some of the possibilities of virtualization.  Unfortunately, the reality is that software development is hard, and all of the elaborate frontends that do exciting EC2-style stuff are extremely incomplete.  They have some rough edges, let us say.

Of the lot, Enomalism's one of the best.

Installing Enomalism2 on CentOS 5.2 is easy enough.  Download the RPM from their website, attempt to install it, then install its dependencies and try to install it again.  Repeat until finished.

Set up MySQL.  (This isn't the only SQL backend, but it ships with RedHat and is convenient.)

# /etc/init.d/mysqld start
# chkconfig mysqld on
# mysqladm password swordfish

Move to Enomalism2's install directory:

# cd /opt/enomalism2

Set it up.

# scripts init-db.sh swordfish enomal swordfish

(That first swordfish is the DB password specified previously, the second is the new password for the enomal user that this script creates.  They should be different.)

# cp default.cfg config/$(hostname).cfg

Edit config/$(hostname).cfg.  Comment all the sqlobject.dburi definitions except the notrans_mysql one, which you should edit with the appropriate name and password.  Set the baseurl and ip_addr.  Generate a UUID and put it in the correct place.

Build an appropriate initrd.  (Note that rather than configure Enomalism2 correctly, I'm just symlinking the kernel image somewhere expected.  This is incorrect.)

# ln -s /boot/vmlinuz-2.6.18-92.el5xen /boot/vmlinuz-2.6.18-xen
# mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk /boot/initrd-2.6.18-xen $(uname -r)

Start Enomalism.

# scripts/enomalism2.sh start

Connect via a web browser to port 8080 on localhost.  (That's the default, anyway.)  Click around.  You should be able to create machines fairly trivially.

Christ.  I'm really going to have to try to update this thing more often.  In my defence, I was vacationing with my mom and sister last week.  *shrug*

Anyway.  Fearless Leader L.'s moved to a Xen-related job at EA, and apparently his first tasks involve management frontends.  So that's what we'll be working on.

I also wrote up some stuff on the state of the art for migrating external devices.  Going to work on extending that as well.  (I'm still really peeved about not being able to get external-device-migrate to work.  I mean, it should be simple and obvious.)