xenophobia

beyond this horizon.

Fri, 05 Mar 2010 02:17:05 -0800

We're working on a virtualization API and web management tool, just like everyone else in the world. What will differentiate ours is that it will something we want to use, and it will display our. . . idiosyncratic approach to UI, authentication, and features.

Actually I'm pretty enthused. We're starting from the rackspace API and throwing in Luke's coercive provisioning ideas, and it's kind of turning into something interesting. (But hungry. Always hungry.) I've thrown myself wholeheartedly into the project, as I tend to do when confronted with the prospect of great effort for uncertain reward.

Yeah, I don't know why that is either. I should probably get that checked.

anticipate.

Tue, 01 Dec 2009 03:34:11 -0800

Okay, I guess that was an adequate break. I worked through some of the book examples with my dad, and man -- I was rusty. It was pretty embarrassing, although it all seemed to come back pretty quickly.

Anyway, though, we've hit a bunch of errata. Mostly little things. Some absolute deal-breakers. There are a couple of bits that we need to expand our explanations of. Some sections where I really just want to say "skip ahead and skim the next two chapters, then come back and this'll make more sense." Not sure how to deal with those.

I guess these things are never done.

automation is the cloud

Tue, 06 Oct 2009 14:51:49 -0800

So, Brandon Burton linked to his Automation is the cloud article on lopsa-discuss mailing list today, and he requested that I post my response here, so those of you in blog-land can see it.

First, I very much like your 'automation is the cloud' thesis. I think that is the only sane way to define 'the cloud'. (of course, I think leaving 'the cloud' to the marketing department might be an even better choice, then going back to automating our stuff.)

I am very interested in this, because I'm trying to pull the good parts of 'the cloud' and use them in my own service.

Right now, my intent is to build something that emulates the existing pxeboot/rebooter setup we have with physical boxes... something that can suck in a dhcp.conf created by cobbler or the like and do the correct thing. This will make it easier to integrate with existing tools so that you can mange the servers you rent using the same tools as the servers you own.

Most of the 'API' stuff looks silly to me. You want Python bindings to provision new servers? really? but then, I'm a janitor, not a developer, so maybe it's just that much easier for them to use python than to set up ssh keys and a script?

Personally, I believe that if the cloud is going to be anything more than a super-high-margin playground for those who don't care about money or performance, we need to decouple virtualization from the cloud.

live migration (and failover) both require shared storage. (failover requires doubling your other compute resources as well) which usually makes it a no-go when both cost and performance/reliability matter. you can get good fast shared storage, but it's not cheap, etc... (amazon seems to have stood smack dab in the middle of the good fast cheap triangle with it's elastic block storage project... It's not super reliable, but it's not horrible. It's not super expensive, but it's not horrible. It's not that fast, but it's usable. I think they probably made reasonable choices with what they had. I also think their decision to go with local disk (thus you can't use live migration) was probably a good one, considering the choices.)

Without those things, virtualization becomes nothing more than a tool to turn big servers into many little servers. Which is great if you need little servers. It is way cheaper to run one 32GiB ram/8 core box than to run 8 4GiB/ 1 core boxes, let me tell you.

this means that if you do need a 32GiB/8 core box, you lose out by virtualizing. Without shared storage, pxeboot and rebooting power strips give you almost everything virtualization gives you in terms of automation. Virtualization also has a cost in terms of security (remember the hyperthreading cache-peeking vulnerability?) having a box all to yourself will always be more secure than sharing one. With virtualization, the amount of cpu/disk bandwidth available is either unknown, or split vary hard (like the above example where I've dedicated a core to each virtual. which is exagerating a bit... you need to dedicate a core to the control server, too, if you want reasonable performance.)

My next project is to set up prgmr.com so that customers can upload dhcpd.conf files or similar from cobbbler and my system can kickstart the DomUs they own, for a more seamless transition between servers they own, and virtual and physical servers they have with me.

That's the other problem, owning is much cheaper than renting. When I say this, most people point at the 'sysadmin time' thing, which is expensive, but the only part of sysadmin time that 'the cloud' covers is hardware installation and replacement, and setup of your provisioning system. you still need a sysadmin for troubleshooting (is it hardware vs. is it my OS) though the provisioning system makes that a little easier, as you can just move your junk to new hardware. You still need to handle configuration management. (EC2 has some basic tools, but realistically you still need someone on hand who knows puppet, chief, cfgengine or who really knows your OS and can code up some fancy perl scripts.) what I'm saying is that the cloud only saves you from schleping hardware. You still need a sysadmin.

But what that means is that you need a system that can handle servers you rent (for short-term stuff, it's reasonable to pay a lot extra to rent if you only need the servers for a few days.) as well as servers you own, in a seamless manner.

project plowshare.

Wed, 02 Sep 2009 21:34:29 -0800

Wow, it's been a while since the last time I wrote here. I'm not apologizing or anything -- we've got good reasons for that. Ish. Good-ish. Anyway.

First, most exciting, the book is done. That's writing, tech review, copyedit, layout, and proofreading, with the attendant reviews by us at each step. It looks increasingly real, and it's actually pretty scary. Look for it in stores this year. Ask for it by name. Any customers who buy it shall have such thanks as befit a king's remembrance.

Second, prgmr's been growing at a pretty intense clip. That's not really an excuse, but we're pretty happy with how things have been going, so it bears mentioning.

Finally, I've been taking a break from virtualization. I spent several months in there saying "I want this book to be somebody else's problem," and so I basically headed off for a break as soon as humanly possible. In spirit at least, if not in body. But I think I'm over that now, and we've got a lot more to work on here.

More on future plans later.

Wed, 13 May 2009 23:19:03 -0800

Writing quote of the day:

"i mean, we shouldn't slavishly avoid double negatives when they make our meaning less unclear."

Take that, English language.

start a getty on the cross-platform hypervisor console.

Sat, 09 May 2009 17:28:52 -0800

The sole problem that I had setting up a pvops domU on a pvops dom0 (mostly following the instructions at http://bderzhavets.wordpress.com/2009/03/29/setup-xen-unstable-dom0-with-2629-tip-pvops-enabled-kernel/ ) was in trying to get the Ubuntu domU's console to work. Eventually I solved it by adding an 'hvc0' file to the domU's /etc/event.d/ directory, like this:


    # cd /etc/event.d

    # cp tty1 hvc0

    # sed -i -e "s/tty1/xvc0/g" hvc0

(Credit to Mr. bderzhavets, again. There's a lot of good and useful information there.)

I also included an "extra='xencons=tty'" in the domU config. (I believe. Might have been hvc. Try it and see.)

onward, not forward.

Tue, 28 Apr 2009 04:09:25 -0800

Ah, progress! Thankfully, this problem seems to be fixed in both Solaris Express and OpenSolaris, so I guess we can take out this section. I'm pretty happy about that, yeah.

Devices and Hotplug

Solaris' hotplug system may give you some trouble. The problem is that the default hotplug configuration (in 2008.05) doesn't react correctly to the Xen virtual devices. This causes domain creation to fail with an error message like "Device 0 (vif) could not be connected. Hotplug scripts not working."

To fix the hotplug scripts, create and run a script like the following, which adds appropriate rules for Xen devices:¹

BASEDIR=${BASEDIR:-/}

/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev > /dev/null 2>&1
if [ $? -ne 0 ]

then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \
'domain=$domain' 'vdev=$vdev' 'device=$device' \
'devclass=$devclass' 'febe=$fob'
fi

/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys > /dev/null 2>&1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \
'shutdown=$shutdown'
fi

# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
then
/usr/sbin/syseventadm restart

1 Script from http://blogs.sun.com/dkumar/entry/problem_bring_up_domu

purely of historical interest.

Sun, 22 Mar 2009 06:34:58 -0800

Some days it seems like the caches have churned just right, and El Goog is giving me more useful information than I know what to do with. . . circa two years ago. This is that day. I know I've used these search strings before, but some of these links are positively delicious.

Anyway, here's an interesting post on using Perl to communicate with Xen via the XenAPI.

http://unixfoo.blogspot.com/2007/12/perl-xen-api.html

virt-manager hang after "mounting /sys filesystem"

Sun, 22 Mar 2009 00:21:05 -0800

So, when installing CentOS 5.2 (and possibly other distros) with virt-manager, console output disappears after the "mounting /sys filesystem" line. This might only happen in low-memory conditions. (The fact that anything less than 512 MB is considered low-memory for an _installer_, by the way, I find appalling.)

To work around this issue, install using the extra kernel argument "xencons=tty" (no quotes.)

on xen vanilla kernels

Sun, 08 Mar 2009 21:21:41 -0800

Interesting thread on merging Xen dom0 support into the kernel:

http://thread.gmane.org/gmane.linux.kernel/800658/focus=800714

Obviously I'm biased toward moving Xen completely in-kernel -- it's ridiculous to rely on out-of-kernel patches that haven't been updated to work with newer kernels -- but I don't have the deep familiarity needed to give reasons why this is bad off the top of my head. Jeremy Fitzhardinge does, and it's quite enlightening.

the very most basic.

Sun, 08 Mar 2009 20:06:45 -0800

One of the basic things that I think we forget to mention in an earlier chapter is how to get domains to start automatically on boot. It can't stay where it is now, so I'm dumping it out here and tagging it as an orphan until we've got the chapter in shape.

By default, Xen ships with a script installed as
/etc/init.d/xendomains , and creates symlinks so that it starts in
runlevels 3, 4, and 5.

The xendomains script will iterate through the /etc/xen/auto
directory, and start each domain with a config file in that directory.
When the machine shuts down, it calls xendomains with a stop argument,
which shuts down all running domains. (Not just the ones in
/etc/xen/auto.)

We usually use symlinks -- simply link each domain that you want
started at boot into that directory

installing opensolaris and solaris express domUs.

Sat, 07 Mar 2009 22:03:05 -0800

Today I spent some time revamping the Solaris sections, which have hitherto been a source of some vexation. Plus side, I can get both the OpenSolaris and Solaris Express installers running in paravirtualized mode, no problem.

See, Solaris actually supports Xen so well that it's intensely frustrating when something doesn't work -- but I also find some of their decisions so confusing that I mess up and make stuff fail to work properly ALL THE TIME.

Sorry, I seem to have been sidetracked. Anyway, one benefit of this modern world is the ability to install distros directly from install media. Both SolExp and OpenSol support booting the installer directly as a domU. So, the first step is:

1. Download either OpenSolaris or Solaris Express.

This is easy enough. Both are available at http://opensolaris.org/os/downloads . OpenSol's a CD, SolExp's a DVD. Pick the one you want.

2. Create an appropriate domU config file.

You'll want to include the usual stuff. The important Solaris-specific bits are the kernel and miniroot location, and some boot arguments.

OpenSolaris:
kernel='/platform/i86xpv/kernel/amd64/unix'
ramdisk= '/boot/x86.microroot'
extra= '/platform/i86xpv/kernel/amd64/unix - nowin -B install_media=cdrom'

Solaris Express:
kernel='/boot/platform/i86xpv/kernel/amd64/unix'
ramdisk= '/boot/amd64/x86.miniroot'
extra='/boot/platform/i86xpv/kernel/amd64/unix -B install_media=cdrom'

(I'm not sure how important those parameters are.) Make sure that you include the usual config file trappings -- stuff like root device, name, and so forth.

If you're using a prgmr.com brand VPS, we'll have done these things already. That leaves the last step:

3. Start the domain and run the installer.

Solaris Express will start the installer automatically. One warning, though -- the installer hangs if you configure the network with it on our setup. Your mileage may vary. I just opted to claim the system was non-networked and set the network up later.

Now, OpenSolaris doesn't include a text-mode installer, and we don't give you framebuffers, so you'll have to do a couple things before you run the OpenSolaris installer. First, change your password. (Use the normal passwd command.) Then, enable sshd:

$ pfexec svcadm enable network/ssh

Give yourself your assigned IP, SSH in, and run the installer:

$ pfexec ifconfig netmask
$ pfexec route add default

(local desktop) $ ssh -X
$ pfexec /usr/bin/gui-install

twirling towards prosperity.

Sat, 14 Feb 2009 14:34:58 -0800

The fact that Xen's been ported to a less-antediluvian kernel version really makes me feel a lot more comfortable with the entire project.

Obviously I should be keeping up with this stuff better, but the truth is that OpenSUSE tends to slip under my radar -- it's another RPM distro in my mind, and so has to compete with RedHat and Fedora for my testing time. But today Luke casually mentioned that "some distro, like SUSE or something" was using a newer kernel version for the dom0.

And that's huge. One of my biggest worries about the long-term viability of Xen is that most installations of it are based on a kernel whose source was copied verbatim from Sumerian clay tablets. Now that we have known, running, major distro instances relying on kernel.org code -- let's just say I feel more comfortable recommending the platform.

return to the primitive.

Thu, 12 Feb 2009 19:27:30 -0800

Hey, look, I exist again.

Luke and I have decided to hole up in my tiny apartment and get this thing finished. We have sworn mighty, some would even say blasphemous oaths regarding our targets for the weekend, and I think we're doing pretty well. Considering we just came off a couple months of hiatus.

(Our publisher has a great deal to do with this. Have I ever mentioned the role of the Japanese editor? It involves a lot more babysitting that the Western counterpart.)

Anyway, we're knocking stuff out at a good pace. Today I found a nice article about the vTPM (from this year, even,) and Luke suggested that I write a blog post about it.

And here you go: http://cybione.org/~cdidier/log/200901221025.html

Suddenly I feel like we might include more information about the TPM. Have to play with the test boxes some more.

conversational snippet.

Sat, 29 Nov 2008 02:58:35 -0800

On logos:

"before you complain that it is not geeky enough, note that i have referenced both the logarithmic spiral and four-color theorem."