Migrating without shared storage
As we all know, migration requires some form of network storage. Unfortunately, this introduces a layer of storage complexity that may not be suited to all Xen deployments.
What we want is some mechanism by which a domain can, at migration time, have its storage copied over to the target host in such a way as to allow seamless migration.
- Host A sends a request to Host B.
- Host B reserves memory and checks for the availability of storage.
- Iterative pre-copy -- Domain runs on Host A.
- Stop and copy.
- Domain running on Host B.
What we need is a hook that Xend calls after the domain's stopped, which transfers the storage between Host A and Host B.
In theory, the external-device-migrate script (in the /etc/xen/scripts directory) is the answer. This script was originally meant for TPM migration, but can be extended to handle any sort of virtual device -- Xend calls it after each migration step with the device type, a handle, and the migration step. You can add calls to further scripts as needed.
External-device-migrate doesn't seem to work with RHEL/CentOS. I would be very interested in getting it functional.
The distributed replicated block device (DRBD) people include a block-drbd script that allows you to use drbd: block devices directly as domU backing store.
A blog post  claims that DRBD enables Xen live migration, probably through a similar mechanism. I believe, based on perusal of the block-drbd script, that they mean it "works for two-node setups," but I could be wrong. (Since that would imply no benefit over specifying a DRBD phy: device.) Workflow should run as follows:
- Domain running on Host A with generic DRBD backing store
- Initiate migration. Host B checks storage requirement.
- Stop and copy
- Host B creates backing store, syncs it with Host A
- Domain resumes on Host B with local store.
Alternatively the domain could begin running on Host B immediately and rely on the DRBD backend to transfer data to B's mirror as needed.