Xen hosting: Lessons from the Trenches

I'm heading out to fix it right now. 

(there are only a few of you on it)  

we're rebooting our experimental IPv6 router for testing... it shouldn't be more than a few minutes downtime for IPv6. 

my provider tells me there was an intermittent network outage at my Fremont he.net location (my reseller, not he.net)  from 11pm to 1am PST.  

they are in one of our supermicro 2 in 1u units

we're trying to see if we can xm save like we did on lion, unlike we did on boar, but it's a pretty old box, so we might be rebooting you. 

[root@hydra /]# uptime
 18:56:26 up 410 days, 15:44,  2 users,  load average: 0.09, 0.29, 0.25

just like lion, save for that there are fewer customers on boar.

[root@boar ~]# uptime
 16:24:55 up 410 days, 15:32,  3 users,  load average: 0.00, 0.01, 0.00

[root@boar ~]# xm list |wc -l
18


starting upgrade now (service won't be impacted until we start the reboot)

all servers will be rebooted (as lion was today)  for some kernel upgrades, and to consolidate all my he.net servers to one rack.  

from 11:58 to 12:19 PST  -  We suspect upstream network trouble as the cause. 

root@lion ~]# uptime
 18:12:12 up 451 days, 17:42,  8 users,  load average: 0.01, 0.03, 0.00


as usual, if we don't screw it up it will be 20 minutes downtime and no reboot for you, due to xm save/restore

 

I screwed up. I gave two customers the same IP and mac address. this took down the network. the problem should be fixed now

as far as I can tell we were down about two hours.