Xen hosting: Lessons from the Trenches

sloooow disk I/O on horn due to bad disk

2010-02-07T21:32:32Z

I'm heading out to fix it right now.

IPv6 router upgrade at SVTIX

2010-01-31T20:33:42Z

(there are only a few of you on it)

we're rebooting our experimental IPv6 router for testing... it shouldn't be more than a few minutes downtime for IPv6.

partial network outage last night

2010-01-04T01:39:22Z

my provider tells me there was an intermittent network outage at my Fremont he.net location (my reseller, not he.net) from 11pm to 1am PST.

stables and birds going down for update and move

2009-12-31T06:04:55Z

they are in one of our supermicro 2 in 1u units

hydra rebooting shortly

2009-12-31T02:55:00Z

we're trying to see if we can xm save like we did on lion, unlike we did on boar, but it's a pretty old box, so we might be rebooting you.

[root@hydra /]# uptime
18:56:26 up 410 days, 15:44, 2 users, load average: 0.09, 0.29, 0.25

boar.prgmr.com going down shortly

2009-12-31T00:24:57Z

just like lion, save for that there are fewer customers on boar.

[root@boar ~]# uptime
16:24:55 up 410 days, 15:32, 3 users, load average: 0.00, 0.01, 0.00

[root@boar ~]# xm list |wc -l
18

starting upgrade now (service won't be impacted until we start the reboot)

(short) downtime on hydra, stables, birds and boar - tomorow

2009-12-30T07:15:13Z

all servers will be rebooted (as lion was today) for some kernel upgrades, and to consolidate all my he.net servers to one rack.

~20 min network outage today

2009-12-30T04:25:23Z

from 11:58 to 12:19 PST - We suspect upstream network trouble as the cause.

lion rebooting for kernel refresh and a move

2009-12-30T02:13:28Z

root@lion ~]# uptime
18:12:12 up 451 days, 17:42, 8 users, load average: 0.01, 0.03, 0.00

as usual, if we don't screw it up it will be 20 minutes downtime and no reboot for you, due to xm save/restore

prgmr.com network outage tonight

2009-12-09T13:21:47Z

I screwed up. I gave two customers the same IP and mac address. this took down the network. the problem should be fixed now

as far as I can tell we were down about two hours.

rebooting cerberus to replace a bad drive. Ugh. never again sata_nv

2009-12-03T06:59:46Z

yeah. so sata_nv doesn't hot swap on the 2.6.18 xen kernel that comes with xen 3.3. ugh.

if nothing goes too badly, you won't notice the reboot; your host will only be unreachable for
a few minutes, you will not actually experience a reboot.

IPv6 RA blocked in Fremont

2009-11-23T22:50:24Z

So last week at the he.net data center in Fremont, one of our providers switches rebooted and afterwards we haven't been receiving any IPv6 router advertisements on our lan. We contacted them immediately, but it still hasn't been fixed. Meanwhile, the workaround is to manually add a static default route and what the autoconfigured address would be if the RAs were being received. For example in linux, the command would be ip -6 route add default via 2001:470:1:41::1 dev eth0 for the default route and ip addr add 2001:470:1:41:(replace with host portion of your ipv6 address)/64 dev eth0.

do pci-express post diagnostic cards exist?

2009-11-06T04:39:58Z

maybe I"m just lazy and blind, but I can't find any post diagnostic cards that work with modern pci-express only servers

for those of you who are not janitors (or who work exclusively with tyan boards, which have an awesome post diagnostic LED on-board) a post diagnostic card reads I/O port 0080h, and prints out a 2 digit hex number on an LED display, which you can look up in the back of your motherboard manual to figure out what the hell is going on when your server is so goddamn loud you can't hear the goddamn beep codes. I had one that was PCI, and see many that are PCI and ISA, but I see none that would fit in my PCI-e only server. And I've got a supermicro that won't boot. One side of a 2 in 1u, too.

Example of a ISA/PCI post diagnostic card: http://www.elstonsystems.com/prod/pc_post_diagnostic_card.html

I need one of those, only I need it to plug into usb or serial or pci-express or some port my supermicro 2 in 1u servers have.

Meanwhile, I went ahead and overpaid for another stand-alone supermicro motherboard and hooked it into one of my old 3U chassis, so I should have capacity sometime later this week.

Xen domains not starting up.

2009-10-19T03:12:34Z

http://book.xen.prgmr.com/mediawiki/index.php/Vif_doesnt_go_away_when_shutting_down

also: http://lists.xensource.com/archives/html/xen-devel/2009-10/msg00873.html

worst-case I will reboot the server with a known-good xen kernel. (I was using 3.4.2-rc; I will downgrade to 3.4.1.)

One way or another, the problem will be solved tonight.

Users who have not shut down are thusfar unaffected. I will try to reboot without interfering much with their operation.

I should have reported this the other night

Update: rebooting branch (2:56 PST on Oct 19th. )
update: branch is up and everything is confirmed good (4:08 PSD on Oct 19th)

emergency network maintainince (also, upgrading to gigabit)

2009-10-15T02:11:05Z

we've been having some mysterious packet loss issues that look a lot like we are oversubscribing a 50Mbps connection, but we're not. we have a 100Mbps commit on a gig port. Our upstream believes the problem to be with our router, so I've been working on it. anyhow, I found this in my foundry:

BR-charon#show ip traffic
IP Statistics
  1916350241 received, 52550054 sent, 585142657 forwarded
  629624 filtered, 67 fragmented, 78 reassembled, 2033812 bad header
  14173 no route, 0 unknown proto, 0 no buffer, 632845943 other errors

so we're swapping it out with a procurve 2824 with firmware that was fresh this year (downright modern!)

anyhow, there shouldn't be more than 120 seconds or so downtime for anyone. we're doing the move incrementally. It should be done tonight.

Update: we are done.