Xen hosting: Lessons from the Trenches tag:wiki.xen.prgmr.com,2008-03-02:/xenophilia/2 2010-03-09T22:31:11Z Movable Type Open Source 4.1 Horn (and the VMs on it) will be rebooted shortly tag:wiki.xen.prgmr.com,2010:/xenophilia//2.191 2010-03-09T21:29:36Z 2010-03-09T22:31:11Z luke http://prgmr.com

update:  it is coming back up now, restoring domains.  those of you running debian might need another reboot (I've had problems with debian save/restore)

update:  it looks like all domains but one came back up successfully (with a save/restore, no reboot) 

we'll be rebuilding the RAID now, so expect disk performance to suck for a while. 


]]> Horn (and the VMs on it) will be rebooted shortly tag:wiki.xen.prgmr.com,2010:/xenophilia//2.190 2010-03-09T21:29:36Z 2010-03-09T21:30:21Z luke http://prgmr.com the next server will be limited to 512M accounts and below tag:wiki.xen.prgmr.com,2010:/xenophilia//2.188 2010-02-24T11:22:56Z 2010-02-24T11:48:12Z luke http://prgmr.com
There are two reasons why I am doing this.  First, I want to move to a system where the small accounts are segregated from the large accounts;  ultimately, I'd like to put all the 64MiB customers on one server, all the 128MiB customers on one server, etc... but I don't have enough servers for that, so I'm starting this way. 

The reason to segregate in this manner is to isolate performance problems that might be caused by the smaller domains using swap more, ah, vigorously than the larger domains.  As you all know, the sata disk I use is by far the weakest link in my setup, and I hope this change will help people 'get what they pay for'    (I will be implementing other procedures to see to it that everyone gets a fair shake at the disk on these smaller servers)

The other (and perhaps larger) reason why I want to do this is financial.  Right now, prgmr.com has more available labor than capital, and we are bottlenecking pretty hard on capital, as evidenced by the fact that the 'we are out of servers' sign is up more often than not.   Now, there are several ways this could be solved

1. I could raise prices

2. I could charge a setup fee whenever I got below a certain capacity (Like 1, but temporary)

3. I could  see to it that my more profitable customers have access to new capacity before my less profitable customers do.

(Yes, I could also get investors or a loan, but those both come with their own irritations.  I'm considering doing contracting, but that has it's own irritations as well.) 

I am trying 3, mostly because I don't like the idea of raising prices.  (In this industry, you raise your prices by keeping your prices the same)  My pricing model is $4 per month per account, plus $1 per month per every 64MiB ram, so a 64MiB ram guest is $5, a 128MiB guest is $6, a 256MiB guest is $8, etc...   so obviously, for any given amount of ram, I make more money the more small guests I sell. 



So, here is the plan.  Knife, the next server, will host 64MiB, 128MiB, 256MiB, and 512MiB guests.  this server should  be up within a day or two.  (if I haven't made any more mistakes, it will be done tonight. )    Note, I don't know if it will get filled by the waiting list or not.  It very well might.  

Then, within the next two weeks (I have all the parts; ram is in the mail)  I plan on setting up a 16GiB/ single socket server for 64MiB guests only.  

Also within the next two weeks I  will have another 32MiB/8 core server returned from a dedicated server customer who is leaving.  If knife fills up with 512MiB and below domains, it will be another 512MiB and below server.   Otherwise, it will service 1024MiB and above domains. 

Beyond that, I need to buy more hardware, something that this new scheme will hopefully facilitate. 


]]> bad disk on robe; expect perormance problems tag:wiki.xen.prgmr.com,2010:/xenophilia//2.187 2010-02-22T16:04:51Z 2010-02-22T16:05:57Z expect slow I/O while I rebuild the raid on robe.... luke http://prgmr.com


]]> (short) network outage at SVTIX, midnight, 2010-02-13 tag:wiki.xen.prgmr.com,2010:/xenophilia//2.186 2010-02-14T09:46:11Z 2010-02-14T09:47:38Z luke http://prgmr.com Chris and I will be speaking at the local ACCU meeting tonight. tag:wiki.xen.prgmr.com,2010:/xenophilia//2.185 2010-02-11T01:28:28Z 2010-02-11T01:29:17Z http://accu.org/index.php/accu_branches/accu_usa/next... luke http://prgmr.com http://accu.org/index.php/accu_branches/accu_usa/next sloooow disk I/O on horn due to bad disk tag:wiki.xen.prgmr.com,2010:/xenophilia//2.184 2010-02-07T21:32:32Z 2010-02-07T21:32:57Z luke http://prgmr.com IPv6 router upgrade at SVTIX tag:wiki.xen.prgmr.com,2010:/xenophilia//2.183 2010-01-31T20:33:42Z 2010-01-31T20:35:26Z luke http://prgmr.com
we're rebooting our experimental IPv6 router for testing... it shouldn't be more than a few minutes downtime for IPv6. 
]]> partial network outage last night tag:wiki.xen.prgmr.com,2010:/xenophilia//2.182 2010-01-04T01:39:22Z 2010-01-04T01:43:22Z luke http://prgmr.com stables and birds going down for update and move tag:wiki.xen.prgmr.com,2009:/xenophilia//2.181 2009-12-31T06:04:55Z 2009-12-31T06:07:46Z they are in one of our supermicro 2 in 1u units... luke http://prgmr.com they are in one of our supermicro 2 in 1u units hydra rebooting shortly tag:wiki.xen.prgmr.com,2009:/xenophilia//2.180 2009-12-31T02:55:00Z 2009-12-31T03:10:31Z luke http://prgmr.com
[root@hydra /]# uptime
 18:56:26 up 410 days, 15:44,  2 users,  load average: 0.09, 0.29, 0.25

]]> boar.prgmr.com going down shortly tag:wiki.xen.prgmr.com,2009:/xenophilia//2.179 2009-12-31T00:24:57Z 2009-12-31T00:27:10Z luke http://prgmr.com
[root@boar ~]# uptime
 16:24:55 up 410 days, 15:32,  3 users,  load average: 0.00, 0.01, 0.00

[root@boar ~]# xm list |wc -l
18


starting upgrade now (service won't be impacted until we start the reboot)


]]> (short) downtime on hydra, stables, birds and boar - tomorow tag:wiki.xen.prgmr.com,2009:/xenophilia//2.178 2009-12-30T07:15:13Z 2009-12-30T07:18:29Z luke http://prgmr.com ~20 min network outage today tag:wiki.xen.prgmr.com,2009:/xenophilia//2.177 2009-12-30T04:25:23Z 2009-12-30T04:27:20Z luke http://prgmr.com lion rebooting for kernel refresh and a move tag:wiki.xen.prgmr.com,2009:/xenophilia//2.176 2009-12-30T02:13:28Z 2009-12-30T02:15:13Z luke http://prgmr.com  18:12:12 up 451 days, 17:42,  8 users,  load average: 0.01, 0.03, 0.00


as usual, if we don't screw it up it will be 20 minutes downtime and no reboot for you, due to xm save/restore

]]>