http://wiki.xen.prgmr.com/xenophilia/ en-US Copyright 2010 Tue, 16 Mar 2010 02:50:41 -0800 http://www.sixapart.com/movabletype/ http://www.rssboard.org/rss-specification http://wiki.xen.prgmr.com/xenophilia/2010/03/1hr-network-outage-on-branch-d.html http://wiki.xen.prgmr.com/xenophilia/2010/03/1hr-network-outage-on-branch-d.html outage Tue, 16 Mar 2010 02:50:41 -0800
The problem is that people get a 4GiB domain and then are all disappointed, because I/O sucks.  The thing is, Virtualization sucks for storage.  If I have a mirror, and I run 8 4GiB guests on it, when there is contention for disk, each domain gets a lot less than 1/8th the disk performance they would get if they had a disk to themselves;  you see, SATA disk is pretty good for sequential transfers, but it sucks for random access.  If you have 8 guys trying to sequentially access the disk, guess what?  all that sequential access is now random, in an attempt to service each request in a reasonable amount of time. 

(now, I/O also sucks for smaller domains, but people don't complain as much there.   I mean,  you are paying $8 a month, what do you want?  If you get enough ram, the thing ends up being reasonably usable once you warm up the cache.)

So, my plan for the 4GiB domains is to jack up the prices (I plan on staying well below slicehost and linode prices, still)  and giving you your own mirror.  Current pricing plan is something like the current price ($68/month)  plus $0.10 per gigabyte of disk (so around $100 for a 1TB disk)  with the caveat that you must buy the entire disk.   (you would get a mirror, not an individual disk, so I think $.10 per gigabyte month is a reasonably good deal.)   

So, I have a bunch of disk laying about on many of my servers, sitting idle;  it is possible I  will use some of them as 4TiB or 8TiB guests.  It is also possible I'll just start buying servers with 16 drive bays and going at it that way.  (maybe with the 500GB 2.5" disks?  500GB is a nicer disk size and disk cost than making people buy a whole TB and a half, which are what the disks I have now are.)  

note, as the title implies, these are 'forward looking statements'  - I have not tested this out;  I have not shaken out the bugs.  It's possible I'll never get around to doing this.  It's even more likely that I'll run out of free disks in short order and it will be a while before I accept new larger orders. 
]]> http://wiki.xen.prgmr.com/xenophilia/2010/03/forward-looking-statements.html http://wiki.xen.prgmr.com/xenophilia/2010/03/forward-looking-statements.html Sun, 14 Mar 2010 18:32:48 -0800

update:  it is coming back up now, restoring domains.  those of you running debian might need another reboot (I've had problems with debian save/restore)

update:  it looks like all domains but one came back up successfully (with a save/restore, no reboot) 

we'll be rebuilding the RAID now, so expect disk performance to suck for a while. 

Update 2010-03-11 02:28 PST:

wow, disk is sucking a lot more than we expected.  I am wondering if the remaining disk also has problems.   Either way, we've gotta wait for it to finish rebuilding, and that will take on the order of another 24 hours, according to my best estimate.   All users of horn will receive a month worth of credit.  
]]> http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be-1.html http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be-1.html Tue, 09 Mar 2010 13:29:36 -0800 http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be.html http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be.html Tue, 09 Mar 2010 13:29:36 -0800
There are two reasons why I am doing this.  First, I want to move to a system where the small accounts are segregated from the large accounts;  ultimately, I'd like to put all the 64MiB customers on one server, all the 128MiB customers on one server, etc... but I don't have enough servers for that, so I'm starting this way. 

The reason to segregate in this manner is to isolate performance problems that might be caused by the smaller domains using swap more, ah, vigorously than the larger domains.  As you all know, the sata disk I use is by far the weakest link in my setup, and I hope this change will help people 'get what they pay for'    (I will be implementing other procedures to see to it that everyone gets a fair shake at the disk on these smaller servers)

The other (and perhaps larger) reason why I want to do this is financial.  Right now, prgmr.com has more available labor than capital, and we are bottlenecking pretty hard on capital, as evidenced by the fact that the 'we are out of servers' sign is up more often than not.   Now, there are several ways this could be solved

1. I could raise prices

2. I could charge a setup fee whenever I got below a certain capacity (Like 1, but temporary)

3. I could  see to it that my more profitable customers have access to new capacity before my less profitable customers do.

(Yes, I could also get investors or a loan, but those both come with their own irritations.  I'm considering doing contracting, but that has it's own irritations as well.) 

I am trying 3, mostly because I don't like the idea of raising prices.  (In this industry, you raise your prices by keeping your prices the same)  My pricing model is $4 per month per account, plus $1 per month per every 64MiB ram, so a 64MiB ram guest is $5, a 128MiB guest is $6, a 256MiB guest is $8, etc...   so obviously, for any given amount of ram, I make more money the more small guests I sell. 



So, here is the plan.  Knife, the next server, will host 64MiB, 128MiB, 256MiB, and 512MiB guests.  this server should  be up within a day or two.  (if I haven't made any more mistakes, it will be done tonight. )    Note, I don't know if it will get filled by the waiting list or not.  It very well might.  

Then, within the next two weeks (I have all the parts; ram is in the mail)  I plan on setting up a 16GiB/ single socket server for 64MiB guests only.  

Also within the next two weeks I  will have another 32MiB/8 core server returned from a dedicated server customer who is leaving.  If knife fills up with 512MiB and below domains, it will be another 512MiB and below server.   Otherwise, it will service 1024MiB and above domains. 

Beyond that, I need to buy more hardware, something that this new scheme will hopefully facilitate. 


]]> http://wiki.xen.prgmr.com/xenophilia/2010/02/the-next-server-will-be-limite.html http://wiki.xen.prgmr.com/xenophilia/2010/02/the-next-server-will-be-limite.html Wed, 24 Feb 2010 03:22:56 -0800


]]> http://wiki.xen.prgmr.com/xenophilia/2010/02/bad-disk-on-robe-expect-perorm.html http://wiki.xen.prgmr.com/xenophilia/2010/02/bad-disk-on-robe-expect-perorm.html Mon, 22 Feb 2010 08:04:51 -0800 http://wiki.xen.prgmr.com/xenophilia/2010/02/short-network-outage-at-svtix.html http://wiki.xen.prgmr.com/xenophilia/2010/02/short-network-outage-at-svtix.html Sun, 14 Feb 2010 01:46:11 -0800 http://accu.org/index.php/accu_branches/accu_usa/next http://wiki.xen.prgmr.com/xenophilia/2010/02/chris-and-i-will-be-speaking-a.html http://wiki.xen.prgmr.com/xenophilia/2010/02/chris-and-i-will-be-speaking-a.html Wed, 10 Feb 2010 17:28:28 -0800 http://wiki.xen.prgmr.com/xenophilia/2010/02/sloooow-disk-io-on-horn-due-to.html http://wiki.xen.prgmr.com/xenophilia/2010/02/sloooow-disk-io-on-horn-due-to.html Sun, 07 Feb 2010 13:32:32 -0800
we're rebooting our experimental IPv6 router for testing... it shouldn't be more than a few minutes downtime for IPv6. 
]]> http://wiki.xen.prgmr.com/xenophilia/2010/01/ipv6-router-upgrade-at-svtix.html http://wiki.xen.prgmr.com/xenophilia/2010/01/ipv6-router-upgrade-at-svtix.html Sun, 31 Jan 2010 12:33:42 -0800 http://wiki.xen.prgmr.com/xenophilia/2010/01/partial-network-outage-last-ni.html http://wiki.xen.prgmr.com/xenophilia/2010/01/partial-network-outage-last-ni.html Sun, 03 Jan 2010 17:39:22 -0800 they are in one of our supermicro 2 in 1u units http://wiki.xen.prgmr.com/xenophilia/2009/12/stables-and-birds-going-down-f.html http://wiki.xen.prgmr.com/xenophilia/2009/12/stables-and-birds-going-down-f.html Wed, 30 Dec 2009 22:04:55 -0800
[root@hydra /]# uptime
 18:56:26 up 410 days, 15:44,  2 users,  load average: 0.09, 0.29, 0.25

]]> http://wiki.xen.prgmr.com/xenophilia/2009/12/hydra-rebooting-shortly.html http://wiki.xen.prgmr.com/xenophilia/2009/12/hydra-rebooting-shortly.html hosting status outage Wed, 30 Dec 2009 18:55:00 -0800
[root@boar ~]# uptime
 16:24:55 up 410 days, 15:32,  3 users,  load average: 0.00, 0.01, 0.00

[root@boar ~]# xm list |wc -l
18


starting upgrade now (service won't be impacted until we start the reboot)


]]> http://wiki.xen.prgmr.com/xenophilia/2009/12/boarprgmrcom-going-down-shortl.html http://wiki.xen.prgmr.com/xenophilia/2009/12/boarprgmrcom-going-down-shortl.html Wed, 30 Dec 2009 16:24:57 -0800 http://wiki.xen.prgmr.com/xenophilia/2009/12/short-downtime-on-hydra-stable.html http://wiki.xen.prgmr.com/xenophilia/2009/12/short-downtime-on-hydra-stable.html hosting status outage Tue, 29 Dec 2009 23:15:13 -0800