<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
    <channel>
        <title>Xen hosting: Lessons from the Trenches</title>
        <link>http://wiki.xen.prgmr.com/xenophilia/</link>
        <description></description>
        <language>en-US</language>
        <copyright>Copyright 2010</copyright>
        <lastBuildDate>Tue, 09 Mar 2010 13:29:36 -0800</lastBuildDate>
        <generator>http://www.sixapart.com/movabletype/</generator>
        <docs>http://www.rssboard.org/rss-specification</docs>
        
        <item>
            <title>Horn (and the VMs on it)  will be rebooted shortly</title>
            <description><![CDATA[we've got to replace a disk.&nbsp; <br /><br /><br />update:&nbsp; it is coming back up now, restoring domains.&nbsp; those of you running debian might need another reboot (I've had problems with debian save/restore)<br /><br />update:&nbsp; it looks like all domains but one came back up successfully (with a save/restore, no reboot)&nbsp; <br /><br />we'll be rebuilding the RAID now, so expect disk performance to suck for a while.&nbsp; <br /><br /><br />]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be-1.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be-1.html</guid>
            
            
            <pubDate>Tue, 09 Mar 2010 13:29:36 -0800</pubDate>
        </item>
        
        <item>
            <title>Horn (and the VMs on it)  will be rebooted shortly</title>
            <description><![CDATA[in order to replace a bad disk &nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/03/horn-and-the-vms-on-it-will-be.html</guid>
            
            
            <pubDate>Tue, 09 Mar 2010 13:29:36 -0800</pubDate>
        </item>
        
        <item>
            <title>the next server will be limited to 512M accounts and below</title>
            <description><![CDATA[we will do our best to put anyone who is in the preorder queue who needs something larger on an older server.&nbsp; <br /><br />There are two reasons why I am doing this.&nbsp; First, I want to move to a system where the small accounts are segregated from the large accounts;&nbsp; ultimately, I'd like to put all the 64MiB customers on one server, all the 128MiB customers on one server, etc... but I don't have enough servers for that, so I'm starting this way.&nbsp; <br /><br />The reason to segregate in this manner is to isolate performance problems that might be caused by the smaller domains using swap more, ah, vigorously than the larger domains.&nbsp; As you all know, the sata disk I use is by far the weakest link in my setup, and I hope this change will help people 'get what they pay for'&nbsp;&nbsp;&nbsp; (I will be implementing other procedures to see to it that everyone gets a fair shake at the disk on these smaller servers)<br /><br />The other (and perhaps larger) reason why I want to do this is financial.&nbsp; Right now, prgmr.com has more available labor than capital, and we are bottlenecking pretty hard on capital, as evidenced by the fact that the 'we are out of servers' sign is up more often than not.&nbsp;&nbsp; Now, there are several ways this could be solved<br /><br />1. I could raise prices<br /><br />2. I could charge a setup fee whenever I got below a certain capacity (Like 1, but temporary) <br /><br />3. I could&nbsp; see to it that my more profitable customers have access to new capacity before my less profitable customers do.<br /><br />(Yes, I could also get investors or a loan, but those both come with their own irritations.&nbsp; I'm considering doing contracting, but that has it's own irritations as well.)&nbsp; <br /><br />I am trying 3, mostly because I don't like the idea of raising prices.&nbsp; (In this industry, you raise your prices by keeping your prices the same)&nbsp; My pricing model is $4 per month per account, plus $1 per month per every 64MiB ram, so a 64MiB ram guest is $5, a 128MiB guest is $6, a 256MiB guest is $8, etc...&nbsp;&nbsp; so obviously, for any given amount of ram, I make more money the more small guests I sell.&nbsp; <br /><br /><br /><br />So, here is the plan.&nbsp; Knife, the next server, will host 64MiB, 128MiB, 256MiB, and 512MiB guests.&nbsp; this server should&nbsp; be up within a day or two.&nbsp; (if I haven't made any more mistakes, it will be done tonight. )&nbsp;&nbsp;&nbsp; Note, I don't know if it will get filled by the waiting list or not.&nbsp; It very well might.&nbsp;&nbsp; <br /><br />Then, within the next two weeks (I have all the parts; ram is in the mail)&nbsp; I plan on setting up a 16GiB/ single socket server for 64MiB guests only.&nbsp;&nbsp; <br /><br />Also within the next two weeks I&nbsp; will have another 32MiB/8 core server returned from a dedicated server customer who is leaving.&nbsp; If knife fills up with 512MiB and below domains, it will be another 512MiB and below server.&nbsp;&nbsp; Otherwise, it will service 1024MiB and above domains.&nbsp; <br /><br />Beyond that, I need to buy more hardware, something that this new scheme will hopefully facilitate.&nbsp; <br /><br /><br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/02/the-next-server-will-be-limite.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/02/the-next-server-will-be-limite.html</guid>
            
            
            <pubDate>Wed, 24 Feb 2010 03:22:56 -0800</pubDate>
        </item>
        
        <item>
            <title>bad disk on robe; expect perormance problems</title>
            <description><![CDATA[expect slow I/O while I rebuild the raid on robe.<br /><br /><br /><br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/02/bad-disk-on-robe-expect-perorm.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/02/bad-disk-on-robe-expect-perorm.html</guid>
            
            
            <pubDate>Mon, 22 Feb 2010 08:04:51 -0800</pubDate>
        </item>
        
        <item>
            <title>(short) network outage at SVTIX, midnight, 2010-02-13</title>
            <description><![CDATA[uh, happened just a bit ago.&nbsp; maybe 15 minutes of downtime.&nbsp; SVTIX warned me of emergency maintenance of a UPS maybe three hours before that.&nbsp; Nothing was rebooted; we just lost network for a bit. &nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/02/short-network-outage-at-svtix.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/02/short-network-outage-at-svtix.html</guid>
            
            
            <pubDate>Sun, 14 Feb 2010 01:46:11 -0800</pubDate>
        </item>
        
        <item>
            <title>Chris and I will be speaking at the local ACCU meeting tonight.  </title>
            <description>http://accu.org/index.php/accu_branches/accu_usa/next </description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/02/chris-and-i-will-be-speaking-a.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/02/chris-and-i-will-be-speaking-a.html</guid>
            
            
            <pubDate>Wed, 10 Feb 2010 17:28:28 -0800</pubDate>
        </item>
        
        <item>
            <title>sloooow disk I/O on horn due to bad disk</title>
            <description><![CDATA[I'm heading out to fix it right now.&nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/02/sloooow-disk-io-on-horn-due-to.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/02/sloooow-disk-io-on-horn-due-to.html</guid>
            
            
            <pubDate>Sun, 07 Feb 2010 13:32:32 -0800</pubDate>
        </item>
        
        <item>
            <title>IPv6 router upgrade at SVTIX</title>
            <description><![CDATA[(there are only a few of you on it) &nbsp; <br /><br />we're rebooting our experimental IPv6 router for testing... it shouldn't be more than a few minutes downtime for IPv6.&nbsp; <br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/01/ipv6-router-upgrade-at-svtix.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/01/ipv6-router-upgrade-at-svtix.html</guid>
            
            
            <pubDate>Sun, 31 Jan 2010 12:33:42 -0800</pubDate>
        </item>
        
        <item>
            <title>partial network outage last night</title>
            <description><![CDATA[my provider tells me there was an intermittent network outage at my Fremont he.net location (my reseller, not he.net)&nbsp; from 11pm to 1am PST. &nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2010/01/partial-network-outage-last-ni.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2010/01/partial-network-outage-last-ni.html</guid>
            
            
            <pubDate>Sun, 03 Jan 2010 17:39:22 -0800</pubDate>
        </item>
        
        <item>
            <title>stables and birds going down for update and move</title>
            <description>they are in one of our supermicro 2 in 1u units </description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/stables-and-birds-going-down-f.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/stables-and-birds-going-down-f.html</guid>
            
            
            <pubDate>Wed, 30 Dec 2009 22:04:55 -0800</pubDate>
        </item>
        
        <item>
            <title>hydra rebooting shortly</title>
            <description><![CDATA[we're trying to see if we can xm save like we did on lion, unlike we did on boar, but it's a pretty old box, so we might be rebooting you.&nbsp; <br /><br />[root@hydra /]# uptime<br />&nbsp;18:56:26 up 410 days, 15:44,&nbsp; 2 users,&nbsp; load average: 0.09, 0.29, 0.25<br /><br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/hydra-rebooting-shortly.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/hydra-rebooting-shortly.html</guid>
            
                <category domain="http://www.sixapart.com/ns/types#category">hosting status</category>
            
                <category domain="http://www.sixapart.com/ns/types#category">outage</category>
            
            
            <pubDate>Wed, 30 Dec 2009 18:55:00 -0800</pubDate>
        </item>
        
        <item>
            <title>boar.prgmr.com going down shortly</title>
            <description><![CDATA[just like lion, save for that there are fewer customers on boar.<br /><br />[root@boar ~]# uptime<br />&nbsp;16:24:55 up 410 days, 15:32,&nbsp; 3 users,&nbsp; load average: 0.00, 0.01, 0.00<br /><br />[root@boar ~]# xm list |wc -l<br />18<br /><br /><br />starting upgrade now (service won't be impacted until we start the reboot)<br /><br /><br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/boarprgmrcom-going-down-shortl.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/boarprgmrcom-going-down-shortl.html</guid>
            
            
            <pubDate>Wed, 30 Dec 2009 16:24:57 -0800</pubDate>
        </item>
        
        <item>
            <title>(short) downtime on hydra, stables, birds and boar - tomorow</title>
            <description><![CDATA[all servers will be rebooted (as lion was today)&nbsp; for some kernel upgrades, and to consolidate all my he.net servers to one rack. &nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/short-downtime-on-hydra-stable.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/short-downtime-on-hydra-stable.html</guid>
            
                <category domain="http://www.sixapart.com/ns/types#category">hosting status</category>
            
                <category domain="http://www.sixapart.com/ns/types#category">outage</category>
            
            
            <pubDate>Tue, 29 Dec 2009 23:15:13 -0800</pubDate>
        </item>
        
        <item>
            <title>~20 min network outage today</title>
            <description><![CDATA[from 11:58 to 12:19 PST&nbsp; -&nbsp; We suspect upstream network trouble as the cause.&nbsp; ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/20-min-network-outage-today.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/20-min-network-outage-today.html</guid>
            
            
            <pubDate>Tue, 29 Dec 2009 20:25:23 -0800</pubDate>
        </item>
        
        <item>
            <title>lion rebooting for kernel refresh and a move</title>
            <description><![CDATA[root@lion ~]# uptime<br />&nbsp;18:12:12 up 451 days, 17:42,&nbsp; 8 users,&nbsp; load average: 0.01, 0.03, 0.00<br /><br /><br />as usual, if we don't screw it up it will be 20 minutes downtime and no reboot for you, due to xm save/restore<br /><br /> ]]></description>
            <link>http://wiki.xen.prgmr.com/xenophilia/2009/12/lion-rebooting-for-kernel-refr.html</link>
            <guid>http://wiki.xen.prgmr.com/xenophilia/2009/12/lion-rebooting-for-kernel-refr.html</guid>
            
                <category domain="http://www.sixapart.com/ns/types#category">hardware</category>
            
                <category domain="http://www.sixapart.com/ns/types#category">hosting status</category>
            
                <category domain="http://www.sixapart.com/ns/types#category">outage</category>
            
            
            <pubDate>Tue, 29 Dec 2009 18:13:28 -0800</pubDate>
        </item>
        
    </channel>
</rss>
