Recently in hosting status Category

I brought the first new server online last weekend, and I have two more ready to go. check out the ridiculous SuperMicro 1U twin I was talking about the other day. Note the ESD precautions. But yeah, each motherboard in that dual-motherboard 1u? 32GB ram. Ridiculous.

Some have suggested that my new slogan should be "We save money on web design, and pass the savings on to you!" but really, we're working on it, hopefully we will have a new layout soon. One of the new guys knows css, and Chris is an excellent designer, so between the two of them, I should get something reasonable.

We're working on a new provisioning system... all my competitors have wiz-bang systems that let you build out capacity on demand. Kindof irritating that I'm four years in and still don't have a good provisioning system.

I am self-funding, so the downturn hasn't effected me yet. (the self-funding mostly comes from my contracting operations, so it is possible I will be effected, but if I fill up the servers I have in the garage, well, the consulting operations will start to matter a lot less.) I'm going to try to use this time when my competitors can't get loans to ramp up my customer base. Maybe price will matter more now than it did before.

note to existing customers: if you let me move you to hydra-new, you get to keep your current disk allocation and still get the lower prices for ram.

downtime should be under 10 minutes.  you are being moved on to the new server, which means that our new,  low-priced plans are available.  (I'm not done tweaking the prices/ram, I will probably even out the curve a little, but the price per megabyte of ram won't be going up.) 

I will be removing two of our legacy servers to make room for our new server.   (one of them doesn't have customers on it, only customers on hydra should be impacted by this.) 

It has begun,  hind is going down now.  After that, hydra

Late Friday night, I'm going to head down to the co-lo and replace the failed drive on Boar's mirror.  If this task goes well, customers will notice nothing more than a short period of unreachability. the save/restore should put them back where they were without a reboot. 

ok starting now

ok, bad drive is samsung w/ serial s13Uj1nq108089

always mark serials before replacing drives... this much I have learned.

I'm moving all the prgmr.com servers at he.net into one rack. Irritating, I know, but it makes server management much easer, and I'll have more control over the network.

Unless I screw up the dhcp server (like I did last time) you should not notice anything more than being unreachable for 30 minutes. The server should automatically save and then restore your domain, with all programs running. If you get rebooted, I screwed something up.

I had a messed up $ORGIN statement, so rdns has never been working on those addresses.  It's  working now

if I do it right, all you will notice is 5 minutes of inactivity, as Xen is configured to save all the DomUs to disk.  (much like how you can  hibernate your laptop)

In the spirit of "only make each mistake once" we are installing serial consoles after the problem on boar the other day.   We will also be upgrading dom0 kernels to the centos latest.

http://book.xen.prgmr.com/mediawiki/index.php/Serial_console

We're going to need to reboot Coloma for maintenance and troubleshooting.  I'm targeting midnight tomorrow, PDT -- approximately 24 hours from now.

Coloma is my old i386-PAE box.  dual xeons in a supermicro chassis.   kinda, well, old. 

There were three problems.   First, I let the userland xen tools get out of sync with the kernel.  (uncontrolled yum update is not a good thing)    Second,  on this old box I never tested the 'save domains on reboot' functionality (on the new servers, if I reboot the dom0, it does an 'xm save' on every running DomU, and an 'xm restore' upon reboot, meaning that rather than seeing a reboot, the DomU owner might notice that the DomU was unavailable for 5-10 minutes, but everything that was running on it before was still running-  it would be like unplugging the ethernet cable for a while and plugging it back in)    The third (and perhaps largest) problem was that I rebooted the server to deal with the first problem without scheduling it  (would have *maybe* been acceptable (but not good) on the new servers, but on the old ones, this was a mistake.) 

I'll schedule things better in the future. 

Tahoe has problems, the worst of which are 32-on-64 problems, but it's also not mirrored, so I'm trying to move everything off of it.  Tonight I moved the main webserver and the movable type server (which is called wiki.xen.prgmr.com, for historical reasons.   our mediawiki server is book.xen.prgmr.com)  I re-ip'd both, and both are running on both old and new servers until DNS finishes updating.  I moved http to a brand new 64-bit image on boar at he.net-  it should be much more reliable.  I moved wiki to Coloma, a native i386-PAE box hosted at rippleweb in Sacramento. 

Customers on tahoe are encouraged to move to new servers;

the samsung 750G drives (with 32M cache)  are almost twice as fast as the segate 750G drives (with 16M cache)  -  even after removing the limiters (the seagates came with jumpers
that limited them to 1.5G sata1) 

Anyhow, I've started to put system domains on the new server, Boar, and they are looking pretty good.