xenophilia

GNU screen for collaboration

2008-10-03T11:42:25Z

GNU Screen is often used for keeping programs running while their user is logged out, but it is also very useful for collaboration when more than one user connects to a session. To do this, "multiuser on" must be set in the session, either in the screenrc file or with : in the session. Then the user who didn't start screen must connect to it with screen -r and the username of the owner of the session and the name of the session like "screen -r nick/31346.pts-15.lion". The name of the session can be found by the owner doing screen -ls or root looking in /var/run/screen/ in the owners session directory where there is a named pipe with the name of the session. The screen binary must be setuid root with chmod u+s and owner root. The session owner must explicitly allow the other users to connect with "addacl otherusernames" either in the screenrc or in the session. Then when both users are connected they have the privileges of the session owner and can see what each other are doing, allowing demonstration of various system tasks or issues.

issuing credits tonight for Sept. downtime

2008-10-01T03:15:13Z

if I missed you, please send me an e-mail. Note, you must be in the freeside billing system to get credits.

You can check your account balance here

pre-payment discounts

2008-10-01T03:00:02Z

Hey, so dealing with payments is something of a pain. It's easier if you send me fewer, larger payments. So to encourage this behavior, I am offering some discounts:

pay for 3 months, take 5% off
pay for 6 months, take 10% off
pay for 12 months, take 20% off

right now there is no minimum cheque size, but once I get Paypal hooked into freeside to automatically input payments, I will institute a minimum cheque size. (until I automate the paypal stuff, it doesn't really matter... right now, both mean that I have to manually input the payment)

scheduled downtime. 30 mins. starting 20081003 at 22:00 PST

2008-10-01T02:46:53Z

I'm moving all the prgmr.com servers at he.net into one rack. Irritating, I know, but it makes server management much easer, and I'll have more control over the network.

Unless I screw up the dhcp server (like I did last time) you should not notice anything more than being unreachable for 30 minutes. The server should automatically save and then restore your domain, with all programs running. If you get rebooted, I screwed something up.

rdns for 216.218.210.65/27 fixed

2008-09-21T18:44:06Z

I had a messed up $ORGIN statement, so rdns has never been working on those addresses. It's working now

has anyone experience with SVTIX?

2008-09-21T01:31:40Z

So, I need more co-lo space. I'm currently at he.net. They are OK, I guess. My international customers seem to be pretty impressed with my latency, and I like the native IPv6.

The problem is that they will only sell you 15A circuts (that's only 1125 watts usable. A competent admin is never going to go above 75% of a circuit's rated capacity, so we are talking fitting 9 or 10 of my 1U servers in that full rack.) at the freemont location where I am, and they were full last time I asked anyhow.

I need 20A of power (if I can get 30A single-phase 208V, that's even better) a full (locking, 4 post) cabinet, and a 10Mbps commit on a 100Mbps pipe (bonus if it's a 1000Mbps pipe) with reasonable overage charges. I also want the ability to bring in cross connects from other providers and exchange points as my bandwidth use grows.

(now, no two ways about it. He.net is a cheap provider. you can hear the whine of the alarms on the failed PDUs as you walk through the place. But unless your hardware is a lot better than mine, even a crappy datacenter is going to be a whole lot more reliable than x86 hardware. I like cheap.)

So I've been looking around. I found some ridiculously good prices at egihosting for 20A circuits, and very fair bandwidth prices. they are in at SVTIX in san jose. the big problem here is that they don't allow cross connects from other providers (which in my experience means that once they have you locked in, they will screw you on bandwidth charges.)

I'm asking SVTIX for a direct quote, we'll see what they have to say. If any of you have used them, let me know what you thought.

But yeah. I want to play with BGP and peering... this means I need access to cheap bandwidth so I can get my usage up to something worth peering with without breaking my budget, and I need to get access to more than one provider, and once my usage is up, I need to get access to something like metroPAIX or some other major exchange.

the NetBSD images are no longer a lie, sortof.

2008-09-14T06:44:45Z

No, I've not setup non-PAE servers, but NetBSD-CURRENT has supported x86_64 and i386-PAE (as a DomU only) for some time now. I just go around to testing and setting it up.

full instructions here: http://book.xen.prgmr.com/mediawiki/index.php/NetBSD_as_a_DomU

or if you are a new customer, I've included the NetBSD install kernel in the menu.lst in all my debian x86_64 images.

It looks to work (though it is still -current) - It hasn't crashed on me yet.

rebooting hydra and lion

2008-09-07T04:18:34Z

if I do it right, all you will notice is 5 minutes of inactivity, as Xen is configured to save all the DomUs to disk. (much like how you can hibernate your laptop)

In the spirit of "only make each mistake once" we are installing serial consoles after the problem on boar the other day. We will also be upgrading dom0 kernels to the centos latest.

http://book.xen.prgmr.com/mediawiki/index.php/Serial_console

more unplanned downtime.

2008-09-05T20:52:21Z

Or, wouldn't remote console access be nice?

Boar.prgmr.com appears to be down. We're bringing it back up as soon as possible. Here's the back story:

I stupidly created two domUs with identical MAC addresses. As far as we can tell, this destroyed Xen's virtual bridge. . . First my SSH connection went. Twenty minutes later, the entire box and domains hosted on it became unpingable. I feel terrible about the entire thing.

Luke's headed out now to reboot the machine.

I think I'm going to write some Python to automatically generate MAC addresses from IP addresses.

Coloma reboot.

2008-08-18T06:45:52Z

We're going to need to reboot Coloma for maintenance and troubleshooting. I'm targeting midnight tomorrow, PDT -- approximately 24 hours from now.

We expect only a few minutes of downtime. After that you should be able to log in and restart your VM normally. Unfortunately, the problem that's provoking the reboot seems to make console access impossible for now.

IPv6 RDNS setup

2008-07-31T17:57:00Z

so IPv6 rdns is a little different from IPv4 rdns. With IPv4 rdns, you split each IP address on byte boundries, reverse the octets, and append in-addr.arpa.   all data is represented in decimal, and you don't pad zeros. for example, to get the ipv4 rdns of 216.218.223.67 you look for a ptr record named 67.223.218.216.in-addr.arpa.

IPv6 rdns is similar on the surface;   instead of .in-addr.arpa. you append ip6.arpa, but you split the address in unexpected ways, too:

IPv6 addresses are written out in hex, two byte chunks seperated by colon charaters.   IPv6 rdns writes out the full address in hex including padding out all zeros, and then splits it into 4 bit chunks (single hex characters) and reverses those.

so to get the rdns of, say, ns2.prgmr.com, IPv6 address: 2001:470:1:41:a800:ff:fe50:3143
you would look for a PTR record that looked like this:; 3.4.1.3.0.5.e.f.f.f.0.0.0.0.8.a.1.4.0.0.1.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa.

note that you must pad out the zeros so that each two-byte chunk seperated by the ':' character is represented by four characters.

Of course, dig -x does this for us....

dig -x 2001:470:1:41:a800:ff:fe50:3143

; <<>> DiG 9.3.4-P1 <<>> -x 2001:470:1:41:a800:ff:fe50:3143
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20189
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;3.4.1.3.0.5.e.f.f.f.0.0.0.0.8.a.1.4.0.0.1.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa

unplanned reboot of coloma

2008-07-29T17:30:46Z

Coloma is my old i386-PAE box. dual xeons in a supermicro chassis. kinda, well, old.

There were three problems. First, I let the userland xen tools get out of sync with the kernel. (uncontrolled yum update is not a good thing) Second, on this old box I never tested the 'save domains on reboot' functionality (on the new servers, if I reboot the dom0, it does an 'xm save' on every running DomU, and an 'xm restore' upon reboot, meaning that rather than seeing a reboot, the DomU owner might notice that the DomU was unavailable for 5-10 minutes, but everything that was running on it before was still running- it would be like unplugging the ethernet cable for a while and plugging it back in) The third (and perhaps largest) problem was that I rebooted the server to deal with the first problem without scheduling it (would have *maybe* been acceptable (but not good) on the new servers, but on the old ones, this was a mistake.)

I'll schedule things better in the future.

IPv6 by accident.

2008-07-08T08:37:14Z

Yesterday I got a funny e-mail from a customer. He was asking for RDNS for his IPv6 address. The funny part is that I thought that I had not yet setup IPv6.

I currently claim to support only IPv4. I did ask for an IPv6 allocation, and about a week ago my provider got back to me with a netblock from he.net, but I've been busy, and I have yet to get around to actually playing with the stuff. Besides, we still have north of 900 days before we run out of IPv4 at the current burn rate.[1]

Now, I've read about this 'IPv6 stateless auto-configuration'[2] The idea is that IPv6 routers announce prefixes via ICMP6 once every 10 seconds, and then the client does some math to create the rest of the IP address using the mac address.

I ask my customer what the IPv6 address he has is, and if it actually works. Yup, the address is in my block, and yes, it works. He just needs an RDNS mapping.

Hopefully, my provider will get me the rdns delegation Tuesday or Wednesday.

Easy Peasy.

[1]http://www.potaroo.net/tools/ipv4/index.html
[2]http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_7-2/ipv6_autoconfig.html

http.xen.prgmr.com and wiki.xen.prgmr.com moved off of tahoe

2008-07-01T08:43:57Z

Tahoe has problems, the worst of which are 32-on-64 problems, but it's also not mirrored, so I'm trying to move everything off of it. Tonight I moved the main webserver and the movable type server (which is called wiki.xen.prgmr.com, for historical reasons. our mediawiki server is book.xen.prgmr.com) I re-ip'd both, and both are running on both old and new servers until DNS finishes updating. I moved http to a brand new 64-bit image on boar at he.net- it should be much more reliable. I moved wiki to Coloma, a native i386-PAE box hosted at rippleweb in Sacramento.

Customers on tahoe are encouraged to move to new servers;

drive performance problem solved

2008-06-28T08:28:44Z

the samsung 750G drives (with 32M cache) are almost twice as fast as the segate 750G drives (with 16M cache) - even after removing the limiters (the seagates came with jumpers
that limited them to 1.5G sata1)

Anyhow, I've started to put system domains on the new server, Boar, and they are looking pretty good.