Zfs storage server setup

remember, it is very important you not hand the wrong disk to the DomU, so instead of passing in /dev/sdc... I passed in /dev/by-uuid/....

#PyGRUB
bootloader = '/usr/bin/pygrub'
kernel = "/platform/i86xpv/kernel/amd64/unix"
ramdisk = "/boot/x86.microroot"
extra = "/platform/i86xpv/kernel/amd64/unix -- nowin -B install_media=cdrom"

#kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz"
#extra = "(hd1,0)/boot/grub/menu.lst"
cpu_weight=16384
memory = 16384
name = "storage"
vif = ['vifname=storage,ip=64.71.167.137,bridge=xenbr0,mac=aa:00:00:59:A7:89' ]
disk = [
        'file:/home/lsc/osol-0811.iso,xvdi:cdrom,r',
        'phy:/dev/mares_domU/storage,xvda,w',
        'phy:/dev/mares_domU/rescue,xvdb,r',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS0A958,xvdc,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS11N7C,xvdd,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS139NN,xvde,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS13RRY,xvdf,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS13YX2,xvdg,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS1BTX8,xvdh,w'
]

this boots you into opensolaris. login with user jack and password jack. (note, we haven't yet setup network, so we are safe) change jack's password with passwd. now, fix the network

 pfexec ifconfig xnf0 64.71.167.137/25
 pfexec route add default 64.71.167.129
 pfexec svcadm enable ssh 

then from a linux box,

ssh -X 64.71.167.137

and within that ssh

pfexec /usr/bin/gui-install

run the install as normal with the gui (hah. not confidence-inspiring)

now, I'm trying to boot it. trying with pygrub. current version of pygrub in -testing doesn't support the version of zfs in 2008.11, so I'm going to install the -unstable pygrub. (note, I don't want the -unstable hypervisor or tools, just libfsimage and pygrub) :

http://xenbits.xensource.com/xen-unstable.hg

make tools

cd xen-unstable.hg/tools/pygrub; make install
cd xen-unstable.hg/tools/libfsimage; make install

now update the /etc/xen/storage file:

bootloader = '/usr/bin/pygrub'

cpu_weight=16384
memory = 16384
cpus= "2"
vcpus=1
name = "storage"
vif = ['vifname=storage,ip=64.71.167.137,bridge=xenbr0,mac=aa:00:00:59:A7:89' ]
disk = [
        'phy:/dev/mares_domU/storage,xvda,w',
        'phy:/dev/mares_domU/rescue,xvdb,r', 
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS0A958,xvdc,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS11N7C,xvdd,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS139NN,xvde,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS13RRY,xvdf,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS13YX2,xvdg,w',
        'phy:/dev/disk/by-id/scsi-SATA_ST31500341AS_9VS1BTX8,xvdh,w'
]

login with one of the users you created in the install and you are done.

after reboot, I am trying to setup zfs. to see available drives,

pfexec format

and guess at what is what.

in my case

pfexec zpool create tank raidz c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t6d0 c3t7d0

now you need to create a partition

pfexec zfs create tank/test

and you want to compress it, probably

 pfexec zfs set compression=gzip tank/test

and export it via nfs

pfexec svcadm enable nfs/server

zfs set sharenfs=on tank/test

pfexec zfs set compression=gzip tank/test

set up networking

(first, kill nwam. I'm told it has something to do with network autoconfig, but personally I think it's part of sun's master plan to make solaris impenetrable to *NIX people.)

pfexec svcadm disable -s network/physical:nwam

pfexec vi /etc/hostname.xnf0
64.71.167.137/25

pfexec vi /etc/defaultrouter
64.71.167.129

then make network changes active

pfexec svcadm enable -s network/physical:default

setup /etc/resolv.conf as usual.