Vif doesnt go away when shutting down
From PrgmrWiki
In a dom0 running xen 3.4.2-rc1-pre and centos 5.3 when domUs would shutdown their vif wasn't being deleted.
This is in /var/log/xend.log
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:92) XendDomainInfo.create(['vm', ['name', 'rippond'], ['memory', 256], ['vcpus', 1], ['on_xend_start', 'igno
re'], ['on_xend_stop', 'ignore'], ['cpus', '1-7'], ['cpu_weight', 256], ['image', ['linux', ['kernel', '/usr/lib/xen/boot/pv-grub-x86_64.gz'], ['videoram', 4
], ['args', '(hd1,0)/boot/grub/menu.lst']]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'phy:/dev/guests/rippond'], ['dev', 'xvda'], ['mode', 'w']]],
['device', ['vbd', ['uname', 'phy:/dev/guests/rescue'], ['dev', 'xvde'], ['mode', 'r']]], ['device', ['vif', ['ip', '68.68.97.73'], ['mac', 'aa:00:00:44:61:4
9'], ['vifname', 'rippond'], ['bridge', 'xenbr0']]]])
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:2296) XendDomainInfo.constructDomain
[2009-10-16 22:36:22 2997] DEBUG (balloon:166) Balloon: 3810280 KiB free; need 4096; done.
[2009-10-16 22:36:22 2997] DEBUG (XendDomain:452) Adding Domain: 437
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:2497) XendDomainInfo.initDomain: 437 256
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:2521) _initDomain:shadow_memory=0x0, memory_static_max=0x10000000, memory_static_min=0x0.
[2009-10-16 22:36:22 2997] DEBUG (balloon:166) Balloon: 3810172 KiB free; need 263168; done.
[2009-10-16 22:36:22 2997] INFO (image:173) buildDomain os=linux dom=437 vcpus=1
[2009-10-16 22:36:22 2997] DEBUG (image:659) domid = 437
[2009-10-16 22:36:22 2997] DEBUG (image:660) memsize = 256
[2009-10-16 22:36:22 2997] DEBUG (image:661) image = /usr/lib/xen/boot/pv-grub-x86_64.gz
[2009-10-16 22:36:22 2997] DEBUG (image:662) store_evtchn = 1
[2009-10-16 22:36:22 2997] DEBUG (image:663) console_evtchn = 2
[2009-10-16 22:36:22 2997] DEBUG (image:664) cmdline = (hd1,0)/boot/grub/menu.lst
[2009-10-16 22:36:22 2997] DEBUG (image:665) ramdisk =
[2009-10-16 22:36:22 2997] DEBUG (image:666) vcpus = 1
[2009-10-16 22:36:22 2997] DEBUG (image:667) features =
[2009-10-16 22:36:22 2997] DEBUG (image:668) flags = 0
[2009-10-16 22:36:22 2997] INFO (XendDomainInfo:2160) createDevice: vbd : {'uuid': 'd2ecf3bb-5c4e-0f6a-ac8f-d6780656ecd1', 'bootable': 1, 'driver': 'paravirt
ualised', 'dev': 'xvda', 'uname': 'phy:/dev/guests/rippond', 'mode': 'w'}
[2009-10-16 22:36:22 2997] DEBUG (DevController:95) DevController: writing {'virtual-device': '51712', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'back
end-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/437/51712'} to /local/domain/437/device/vbd/51712.
[2009-10-16 22:36:22 2997] DEBUG (DevController:97) DevController: writing {'domain': 'rippond', 'frontend': '/local/domain/437/device/vbd/51712', 'uuid': 'd
2ecf3bb-5c4e-0f6a-ac8f-d6780656ecd1', 'bootable': '1', 'dev': 'xvda', 'state': '1', 'params': '/dev/guests/rippond', 'mode': 'w', 'online': '1', 'frontend-id
': '437', 'type': 'phy'} to /local/domain/0/backend/vbd/437/51712.
[2009-10-16 22:36:22 2997] INFO (XendDomainInfo:2160) createDevice: vbd : {'uuid': 'c9245e80-ffd0-4cb6-31e1-791c48e25988', 'bootable': 0, 'driver': 'paravirt
ualised', 'dev': 'xvde', 'uname': 'phy:/dev/guests/rescue', 'mode': 'r'}
[2009-10-16 22:36:22 2997] DEBUG (DevController:95) DevController: writing {'virtual-device': '51776', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'back
end-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/437/51776'} to /local/domain/437/device/vbd/51776.
[2009-10-16 22:36:22 2997] DEBUG (DevController:97) DevController: writing {'domain': 'rippond', 'frontend': '/local/domain/437/device/vbd/51776', 'uuid': 'c
9245e80-ffd0-4cb6-31e1-791c48e25988', 'bootable': '0', 'dev': 'xvde', 'state': '1', 'params': '/dev/guests/rescue', 'mode': 'r', 'online': '1', 'frontend-id'
: '437', 'type': 'phy'} to /local/domain/0/backend/vbd/437/51776.
[2009-10-16 22:36:22 2997] INFO (XendDomainInfo:2160) createDevice: vif : {'bridge': 'xenbr0', 'ip': '68.68.97.73', 'mac': 'aa:00:00:44:61:49', 'uuid': 'fe4e
f6f0-1184-ee9f-2e05-5b443b46e59d', 'vifname': 'rippond'}
[2009-10-16 22:36:22 2997] DEBUG (DevController:95) DevController: writing {'mac': 'aa:00:00:44:61:49', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id'
: '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/437/0'} to /local/domain/437/device/vif/0.
[2009-10-16 22:36:22 2997] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'rippond', 'handle': '0', 'uuid': 'fe4ef6f0-1184-ee
9f-2e05-5b443b46e59d', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '68.68.97.73', 'mac': 'aa:00:00:44:61:49', 'frontend-id': '437', 'state': '1', 'vifname
': 'rippond', 'online': '1', 'frontend': '/local/domain/437/device/vif/0'} to /local/domain/0/backend/vif/437/0.
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:3052) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '40797849-77b8-a2e8-bea8-
cae425351d64', 'on_reboot': 'restart', 'start_time': '1255757782.47', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash':
'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '', 'image': "(linux (kernel /usr/lib/xen/boot/pv-grub-x86_64.gz) (args
'(hd1,0)/boot/grub/menu.lst') (videoram 4) (notes))", 'name': 'rippond'}
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:1622) Storing domain details: {'console/ring-ref': '4891682', 'console/port': '2', 'name': 'rippond', 'conso
le/limit': '1048576', 'vm': '/vm/40797849-77b8-a2e8-bea8-cae425351d64', 'domid': '437', 'store/port': '1', 'cpu/0/availability': 'online', 'memory/target': '
262144', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '4891683', 'console/type': 'xenconsoled'}
[2009-10-16 22:36:22 2997] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/dom
ain/0/backend/console/437/0'} to /local/domain/437/device/console/0.
[2009-10-16 22:36:22 2997] DEBUG (DevController:97) DevController: writing {'domain': 'rippond', 'frontend': '/local/domain/437/device/console/0', 'uuid': '8
1aa5d18-ffd4-dbfd-faae-ba570963b676', 'frontend-id': '437', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/con
sole/437/0.
[2009-10-16 22:36:22 2997] DEBUG (XendDomainInfo:1709) XendDomainInfo.handleShutdownWatch
[2009-10-16 22:36:22 2997] DEBUG (DevController:139) Waiting for devices vif.
[2009-10-16 22:36:22 2997] DEBUG (DevController:144) Waiting for 0.
[2009-10-16 22:36:22 2997] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/437/0/hotplug-status.
[2009-10-16 22:38:02 2997] DEBUG (XendDomainInfo:2724) XendDomainInfo.destroy: domid=437
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2199) Destroying device model
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2206) Releasing devices
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2219) Removing vif/0
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:1134) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2219) Removing vbd/51712
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:1134) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2219) Removing vbd/51776
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:1134) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51776
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2219) Removing console/0
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:1134) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2204) No device model
[2009-10-16 22:38:03 2997] DEBUG (XendDomainInfo:2206) Releasing devices
and this is in /var/log/xen/xend-debug.log from the same time
Traceback (most recent call last):
File "/usr/lib64/python2.4/SocketServer.py", line 463, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.4/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 105, in <lambda>
(lambda x, y, z:
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 65, in __init__
server)
File "/usr/lib64/python2.4/SocketServer.py", line 521, in __init__
self.handle()
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 310, in handle_one_request
method()
File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 82, in do_POST
self.send_response(200)
File "/usr/lib64/python2.4/BaseHTTPServer.py", line 367, in send_response
self.wfile.write("%s %d %s\r\n" %
File "/usr/lib64/python2.4/socket.py", line 256, in write
self.flush()
File "/usr/lib64/python2.4/socket.py", line 243, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
things we've tried that didn't help
xm destroy domain
restart xend.
brctl delif eth0 trouclec
see lots of:
(XEN) mm.c:739:d77 Non-privileged (77) attempt to map I/O space 000000f0 (XEN) mm.c:739:d77 Non-privileged (77) attempt to map I/O space 000000f0
which is okay, see: http://lists.xensource.com/archives/html/xen-devel/2007-05/msg00101.html
there are not extra domains in xenstore: (well, there is 1 assuming 0000 is dom0, because xm list has a label colum)
[root@branch ~]# xenstore-list /vm|wc -l 62 [root@branch ~]# xm list |wc -l 62
interesting
[root@branch ~]# xenstore-list /local/domain|wc -l 61
Not actually interesting because someone shut down between:
[root@branch ~]# xenstore-list /vm|wc -l 61 [root@branch ~]# xenstore-list /local/domain|wc -l 61
[root@branch ~]# for i in `xenstore-list /local/domain/0/backend/vif` ; do echo "reading domain ${i}:" ;xenstore-read /local/domain/0/backend/vif/${i}/0/vifname; done
reading domain 9:
muziboo
reading domain 10:
mikewest
reading domain 11:
stela
reading domain 16:
tims
reading domain 19:
mcmillen
reading domain 21:
erp4every1
reading domain 23:
kentaro
reading domain 28:
dasil003
reading domain 29:
bip.jurun
reading domain 30:
anu
reading domain 32:
radford
reading domain 35:
trafficlight
reading domain 38:
dbpatterson
reading domain 45:
cosa
reading domain 46:
oz
reading domain 55:
srobertson
reading domain 59:
osakichi
reading domain 63:
snak
reading domain 68:
toraneko
reading domain 69:
ttuttle2
reading domain 70:
icarus
reading domain 71:
david
reading domain 73:
jlugert
reading domain 75:
satendra
reading domain 77:
miron
reading domain 79:
aoustis
reading domain 82:
mveytsman
reading domain 93:
phunet
reading domain 117:
clayton
reading domain 118:
ml
reading domain 120:
larues
reading domain 137:
io
reading domain 149:
corbanj
reading domain 150:
mikeymckay
reading domain 151:
vonfreud
reading domain 156:
presto8
reading domain 158:
gregh
reading domain 163:
petrie
reading domain 171:
logan
reading domain 172:
njnear
reading domain 180:
mmagin
reading domain 181:
aliasd
reading domain 186:
soyapi
reading domain 187:
lxe
reading domain 191:
flight107
reading domain 192:
dodgio
reading domain 205:
kevin
reading domain 207:
rlasch
reading domain 215:
utaani
reading domain 229:
spoonie
reading domain 243:
vdx0610
reading domain 268:
hogetaro
reading domain 271:
ergoen
reading domain 273:
ematoda
reading domain 329:
atlan
reading domain 333:
mishelangelo
reading domain 334:
cyj
reading domain 336:
dothername
reading domain 343:
rockettools
reading domain 347:
rippond
reading domain 354:
trouclec
reading domain 355:
renegade
reading domain 364:
aepiak
reading domain 374:
jfoong
reading domain 375:
jfoong
reading domain 376:
jfoong
reading domain 377:
jfoong
reading domain 378:
jfoong
reading domain 379:
jfoong
reading domain 380:
jfoong
reading domain 381:
jfoong
reading domain 382:
jfoong
reading domain 383:
jfoong
reading domain 384:
jfoong
reading domain 385:
jfoong
reading domain 386:
renegade
reading domain 387:
renegade
reading domain 388:
renegade
reading domain 389:
renegade
reading domain 390:
renegade
reading domain 391:
renegade
reading domain 392:
renegade
reading domain 393:
ren9gade
reading domain 394:
ren9gade
reading domain 395:
ren9gade
reading domain 396:
ren9gade
reading domain 397:
jfoong
reading domain 398:
jfoong
reading domain 399:
jfoong
reading domain 400:
ren9gade
reading domain 401:
ren9gade
reading domain 402:
ren9gade
reading domain 403:
ren9gade
reading domain 404:
jfoong
reading domain 405:
jfoong
reading domain 406:
ren9gade
reading domain 407:
jfoong
reading domain 408:
anu
reading domain 409:
anu
reading domain 410:
anu
reading domain 411:
anu
reading domain 412:
anu
reading domain 413:
anu
reading domain 414:
anu
reading domain 415:
jfoong
reading domain 416:
anu
reading domain 417:
jfoong
reading domain 418:
ren9gade
reading domain 419:
trouclec
reading domain 420:
trouclec
reading domain 421:
trouclec
reading domain 422:
trouclec
reading domain 423:
rippond
reading domain 424:
rippond
reading domain 425:
rippond
reading domain 426:
rippond
reading domain 427:
rippond
reading domain 428:
rippond
reading domain 429:
rippond
reading domain 430:
rippond
reading domain 431:
rippond
reading domain 432:
rippond
reading domain 433:
rippond
reading domain 434:
rippond
reading domain 435:
trouclec
reading domain 436:
trouclec
reading domain 437:
rippond
reading domain 438:
rippond
reading domain 439:
trouclec
reading domain 440:
trouclec
reading domain 441:
rippond
Our Problem
we have zombies:
[root@branch ~]# xenstore-list /local/domain/0/backend/vbd|wc -l 131 [root@branch ~]# xenstore-list /local/domain/0/backend/vif|wc -l 131 [root@branch ~]# xenstore-list /local/domain/0/backend/console|wc -l 131 [root@branch ~]# xenstore-list /local/domain|wc -l 61
