I inherited a xen server that is setup to have all the VM images hosted on an iSCSI mounted NAS array. We been experiencing a random (about every 2-3 days) issue where xen would crash all the VMs, leaving nothing but the Domain0 running. What appears to be happening is something causes the iSCI mount to hiccup. Running "vgchange -a y" and restarting all the VMs brings everything up. Nothing appears to be wrong with the NAS array - there are a dozen other servers attached to it that never have a problem. The xend log does not have anything useful in it and I'm at a loss to figure out what is causing this. The only suggestion I've heard is maybe the memory usage is too high. If anyone has any suggestions, I'd really appreciate it.
Host OS: CentOS 5.5
Xen kernel: xen.gz-2.6.18-194.11.4.el5
iSCSI libraries: iscsi-initiator-utils-6.2.0.871-0.16.el5
Memory on server: 32G
Total memory allocated for VMs running paravirt: 19,384 M
Total memory allocated for VMs running HVM: 2,688 M
Results of xm top:
xentop - 10:11:06 Xen 3.1.2-194.11.4.el5
39 domains: 1 running, 38 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 25165116k total, 25150528k used, 14588k free CPUs: 8 @ 1995MHz