/etc/pve/nodes/****/openvz - frozen / not responding

all nodes on the cluster are not able to be managed at this point. Seems to be some issue with the pve mount.

While i can access /etc/pve.... i have issues once getting to the nodes/****/openvz directory

if i run an ls on this directory for any of my nodes.... it hangs.


I cannot provision, delete, or modify any vms on the cluster now.... not sure what the issue is.


#############

Troubleshooting:

I went to cluster.conf and see node 11 is id 1 so i rebooted node 11 thinking that for some reason there was issues reading the pve mount from the master node... but after reboot the problem exists.

All nodes are online ( green ) but we cannot make any adjustments to the cluster


#############

output of cluster.conf

<?xml version="1.0"?>
<cluster config_version="49" name="FL-Cluster">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey"/>
<clusternodes>




<clusternode name="proxmox4" votes="1" nodeid="4"/>
<clusternode name="poxmox5" votes="1" nodeid="5"/>
<clusternode name="proxmox6" votes="1" nodeid="6"/>
<clusternode name="proxmox7" votes="1" nodeid="7"/>
<clusternode name="proxmox9" votes="1" nodeid="9"/>
<clusternode name="Proxmox10" votes="1" nodeid="10"/>
<clusternode name="proxmox8" votes="1" nodeid="8"/>


<clusternode name="proxmox11" votes="1" nodeid="1"/>
<clusternode name="proxmox2" votes="1" nodeid="2"/>


<clusternode name="proxmox3a" votes="1" nodeid="3"/></clusternodes>
<rm/>
</cluster>



output of mount:

root@proxmox2:~# mount
/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
none on /sys/kernel/config type configfs (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)

#############

nothing in syslogs

#############

root@proxmox2:~# cat /var/log/syslog | tail
Feb 28 13:15:00 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:15:00 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:17:01 proxmox2 /USR/SBIN/CRON[788634]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 28 13:18:56 proxmox2 pvedaemon[525098]: <root@pam> successful auth for user 'root@pam'
Feb 28 13:27:12 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:28:24 proxmox2 rrdcached[1761]: flushing old values
Feb 28 13:28:24 proxmox2 rrdcached[1761]: rotating journals
Feb 28 13:28:24 proxmox2 rrdcached[1761]: started new journal /var/lib/rrdcached/journal//rrd.journal.1362076104.073732
Feb 28 13:28:24 proxmox2 rrdcached[1761]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1362068904.074127
Feb 28 13:29:21 proxmox2 pmxcfs[525189]: [status] notice: received log