cman running ... will not stop or restart

All nodes are online and accessible via SSH.

cman is said to be running on all nodes...


service cman stop fails

service cman restart fails on cman stop..


clustat output:

root@proxmox11:/var/downloads# clustat
Cluster Status for FL-Cluster @ Sun Mar 3 16:02:05 2013
Member Status: Inquorate


Member Name ID Status
------ ---- ---- ------
proxmox11 1 Online, Local
proxmox2 2 Offline
proxmox3a 3 Offline
proxmox4 4 Offline
poxmox5 5 Offline
proxmox6 6 Offline
proxmox7 7 Offline
proxmox8 8 Offline
proxmox9 9 Offline
Proxmox10 10 Offline

All nodes when local show online all other nodes are considered offline.

service pve-cluster restart is not fixing any issue with all nodes showing as offline.

Any suggestions?

- - - Updated - - -

root@proxmox11:/var/downloads# cat /var/log/syslog | tail
Mar 3 16:13:56 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 60
Mar 3 16:13:57 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 70
Mar 3 16:13:58 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 80
Mar 3 16:13:59 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 90
Mar 3 16:14:00 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 100
Mar 3 16:14:00 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retried 100 times
Mar 3 16:14:00 proxmox11 pmxcfs[1905]: [status] crit: cpg_send_message failed: 6
Mar 3 16:14:01 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 10
Mar 3 16:14:02 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 20
Mar 3 16:14:03 proxmox11 pmxcfs[1905]: [status] notice: cpg_send_message retry 30