New cluster put together to fix current issues with cman leads to new issues from clean iso installation.
- Node 2 of new cluster does not run KVM. this is somewhat acceptable but still an issue. How can I troubleshoot why KVM does not work on this box? Seems the KVM kernel module either doesnt work or did not install correctly.
Is is possible to manually remove reinstall this kernel module? (FYI: This node currently has running cts on it so clean install is not a great option)
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
TASK ERROR: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
- Node 2 & 3 while both show green.... the status of all vms / cts on the nodes dont show at all... there all greyed out even tho they are running.
I have restarted pvestatd / pve-cluster / pvedaemon but was not able to get the vm status to show up...
- Node 2 & 3 while both show green, when using the API to auto provision vms only provisions on node 1.
I need to do more research on this but this might be related to my previous issue. If i solve the one above, i am hoping this one will be solved, in the case the top issue gets solved and this one does not i will work on getting this issue solved.
#######
Copy of Syslogs from node 2 & 3 for troubleshooting purposes:
NODE 2:
Apr 15 11:05:08 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:34 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:48 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:09:26 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:10:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:12:09 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:12:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:13:49 proxmox2 pvedaemon[232518]: <root@pam> starting task Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam:
Apr 15 11:13:49 proxmox2 pvedaemon[721658]: start VM 202: Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam:
Apr 15 11:13:50 proxmox2 pvedaemon[721658]: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Apr 15 11:13:50 proxmox2 pvedaemon[232518]: <root@pam> end task Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Apr 15 11:14:08 proxmox2 pvedaemon[539313]: <root@pam> starting task Proxmox2:000B0407:01F118C5:516C2750:srvrestartveda emon:root@pam:
Apr 15 11:14:08 proxmox2 pvedaemon[721927]: re-starting service pvedaemon: Proxmox2:000B0407:01F118C5:516C2750:srvrestartveda emon:root@pam:
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: received terminate request
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 409506 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 539313 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 232518 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: server closing
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: starting server
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: starting 3 worker(s)
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721934 started
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721935 started
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721936 started
Apr 15 11:14:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:32 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:41 proxmox2 pvestatd[1671]: server closing
Apr 15 11:14:41 proxmox2 pvestatd[722381]: starting server
Apr 15 11:14:59 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:15:39 proxmox2 pvedaemon[721936]: <root@pam> starting task Proxmox2:000B084B:01F13C51:516C27AB:qmigrate:202:r oot@pam:
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/template: -1
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: -1
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/local: -1
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/template) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/template: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/local: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:17:01 proxmox2 /USR/SBIN/CRON[724054]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 15 11:18:54 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:19:00 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:20:11 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:25:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:27:09 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:27:34 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:28:02 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:32:25 proxmox2 pvedaemon[721936]: <root@pam> end task Proxmox2:000B084B:01F13C51:516C27AB:qmigrate:202:r oot@pam: OK
NODE 3:
Apr 15 11:03:55 proxmox3 pvedaemon[1830]: worker 137241 finished
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 137529 finished
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: starting 2 worker(s)
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 146739 started
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 146740 started
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: worker 137671 finished
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: starting 1 worker(s)
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: worker 146888 started
Apr 15 11:05:09 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:08:29 proxmox3 pvedaemon[149271]: stop VM 129: Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam:
Apr 15 11:08:29 proxmox3 pvedaemon[146740]: <root@pam> starting task Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam:
Apr 15 11:08:30 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:08:30 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:08:34 proxmox3 pvedaemon[146740]: <root@pam> end task Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam: OK
Apr 15 11:08:47 proxmox3 ntpd[1269]: Deleting interface #34 tap129i0, fe80::1c84:6ff:feda:aae2#123, interface stats: received=0, sent=0, dropped=0, active_time=74400 secs
Apr 15 11:08:48 proxmox3 pvedaemon[146740]: <root@pam> starting task Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam:
Apr 15 11:08:48 proxmox3 pvedaemon[149428]: start VM 129: Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam:
Apr 15 11:09:24 proxmox3 pvestatd[1861]: WARNING: unable to connect to VM 129 socket - timeout after 31 retries
Apr 15 11:09:26 proxmox3 pvedaemon[149428]: start failed: command '/usr/bin/kvm -id 129 -chardev 'socket,id=qmp,path=/var/run/qemu-server/129.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/129.vnc,x509,password -pidfile /var/run/qemu-server/129.pid -daemonize -name PVM129 -smp 'sockets=3,cores=1' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 1536 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/template/iso/W2k8-R2-all.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/129/vm-129-disk-1.raw,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap129i0,script=/var/lib/qemu-server/pve-bridge' -device 'rtl8139,mac=02:79:53:9B:E8:ED,netdev=net0,bus=pci .0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout
Apr 15 11:09:26 proxmox3 pvedaemon[146740]: <root@pam> end task Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam: unexpected status
Apr 15 11:09:31 proxmox3 kernel: device tap129i0 entered promiscuous mode
Apr 15 11:09:31 proxmox3 kernel: vmbr0: port 2(tap129i0) entering forwarding state
Apr 15 11:09:32 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:09:32 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:09:32 proxmox3 pvestatd[1861]: WARNING: unable to connect to VM 129 socket - Connection refused
Apr 15 11:10:29 proxmox3 pvedaemon[146740]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:12:09 proxmox3 pvedaemon[146888]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:12:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:13:06 proxmox3 rrdcached[1243]: flushing old values
Apr 15 11:13:06 proxmox3 rrdcached[1243]: rotating journals
Apr 15 11:13:06 proxmox3 rrdcached[1243]: started new journal /var/lib/rrdcached/journal//rrd.journal.1366042386.696439
Apr 15 11:13:06 proxmox3 rrdcached[1243]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1366035186.696466
Apr 15 11:13:49 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:13:50 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:08 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:32 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:59 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:15:39 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/template: -1
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: -1
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/local: -1
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/local: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/template) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/template: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:17:01 proxmox3 /USR/SBIN/CRON[153447]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 15 11:18:54 proxmox3 pvedaemon[154449]: start VM 129: Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam:
Apr 15 11:18:54 proxmox3 pvedaemon[146739]: <root@pam> starting task Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam:
Apr 15 11:18:58 proxmox3 kernel: device tap129i0 entered promiscuous mode
Apr 15 11:18:59 proxmox3 kernel: vmbr0: port 2(tap129i0) entering forwarding state
Apr 15 11:19:00 proxmox3 pvedaemon[146739]: <root@pam> end task Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam: OK
Apr 15 11:19:09 proxmox3 kernel: tap129i0: no IPv6 routers present
Apr 15 11:20:11 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:23:47 proxmox3 ntpd[1269]: Listen normally on 35 tap129i0 fe80::4c96:b6ff:fe4a:4bf7 UDP 123
Apr 15 11:25:29 proxmox3 pvedaemon[146740]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:27:09 proxmox3 pvedaemon[146739]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:27:34 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:28:02 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:32:25 proxmox3 pmxcfs[135010]: [status] notice: received log
######
- Node 2 of new cluster does not run KVM. this is somewhat acceptable but still an issue. How can I troubleshoot why KVM does not work on this box? Seems the KVM kernel module either doesnt work or did not install correctly.
Is is possible to manually remove reinstall this kernel module? (FYI: This node currently has running cts on it so clean install is not a great option)
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
TASK ERROR: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
- Node 2 & 3 while both show green.... the status of all vms / cts on the nodes dont show at all... there all greyed out even tho they are running.
I have restarted pvestatd / pve-cluster / pvedaemon but was not able to get the vm status to show up...
- Node 2 & 3 while both show green, when using the API to auto provision vms only provisions on node 1.
I need to do more research on this but this might be related to my previous issue. If i solve the one above, i am hoping this one will be solved, in the case the top issue gets solved and this one does not i will work on getting this issue solved.
#######
Copy of Syslogs from node 2 & 3 for troubleshooting purposes:
NODE 2:
Apr 15 11:05:08 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:34 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:08:48 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:09:26 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:10:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:12:09 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:12:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:13:49 proxmox2 pvedaemon[232518]: <root@pam> starting task Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam:
Apr 15 11:13:49 proxmox2 pvedaemon[721658]: start VM 202: Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam:
Apr 15 11:13:50 proxmox2 pvedaemon[721658]: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Apr 15 11:13:50 proxmox2 pvedaemon[232518]: <root@pam> end task Proxmox2:000B02FA:01F11169:516C273D:qmstart:202:ro ot@pam: start failed: command '/usr/bin/kvm -id 202 -chardev 'socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/202.vnc,x509,password -pidfile /var/run/qemu-server/202.pid -daemonize -name vm202 -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 8192 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/202/vm-202-disk-1.qcow2,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0' -netdev 'type=tap,id=net0,ifname=tap202i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,mac=6A:A6D:6C:BE:98,netdev=net0,bus=pci.0,a ddr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Apr 15 11:14:08 proxmox2 pvedaemon[539313]: <root@pam> starting task Proxmox2:000B0407:01F118C5:516C2750:srvrestartveda emon:root@pam:
Apr 15 11:14:08 proxmox2 pvedaemon[721927]: re-starting service pvedaemon: Proxmox2:000B0407:01F118C5:516C2750:srvrestartveda emon:root@pam:
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: received terminate request
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 409506 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 539313 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: worker 232518 finished
Apr 15 11:14:08 proxmox2 pvedaemon[2053]: server closing
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: starting server
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: starting 3 worker(s)
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721934 started
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721935 started
Apr 15 11:14:11 proxmox2 pvedaemon[721932]: worker 721936 started
Apr 15 11:14:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:20 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:32 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:14:41 proxmox2 pvestatd[1671]: server closing
Apr 15 11:14:41 proxmox2 pvestatd[722381]: starting server
Apr 15 11:14:59 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:15:39 proxmox2 pvedaemon[721936]: <root@pam> starting task Proxmox2:000B084B:01F13C51:516C27AB:qmigrate:202:r oot@pam:
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/template: -1
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: -1
Apr 15 11:15:52 proxmox2 pmxcfs[1819]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/local: -1
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/template) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/template: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox2 rrdcached[1276]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/local: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:17:01 proxmox2 /USR/SBIN/CRON[724054]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 15 11:18:54 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:19:00 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:20:11 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:25:29 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:27:09 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:27:34 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:28:02 proxmox2 pmxcfs[1819]: [status] notice: received log
Apr 15 11:32:25 proxmox2 pvedaemon[721936]: <root@pam> end task Proxmox2:000B084B:01F13C51:516C27AB:qmigrate:202:r oot@pam: OK
NODE 3:
Apr 15 11:03:55 proxmox3 pvedaemon[1830]: worker 137241 finished
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 137529 finished
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: starting 2 worker(s)
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 146739 started
Apr 15 11:03:56 proxmox3 pvedaemon[1830]: worker 146740 started
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: worker 137671 finished
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: starting 1 worker(s)
Apr 15 11:04:20 proxmox3 pvedaemon[1830]: worker 146888 started
Apr 15 11:05:09 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:08:29 proxmox3 pvedaemon[149271]: stop VM 129: Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam:
Apr 15 11:08:29 proxmox3 pvedaemon[146740]: <root@pam> starting task Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam:
Apr 15 11:08:30 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:08:30 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:08:34 proxmox3 pvedaemon[146740]: <root@pam> end task Proxmox3:00024717:01EDFA74:516C25FC:qmstop:129:roo t@pam: OK
Apr 15 11:08:47 proxmox3 ntpd[1269]: Deleting interface #34 tap129i0, fe80::1c84:6ff:feda:aae2#123, interface stats: received=0, sent=0, dropped=0, active_time=74400 secs
Apr 15 11:08:48 proxmox3 pvedaemon[146740]: <root@pam> starting task Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam:
Apr 15 11:08:48 proxmox3 pvedaemon[149428]: start VM 129: Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam:
Apr 15 11:09:24 proxmox3 pvestatd[1861]: WARNING: unable to connect to VM 129 socket - timeout after 31 retries
Apr 15 11:09:26 proxmox3 pvedaemon[149428]: start failed: command '/usr/bin/kvm -id 129 -chardev 'socket,id=qmp,path=/var/run/qemu-server/129.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/129.vnc,x509,password -pidfile /var/run/qemu-server/129.pid -daemonize -name PVM129 -smp 'sockets=3,cores=1' -nodefaults -boot 'menu=on' -vga std -no-hpet -k en-us -m 1536 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/template/iso/W2k8-R2-all.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/129/vm-129-disk-1.raw,if=none,id=drive-ide0,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap129i0,script=/var/lib/qemu-server/pve-bridge' -device 'rtl8139,mac=02:79:53:9B:E8:ED,netdev=net0,bus=pci .0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout
Apr 15 11:09:26 proxmox3 pvedaemon[146740]: <root@pam> end task Proxmox3:000247B4:01EE0256:516C2610:qmstart:129:ro ot@pam: unexpected status
Apr 15 11:09:31 proxmox3 kernel: device tap129i0 entered promiscuous mode
Apr 15 11:09:31 proxmox3 kernel: vmbr0: port 2(tap129i0) entering forwarding state
Apr 15 11:09:32 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:09:32 proxmox3 kernel: vmbr0: port 2(tap129i0) entering disabled state
Apr 15 11:09:32 proxmox3 pvestatd[1861]: WARNING: unable to connect to VM 129 socket - Connection refused
Apr 15 11:10:29 proxmox3 pvedaemon[146740]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:12:09 proxmox3 pvedaemon[146888]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:12:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:13:06 proxmox3 rrdcached[1243]: flushing old values
Apr 15 11:13:06 proxmox3 rrdcached[1243]: rotating journals
Apr 15 11:13:06 proxmox3 rrdcached[1243]: started new journal /var/lib/rrdcached/journal//rrd.journal.1366042386.696439
Apr 15 11:13:06 proxmox3 rrdcached[1243]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1366035186.696466
Apr 15 11:13:49 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:13:50 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:08 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:20 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:32 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:14:59 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:15:39 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/template: -1
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: -1
Apr 15 11:15:53 proxmox3 pmxcfs[135010]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmox/local: -1
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/xtemplates: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/local: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:16:50 proxmox3 rrdcached[1243]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/proxmox/template) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/proxmox/template: illegal attempt to update using time 1366042300 when last update time is 1366042552 (minimum one second step))
Apr 15 11:17:01 proxmox3 /USR/SBIN/CRON[153447]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 15 11:18:54 proxmox3 pvedaemon[154449]: start VM 129: Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam:
Apr 15 11:18:54 proxmox3 pvedaemon[146739]: <root@pam> starting task Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam:
Apr 15 11:18:58 proxmox3 kernel: device tap129i0 entered promiscuous mode
Apr 15 11:18:59 proxmox3 kernel: vmbr0: port 2(tap129i0) entering forwarding state
Apr 15 11:19:00 proxmox3 pvedaemon[146739]: <root@pam> end task Proxmox3:00025B51:01EEEF08:516C286E:qmstart:129:ro ot@pam: OK
Apr 15 11:19:09 proxmox3 kernel: tap129i0: no IPv6 routers present
Apr 15 11:20:11 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:23:47 proxmox3 ntpd[1269]: Listen normally on 35 tap129i0 fe80::4c96:b6ff:fe4a:4bf7 UDP 123
Apr 15 11:25:29 proxmox3 pvedaemon[146740]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:27:09 proxmox3 pvedaemon[146739]: <root@pam> successful auth for user 'root@pam'
Apr 15 11:27:34 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:28:02 proxmox3 pmxcfs[135010]: [status] notice: received log
Apr 15 11:32:25 proxmox3 pmxcfs[135010]: [status] notice: received log
######