Quantcast
Viewing all articles
Browse latest Browse all 171010

Old Proxmox installation and problem with backup

Hello to all.

Yes i know i must upgrade the host, but this machine it a very critical machine running 24/7/365 and isolated from the corp NET.
Now i have a problem, last day during the backup of the second visrtual machine something goes very bad.

Here is the log:

Jun 12 03:44:09 INFO: Starting Backup of VM 104 (qemu)
Jun 12 03:44:09 INFO: running
Jun 12 03:44:09 INFO: status = running
Jun 12 03:44:09 INFO: backup mode: snapshot
Jun 12 03:44:09 INFO: ionice priority: 7
Jun 12 03:44:09 INFO: suspend vm to make snapshot
Jun 12 03:44:10 INFO: Logical volume "vzsnap-proxmox92-0" created
Jun 12 03:44:10 INFO: resume vm
Jun 12 03:44:10 INFO: vm is online again after 1 seconds
Jun 12 03:44:10 INFO: creating archive '/mnt/backup/proxmox92/vzdump-qemu-104-2013_06_12-03_44_09.tgz'




And from kern.log

Jun 12 03:44:10 proxmox92 kernel: EXT3-fs: barriers disabled
Jun 12 03:44:10 proxmox92 kernel: kjournald starting. Commit interval 5 seconds
Jun 12 04:00:23 proxmox92 kernel: r8169 0000:02:00.0: eth0: link up
Jun 12 04:19:20 proxmox92 kernel: r8169 0000:02:00.0: eth0: link up
Jun 12 04:37:19 proxmox92 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 12 04:37:19 proxmox92 kernel: __ratelimit: 61187 callbacks suppressed
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888699
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888700
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888701
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888702
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888703
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888704
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888705
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888706
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888707
Jun 12 04:37:24 proxmox92 kernel: __ratelimit: 4380356 callbacks suppressed
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: __ratelimit: 4366968 callbacks suppressed
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: __ratelimit: 4357097 callbacks suppressed
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: __ratelimit: 4393335 callbacks suppressed
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: __ratelimit: 4379377 callbacks suppressed
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: __ratelimit: 4275234 callbacks suppressed
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 06:25:01 proxmox92 kernel: Kernel logging (proc) stopped.
Jun 12 06:25:01 proxmox92 kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jun 12 23:03:48 proxmox92 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.


i am stoped (SIGKILL) then all backup process (vzdump, vmtar,etc) and unlocked the vm. Then i umount the vzsnapshots (one of then shows me a error notice). and when i go to remove the snapshots i found this:

$ lvs
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
data pve -wi-ao 311.41G
hdsiam pve owi-ao 60.00G
root pve -wi-ao 96.00G
swap pve -wi-ao 7.00G
vzsnap-proxmox92-0 pve Swi-I- 1.00G hdsiam 100.00

$pvscan
/dev/dm-4: read failed after 0 of 4096 at 64424443904: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 64424501248: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
PV /dev/sda2 VG pve lvm2 [558.41 GB / 82.99 GB free]
Total: 1 [558.41 GB] / in use: 1 [558.41 GB] / in no VG: 0 [0 ]

$lvscan
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
ACTIVE '/dev/pve/swap' [7.00 GB] inherit
ACTIVE '/dev/pve/root' [96.00 GB] inherit
ACTIVE '/dev/pve/data' [311.41 GB] inherit
inactive Original '/dev/pve/hdsiam' [60.00 GB] inherit
inactive Snapshot '/dev/pve/vzsnap-proxmox92-0' [1.00 GB] inherit


And now, here is the problem, I think the snapshot goes full during backup (i dont know how) but th VM is running in production without stop, will i encounter any problem removing the snapshot?, where is storing the changes sinde the date of the crash. The hd of the vm in producction is the /dev/pve/hdsiam wich is "inactive". Must i activate manually(vgchange -ay) the volume /dev/pve/hdsiam?

What will be the best practiques to recover from this state?



PD. other info:
pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 1.9-55+ovzfix-2
pve-kernel-2.6.32-6-pve: 2.6.32-55+ovzfix-1
pve-kernel-2.6.32-7-pve: 2.6.32-55+ovzfix-2
qemu-server: 1.1-32
pve-firmware: 1.0-15
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-2
ksm-control-daemon: 1.0-6





Lots of thanks.

Viewing all articles
Browse latest Browse all 171010

Trending Articles