Hi,
I recently upgraded to the latest versions, and after I rebooted into the 2.6.32-18-pve kernel, I started getting errors while accessing a CIFS share. These errors would seem recoverable, but after a few hours passed, the proxmox host (and all the containers and virtual machines on it) would start to crawl until the machine had to be rebooted.
I also noticed some kernel errors in dmesg just before the CIFS errors start.
Rebooting back into 2.6.32-17-pve solves the problem.
From the error messages, I'm guessing there's some sort of bad interaction between the new Broadcom bnx2/bnx2x driver and the CIFS driver.
PVEVersion:
# pveversion -v
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.3-88
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-18-pve: 2.6.32-88
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-17
pve-firmware: 1.0-21
libpve-common-perl: 1.0-48
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-6
ksm-control-daemon: 1.1-1
The CIFS error I am seeing. This error appears hundreds of times in the /var/log/kern.log files when running 2.6.32-18-pve
Mar 11 00:06:05 host kernel: CIFS VFS: Unexpected lookup error -512
Kernel errors I'm seeing in dmesg:
Mar 11 10:10:29 host kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Mar 11 10:10:29 host kernel: ------------[ cut here ]------------
Mar 11 10:10:29 host kernel: WARNING: at drivers/pci/intel-iommu.c:2775 intel_unmap_page+0x15f/0x180() (Not tainted)
Mar 11 10:10:29 host kernel: Hardware name: System x3550 M3 -[7944KEG]-
Mar 11 10:10:29 host kernel: Driver unmaps unmatched page at PFN 0
Mar 11 10:10:29 host kernel: Modules linked in: nls_utf8 cifs bonding ipv6 8021q garp ext4 jbd2 dm_snapshot snd_pcsp snd_pcm snd_timer snd cdc_ether usbnet soundcore mii i2c_i801 snd_page_alloc i7core_edac serio_raw ioatdma edac_core tpm_tis i2c_core dca tpm tpm_bios shpchp ext3 jbd mbcache ata_generic pata_acpi ata_piix megaraid_sas bnx2 [last unloaded: scsi_wait_scan]
Mar 11 10:10:29 host kernel: Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-18-pve #1
Mar 11 10:10:29 host kernel: Call Trace:
Mar 11 10:10:29 host kernel: <IRQ> [<ffffffff8106d228>] ? warn_slowpath_common+0x88/0xc0
Mar 11 10:10:29 host kernel: [<ffffffff8106d316>] ? warn_slowpath_fmt+0x46/0x50
Mar 11 10:10:29 host kernel: [<ffffffff812a6cdb>] ? find_iova+0x5b/0x90
Mar 11 10:10:29 host kernel: [<ffffffff812aaf1f>] ? intel_unmap_page+0x15f/0x180
Mar 11 10:10:29 host kernel: [<ffffffffa000ca65>] ? bnx2_poll_work+0x155/0x11d0 [bnx2]
Mar 11 10:10:29 host kernel: [<ffffffff810eac20>] ? handle_IRQ_event+0x60/0x170
Mar 11 10:10:29 host kernel: [<ffffffff810ed308>] ? handle_edge_irq+0x98/0x180
Mar 11 10:10:29 host kernel: [<ffffffff810a18f3>] ? ktime_get+0x63/0xe0
Mar 11 10:10:29 host kernel: [<ffffffff8102b32d>] ? lapic_next_event+0x1d/0x30
Mar 11 10:10:29 host kernel: [<ffffffffa000db1d>] ? bnx2_poll_msix+0x3d/0xd0 [bnx2]
Mar 11 10:10:29 host kernel: [<ffffffff81457aa3>] ? net_rx_action+0x103/0x2e0
Mar 11 10:10:29 host kernel: [<ffffffff81076063>] ? __do_softirq+0x103/0x260
Mar 11 10:10:29 host kernel: [<ffffffff8100c2ac>] ? call_softirq+0x1c/0x30
Mar 11 10:10:29 host kernel: [<ffffffff8100def5>] ? do_softirq+0x65/0xa0
Mar 11 10:10:29 host kernel: [<ffffffff81075e8d>] ? irq_exit+0xcd/0xd0
Mar 11 10:10:29 host kernel: [<ffffffff81524935>] ? do_IRQ+0x75/0xf0
Mar 11 10:10:29 host kernel: [<ffffffff8100ba93>] ? ret_from_intr+0x0/0x11
Mar 11 10:10:29 host kernel: <EOI> [<ffffffff812d0e7e>] ? intel_idle+0xde/0x170
Mar 11 10:10:29 host kernel: [<ffffffff812d0e61>] ? intel_idle+0xc1/0x170
Mar 11 10:10:29 host kernel: [<ffffffff8109deed>] ? sched_clock_cpu+0xcd/0x110
Mar 11 10:10:29 host kernel: [<ffffffff81420387>] ? cpuidle_idle_call+0xa7/0x140
Mar 11 10:10:29 host kernel: [<ffffffff8100a023>] ? cpu_idle+0xb3/0x110
Mar 11 10:10:29 host kernel: [<ffffffff81503e35>] ? rest_init+0x85/0x90
Mar 11 10:10:29 host kernel: [<ffffffff81c2ef6e>] ? start_kernel+0x412/0x41e
Mar 11 10:10:29 host kernel: [<ffffffff81c2e33a>] ? x86_64_start_reservations+0x125/0x129
Mar 11 10:10:29 host kernel: [<ffffffff81c2e438>] ? x86_64_start_kernel+0xfa/0x109
Mar 11 10:10:29 host kernel: ---[ end trace 796ec5b2544d699d ]---
Mar 11 10:10:29 host kernel: bnx2 0000:0b:00.1: eth1: NIC Copper Link is Up, 1000 Mbps full duplex
Is there any other information I can post to help resolve this issue?
I recently upgraded to the latest versions, and after I rebooted into the 2.6.32-18-pve kernel, I started getting errors while accessing a CIFS share. These errors would seem recoverable, but after a few hours passed, the proxmox host (and all the containers and virtual machines on it) would start to crawl until the machine had to be rebooted.
I also noticed some kernel errors in dmesg just before the CIFS errors start.
Rebooting back into 2.6.32-17-pve solves the problem.
From the error messages, I'm guessing there's some sort of bad interaction between the new Broadcom bnx2/bnx2x driver and the CIFS driver.
PVEVersion:
Quote:
# pveversion -v
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.3-88
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-18-pve: 2.6.32-88
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-17
pve-firmware: 1.0-21
libpve-common-perl: 1.0-48
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-6
ksm-control-daemon: 1.1-1
Quote:
Mar 11 00:06:05 host kernel: CIFS VFS: Unexpected lookup error -512
Quote:
Mar 11 10:10:29 host kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Mar 11 10:10:29 host kernel: ------------[ cut here ]------------
Mar 11 10:10:29 host kernel: WARNING: at drivers/pci/intel-iommu.c:2775 intel_unmap_page+0x15f/0x180() (Not tainted)
Mar 11 10:10:29 host kernel: Hardware name: System x3550 M3 -[7944KEG]-
Mar 11 10:10:29 host kernel: Driver unmaps unmatched page at PFN 0
Mar 11 10:10:29 host kernel: Modules linked in: nls_utf8 cifs bonding ipv6 8021q garp ext4 jbd2 dm_snapshot snd_pcsp snd_pcm snd_timer snd cdc_ether usbnet soundcore mii i2c_i801 snd_page_alloc i7core_edac serio_raw ioatdma edac_core tpm_tis i2c_core dca tpm tpm_bios shpchp ext3 jbd mbcache ata_generic pata_acpi ata_piix megaraid_sas bnx2 [last unloaded: scsi_wait_scan]
Mar 11 10:10:29 host kernel: Pid: 0, comm: swapper veid: 0 Not tainted 2.6.32-18-pve #1
Mar 11 10:10:29 host kernel: Call Trace:
Mar 11 10:10:29 host kernel: <IRQ> [<ffffffff8106d228>] ? warn_slowpath_common+0x88/0xc0
Mar 11 10:10:29 host kernel: [<ffffffff8106d316>] ? warn_slowpath_fmt+0x46/0x50
Mar 11 10:10:29 host kernel: [<ffffffff812a6cdb>] ? find_iova+0x5b/0x90
Mar 11 10:10:29 host kernel: [<ffffffff812aaf1f>] ? intel_unmap_page+0x15f/0x180
Mar 11 10:10:29 host kernel: [<ffffffffa000ca65>] ? bnx2_poll_work+0x155/0x11d0 [bnx2]
Mar 11 10:10:29 host kernel: [<ffffffff810eac20>] ? handle_IRQ_event+0x60/0x170
Mar 11 10:10:29 host kernel: [<ffffffff810ed308>] ? handle_edge_irq+0x98/0x180
Mar 11 10:10:29 host kernel: [<ffffffff810a18f3>] ? ktime_get+0x63/0xe0
Mar 11 10:10:29 host kernel: [<ffffffff8102b32d>] ? lapic_next_event+0x1d/0x30
Mar 11 10:10:29 host kernel: [<ffffffffa000db1d>] ? bnx2_poll_msix+0x3d/0xd0 [bnx2]
Mar 11 10:10:29 host kernel: [<ffffffff81457aa3>] ? net_rx_action+0x103/0x2e0
Mar 11 10:10:29 host kernel: [<ffffffff81076063>] ? __do_softirq+0x103/0x260
Mar 11 10:10:29 host kernel: [<ffffffff8100c2ac>] ? call_softirq+0x1c/0x30
Mar 11 10:10:29 host kernel: [<ffffffff8100def5>] ? do_softirq+0x65/0xa0
Mar 11 10:10:29 host kernel: [<ffffffff81075e8d>] ? irq_exit+0xcd/0xd0
Mar 11 10:10:29 host kernel: [<ffffffff81524935>] ? do_IRQ+0x75/0xf0
Mar 11 10:10:29 host kernel: [<ffffffff8100ba93>] ? ret_from_intr+0x0/0x11
Mar 11 10:10:29 host kernel: <EOI> [<ffffffff812d0e7e>] ? intel_idle+0xde/0x170
Mar 11 10:10:29 host kernel: [<ffffffff812d0e61>] ? intel_idle+0xc1/0x170
Mar 11 10:10:29 host kernel: [<ffffffff8109deed>] ? sched_clock_cpu+0xcd/0x110
Mar 11 10:10:29 host kernel: [<ffffffff81420387>] ? cpuidle_idle_call+0xa7/0x140
Mar 11 10:10:29 host kernel: [<ffffffff8100a023>] ? cpu_idle+0xb3/0x110
Mar 11 10:10:29 host kernel: [<ffffffff81503e35>] ? rest_init+0x85/0x90
Mar 11 10:10:29 host kernel: [<ffffffff81c2ef6e>] ? start_kernel+0x412/0x41e
Mar 11 10:10:29 host kernel: [<ffffffff81c2e33a>] ? x86_64_start_reservations+0x125/0x129
Mar 11 10:10:29 host kernel: [<ffffffff81c2e438>] ? x86_64_start_kernel+0xfa/0x109
Mar 11 10:10:29 host kernel: ---[ end trace 796ec5b2544d699d ]---
Mar 11 10:10:29 host kernel: bnx2 0000:0b:00.1: eth1: NIC Copper Link is Up, 1000 Mbps full duplex