Hi all,
I have some network issues.. i think its because of loops but both RSTP and loop protection is active ..still i may have done it wrong.. anyway.. im asking here cause i need to know what proxmox normal behaviour is..
So i have 3 nodes with two interfaces in bond0 with lacp configured on each node - pretty standard..
I have two lan switches which each interface from each node is connected to. These two switches are also trunked with LACP.. I have tried with both HP static trunk and LACP - made no difference..
what im seeing aka symptoms..
sometimes when i boot a node or restart a node network service it all of sudden start to drop packets. and after some reboots i stops again. I quicker way to solve it is to boot one of the switches. My theroy is the following:
LACP makes only one interface active.. and im thinking in the packet drop scenario that 2 nodes has thire interface active on switch 1 and the last node has his interface active on switch 2 which in turn, because of the trunk between the two switches creates a loop.. does this make sense?
I went to the proxmox nodes and did a ifconfig to find the MACs.. here i noticed ALL my interfaces had the same mac
Proxmox00
Proxmox01
Proxmox02
I figure its because all my interfaces comes in one way or another from bond0 but still eth2 and eth3 on proxmox00 also has the same MAC - is this suppose to be that way?
LACP on the switch trunk side is configured in the following groups:
TRK1: switch trunk
TRK2: Storage NAS - not important
TRK3: BAckup Nas - not important
TRK4: Proxmox00
TRK5: Proxmox01
TRK6: Proxmox02
Here is the mac table of switch 1
Notice there are 3 entries for TRK6 and that the mac is the same for two of them and not all 3?
also i see allot of double mac entries for same trunk groups.. actually i expected to only see one MAC for each trunk group? is that normal proxmox behavior or could it be because of switch loops? Would a normal setup with a proxmox node connected to two switche using LACP look this way or?
THANKS
Casper
EDIT:
Here is mac table of switch 2
I have some network issues.. i think its because of loops but both RSTP and loop protection is active ..still i may have done it wrong.. anyway.. im asking here cause i need to know what proxmox normal behaviour is..
So i have 3 nodes with two interfaces in bond0 with lacp configured on each node - pretty standard..
I have two lan switches which each interface from each node is connected to. These two switches are also trunked with LACP.. I have tried with both HP static trunk and LACP - made no difference..
what im seeing aka symptoms..
sometimes when i boot a node or restart a node network service it all of sudden start to drop packets. and after some reboots i stops again. I quicker way to solve it is to boot one of the switches. My theroy is the following:
LACP makes only one interface active.. and im thinking in the packet drop scenario that 2 nodes has thire interface active on switch 1 and the last node has his interface active on switch 2 which in turn, because of the trunk between the two switches creates a loop.. does this make sense?
I went to the proxmox nodes and did a ifconfig to find the MACs.. here i noticed ALL my interfaces had the same mac
Proxmox00
Code:
root@proxmox00:~# ifconfig
bond0 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2712897 errors:0 dropped:0 overruns:0 frame:0
TX packets:1195000 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1835251437 (1.7 GiB) TX bytes:238454039 (227.4 MiB)
bond0.2 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1023654 errors:0 dropped:0 overruns:0 frame:0
TX packets:754 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1282156890 (1.1 GiB) TX bytes:49772 (48.6 KiB)
bond0.3 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:754 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:49772 (48.6 KiB)
eth2 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:2210271 errors:0 dropped:0 overruns:0 frame:0
TX packets:1148199 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1583448270 (1.4 GiB) TX bytes:232651227 (221.8 MiB)
eth3 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:502626 errors:0 dropped:0 overruns:0 frame:0
TX packets:46801 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:251803167 (240.1 MiB) TX bytes:5802812 (5.5 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:94 errors:0 dropped:0 overruns:0 frame:0
TX packets:94 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16339 (15.9 KiB) TX bytes:16339 (15.9 KiB)
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vmbr0 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
inet addr:10.10.99.20 Bcast:10.10.99.255 Mask:255.255.255.0
inet6 addr: fe80::82c1:6eff:fe64:8d3c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1125198 errors:0 dropped:0 overruns:0 frame:0
TX packets:1094891 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:264351866 (252.1 MiB) TX bytes:226508308 (216.0 MiB)
vmbr1 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
inet6 addr: fe80::82c1:6eff:fe64:8d3c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1272 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:58512 (57.1 KiB) TX bytes:578 (578.0 B)
vmbr3 Link encap:Ethernet HWaddr 80:c1:6e:64:8d:3c
inet6 addr: fe80::82c1:6eff:fe64:8d3c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:578 (578.0 B)
root@proxmox00:~#
Code:
root@proxmox01:~# ifconfig
bond0 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:78985596 errors:0 dropped:0 overruns:0 frame:0
TX packets:101958145 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:47848136947 (44.5 GiB) TX bytes:103293213480 (96.1 GiB)
bond0.2 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:11528034 errors:0 dropped:0 overruns:0 frame:0
TX packets:7606523 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9308896784 (8.6 GiB) TX bytes:13112974005 (12.2 GiB)
eth2 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:11207054 errors:0 dropped:0 overruns:0 frame:0
TX packets:1364502 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7576213855 (7.0 GiB) TX bytes:202885911 (193.4 MiB)
eth3 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:67778542 errors:0 dropped:0 overruns:0 frame:0
TX packets:100593643 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:40271923092 (37.5 GiB) TX bytes:103090327569 (96.0 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1208712 errors:0 dropped:0 overruns:0 frame:0
TX packets:1208712 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11790128367 (10.9 GiB) TX bytes:11790128367 (10.9 GiB)
tap101i0 Link encap:Ethernet HWaddr c2:15:aa:51:80:23
inet6 addr: fe80::c015:aaff:fe51:8023/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:1917199 errors:0 dropped:0 overruns:0 frame:0
TX packets:4189948 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:121642134 (116.0 MiB) TX bytes:5335729264 (4.9 GiB)
tap102i0 Link encap:Ethernet HWaddr 92:76:1b:e4:58:1f
inet6 addr: fe80::9076:1bff:fee4:581f/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:1 carrier:0
collisions:0 txqueuelen:500
RX bytes:300 (300.0 B) TX bytes:0 (0.0 B)
tap112i0 Link encap:Ethernet HWaddr 1a:b3:20:e5:5c:7b
inet6 addr: fe80::18b3:20ff:fee5:5c7b/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:1 carrier:0
collisions:0 txqueuelen:500
RX bytes:300 (300.0 B) TX bytes:0 (0.0 B)
tap113i0 Link encap:Ethernet HWaddr a6:7b:dc:97:43:16
inet6 addr: fe80::a47b:dcff:fe97:4316/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:420012 errors:0 dropped:0 overruns:0 frame:0
TX packets:18693350 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:219791105 (209.6 MiB) TX bytes:22361016498 (20.8 GiB)
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vmbr0 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
inet addr:10.10.99.21 Bcast:10.10.99.255 Mask:255.255.255.0
inet6 addr: fe80::ea39:35ff:feb7:c74c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:43488538 errors:0 dropped:0 overruns:0 frame:0
TX packets:39379002 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:32323093880 (30.1 GiB) TX bytes:86505250883 (80.5 GiB)
vmbr1 Link encap:Ethernet HWaddr e8:39:35:b7:c7:4c
inet6 addr: fe80::ea39:35ff:feb7:c74c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:28063 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1211608 (1.1 MiB) TX bytes:578 (578.0 B)
root@proxmox01:~#
Code:
root@proxmox02:~# ifconfig
bond0 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:44492827 errors:0 dropped:0 overruns:0 frame:0
TX packets:27420072 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14991834618 (13.9 GiB) TX bytes:4781766759 (4.4 GiB)
bond0.2 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:6646040 errors:0 dropped:0 overruns:0 frame:0
TX packets:15976 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3963772204 (3.6 GiB) TX bytes:1054424 (1.0 MiB)
eth2 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:15261688 errors:0 dropped:0 overruns:0 frame:0
TX packets:1036273 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8401169881 (7.8 GiB) TX bytes:131043042 (124.9 MiB)
eth3 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1
RX packets:29231139 errors:0 dropped:0 overruns:0 frame:0
TX packets:26383799 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6590664737 (6.1 GiB) TX bytes:4650723717 (4.3 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1158 errors:0 dropped:0 overruns:0 frame:0
TX packets:1158 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:657942 (642.5 KiB) TX bytes:657942 (642.5 KiB)
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vmbr0 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
inet addr:10.10.99.22 Bcast:10.10.99.255 Mask:255.255.255.0
inet6 addr: fe80::82c1:6eff:fe64:ab2a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27918154 errors:0 dropped:0 overruns:0 frame:0
TX packets:25346691 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6052590170 (5.6 GiB) TX bytes:4530105646 (4.2 GiB)
vmbr1 Link encap:Ethernet HWaddr 80:c1:6e:64:ab:2a
inet6 addr: fe80::82c1:6eff:fe64:ab2a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27945 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1285576 (1.2 MiB) TX bytes:578 (578.0 B)
root@proxmox02:~# ^C
LACP on the switch trunk side is configured in the following groups:
TRK1: switch trunk
TRK2: Storage NAS - not important
TRK3: BAckup Nas - not important
TRK4: Proxmox00
TRK5: Proxmox01
TRK6: Proxmox02
Here is the mac table of switch 1
Code:
MAC Address
Source Port
MAC Type
00:11:32:24:28:9b TRK2 Learned
00:11:32:24:28:d7 TRK3 Learned
00:11:32:24:28:da TRK1 Learned
00:90:fb:40:9c:f6 TRK1 Learned
00:e0:20:11:0a:2e 12 Learned
00:e0:20:11:0a:2e 12 Learned
72:e1:e0:41:93:73 TRK5 Learned
80:c1:6e:64:8d:3c TRK1 Learned
80:c1:6e:64:8d:3c TRK1 Learned
80:c1:6e:64:8d:3d TRK4 Learned
80:c1:6e:64:ab:2a TRK6 Learned
80:c1:6e:64:ab:2a TRK6 Learned
80:c1:6e:64:ab:2b TRK6 Learned
c2:31:d6:cc:cf:4b TRK5 Learned
d4:c9:ef:3a:fe:e0 TRK1 Learned
d4:c9:ef:3a:fe:f8 TRK1 Learned
d4:c9:ef:3c:d9:a0 CPU Management
e8:39:35:b7:c7:4c TRK5 Learned
e8:39:35:b7:c7:4c TRK5 Learned
e8:39:35:b7:c7:4d TRK5 Learned
also i see allot of double mac entries for same trunk groups.. actually i expected to only see one MAC for each trunk group? is that normal proxmox behavior or could it be because of switch loops? Would a normal setup with a proxmox node connected to two switche using LACP look this way or?
THANKS
Casper
EDIT:
Here is mac table of switch 2
Code:
MAC Address
Source Port
MAC Type
00:11:32:24:28:9b TRK1 Learned
00:11:32:24:28:9c TRK2 Learned
00:11:32:24:28:d7 TRK3 Learned
00:11:32:24:28:d8 TRK3 Learned
00:11:32:24:28:da 22 Learned
00:90:fb:40:9c:f6 12 Learned
00:e0:20:11:0a:2e TRK1 Learned
00:e0:20:11:0a:2e TRK1 Learned
72:e1:e0:41:93:73 TRK1 Learned
80:c1:6e:64:8d:3c TRK4 Learned
80:c1:6e:64:8d:3c TRK4 Learned
80:c1:6e:64:ab:2a TRK1 Learned
80:c1:6e:64:ab:2a TRK1 Learned
d4:c9:ef:3a:fe:e0 CPU Management
d4:c9:ef:3c:d9:b8 TRK1 Learned
e8:39:35:b7:c7:4c TRK5 Learned
e8:39:35:b7:c7:4c TRK1 Learned