Quantcast
Channel: Proxmox Support Forum
Viewing all articles
Browse latest Browse all 171679

IPv6 (NDP?) issues on secondary bridge

$
0
0
Hi there,

I've been doing IPv6 routing for OpenVZ CTs with venet for a while and it works like a charm. Notably this allows me *not* to bridge them on the main interface. This is a requirement for us because of the policy of the hosting environment.

Now, I'd like to treat KVMs, with the same requirements, so I set up a secondary bridge (vmbr1), with no physical interface on it, whereas the usual vmbr0 has the main physical interface. The end result is something that works for a few minutes unless I specify all routes manually between the guest and the host using link-local addresses. After a while or a reboot, ping6 does not pass between them. On the other hand, the very same bridge provides IPv4 for NATing without a hitch.

(details on PVE version at the end)

Code:

auto vmbr1
iface vmbr1 inet static
        address  192.168.57.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

iface vmbr0 inet6 static
      address 2001:xxx:200::1/64 # this is also the range used on CTs' venets

iface vmbr1 inet6 static
      address 2001:xxx:201::1/64  # this is also the range for KVM guests

From inside the VM, if I find the link-local address of vmbr1 and use it as default route, and give the host a route to the VM's global address via the VM's link-local address, it works. Even this impractical way of doing is not enough, because the bridge's link-local address changes at each reboot. I had great hopes that fixing the bridge's MAC address would help, but it did not.

Another way to make it work is to give the host an explicit route to the guest, using this time knowledge of the guests' link-local address.

The bridge has only VMs and a CT :

Code:

galilee-~ $ sudo brctl show vmbr1
bridge name    bridge id        STP enabled    interfaces
vmbr1        8000.0001002944ed    no        tap106i0
                                                          tap107i0
                                                          veth100.0

The CT has been added because it starts first in the host boot sequence, and that helps with stability of the MAC address, but the problems were identical before I tried that.

I played a bit with neighbor proxy to no convincing result.
I tried enabling STP, with no result.

I played a bit with tcpdump to investigate. It seems that NDP packets from the host don't reach the guest and vice-versa. I had the feeling for a while that launching tcpdump actually fixed (again, temporarily) the ping, but if there's a side effect of that kind, it's not systematic. For instance, after a reboot, everything is stuck until I set up a static route using link-local addresses

I'm not very knowledgeable in layer2 matters, but it looks as an issue at this level of the stack. Maybe there's something very simple that I'm overlooking, but I'm running out of ideas.

Does all of this ring a bell to anyone ?

Configuration details:

On the host:
Code:

# sysctl net.ipv6.conf.vmbr1
net.ipv6.conf.vmbr1.forwarding = 1
net.ipv6.conf.vmbr1.hop_limit = 64
net.ipv6.conf.vmbr1.mtu = 1500
net.ipv6.conf.vmbr1.accept_ra = 1
net.ipv6.conf.vmbr1.accept_redirects = 1
net.ipv6.conf.vmbr1.autoconf = 0
net.ipv6.conf.vmbr1.dad_transmits = 1
net.ipv6.conf.vmbr1.router_solicitations = 3
net.ipv6.conf.vmbr1.router_solicitation_interval = 4
net.ipv6.conf.vmbr1.router_solicitation_delay = 1
net.ipv6.conf.vmbr1.force_mld_version = 0
net.ipv6.conf.vmbr1.use_tempaddr = 0
net.ipv6.conf.vmbr1.temp_valid_lft = 604800
net.ipv6.conf.vmbr1.temp_prefered_lft = 86400
net.ipv6.conf.vmbr1.regen_max_retry = 5
net.ipv6.conf.vmbr1.max_desync_factor = 600
net.ipv6.conf.vmbr1.max_addresses = 16
net.ipv6.conf.vmbr1.accept_ra_defrtr = 1
net.ipv6.conf.vmbr1.accept_ra_pinfo = 1
net.ipv6.conf.vmbr1.accept_ra_rtr_pref = 1
net.ipv6.conf.vmbr1.router_probe_interval = 60
net.ipv6.conf.vmbr1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.vmbr1.proxy_ndp = 0
net.ipv6.conf.vmbr1.accept_source_route = 0
net.ipv6.conf.vmbr1.optimistic_dad = 0
net.ipv6.conf.vmbr1.mc_forwarding = 0
net.ipv6.conf.vmbr1.disable_ipv6 = 0
net.ipv6.conf.vmbr1.accept_dad = 1

On the guest:
Code:

$ sudo sysctl net.ipv6.conf.ens18
net.ipv6.conf.ens18.accept_dad = 1
net.ipv6.conf.ens18.accept_ra = 1
net.ipv6.conf.ens18.accept_ra_defrtr = 0
net.ipv6.conf.ens18.accept_ra_pinfo = 0
net.ipv6.conf.ens18.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.ens18.accept_ra_rtr_pref = 0
net.ipv6.conf.ens18.accept_redirects = 1
net.ipv6.conf.ens18.accept_source_route = 0
net.ipv6.conf.ens18.autoconf = 1
net.ipv6.conf.ens18.dad_transmits = 1
net.ipv6.conf.ens18.disable_ipv6 = 0
net.ipv6.conf.ens18.force_mld_version = 0
net.ipv6.conf.ens18.force_tllao = 0
net.ipv6.conf.ens18.forwarding = 0
net.ipv6.conf.ens18.hop_limit = 64
net.ipv6.conf.ens18.max_addresses = 16
net.ipv6.conf.ens18.max_desync_factor = 600
net.ipv6.conf.ens18.mc_forwarding = 0
net.ipv6.conf.ens18.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ens18.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ens18.mtu = 1500
net.ipv6.conf.ens18.ndisc_notify = 0
net.ipv6.conf.ens18.optimistic_dad = 0
net.ipv6.conf.ens18.proxy_ndp = 0
net.ipv6.conf.ens18.regen_max_retry = 3
net.ipv6.conf.ens18.router_probe_interval = 60
net.ipv6.conf.ens18.router_solicitation_delay = 1
net.ipv6.conf.ens18.router_solicitation_interval = 4
net.ipv6.conf.ens18.router_solicitations = 3
net.ipv6.conf.ens18.temp_prefered_lft = 86400
net.ipv6.conf.ens18.temp_valid_lft = 604800
net.ipv6.conf.ens18.use_tempaddr = 0

Pinging a VM from the host
Code:

$ ping6 2001:xxx:201::106
PING 2001:xxx:201::106(2001:xxx:201::106) 56 data bytes
From 2001:xxx:201::1 icmp_seq=2 Destination unreachable: Address unreachable

And from inside a VM (CentOS 7 if that matters) :

Code:

$ ip -6 addr show ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2001:xxx:201::106/64 scope global
      valid_lft forever preferred_lft forever
    inet6 fe80::6041:c0ff:fee4:68bc/64 scope link
      valid_lft forever preferred_lft forever

$ ip -6 addr show ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2001:xxx:201::106/64 scope global
      valid_lft forever preferred_lft forever
    inet6 fe80::6041:c0ff:fee4:68bc/64 scope link
      valid_lft forever preferred_lft forever

$ ping6 -w 1 2001:xxx:201::1
PING 2001:xxx:201::1(2001:xxx:201::1) 56 data bytes

--- 2001:xxx:201::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

$ ip -6 neigh # (the stable MAC address worked, but the link-local of the router is still random)
2001:xxx:201::1 dev ens18 lladdr 00:01:00:29:44:ed router REACHABLE
fe80::eca0:8aff:feed:272a dev ens18 lladdr 00:01:00:29:44:ed router REACHABLE

On the host:

Code:

$ ip -6 route | grep vmbr
2001:xxx:200::/64 dev vmbr0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::/64 dev vmbr1  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr1  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
default via fe80::225:45ff:fef6:947f dev vmbr0  metric 1024  mtu 1500 advmss 1440 hoplimit 0

Now a setup that works (using link-local addresses everywhere).

Host:
Code:

$ ip -6 route | grep vmbr
2001:xxx:200::1 dev vmbr0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::106 via fe80::6041:c0ff:fee4:68bc dev vmbr1  metric 1024  mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::/64 dev vmbr1  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr1  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 0
default via fe80::225:45ff:fef6:947f dev vmbr0  metric 1024  mtu 1500 advmss 1440 hoplimit 0

Guest:
Code:

$ ip -6 route | grep -v unreachable
fe80::eca0:8aff:feed:272a dev ens18  proto static  metric 100
fe80::/64 dev ens18  proto kernel  metric 256
default via fe80::eca0:8aff:feed:272a dev ens18  proto static  metric 100

With this ping6 between host and guest work, and on top of that, I can reach the VM from the internet.
After rebooting the host, the LL address of vmbr1 has changed:

Code:

$ /sbin/ifconfig vmbr1 | grep -v 2001
vmbr1    Link encap:Ethernet  HWaddr 00:01:00:29:44:ed 
          inet adr:192.168.57.1  Bcast:192.168.57.255  Masque:255.255.255.0
          adr inet6: 2001:xxx:201::1/64 Scope:Global
          adr inet6: fe80::9cfa:64ff:fe7c:ab41/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:0
          RX bytes:184 (184.0 B)  TX bytes:686 (686.0 B)

and now for something really puzzling, right after the reboot I did to write this message, the ping worked although the guest's default route is on a LL address that does not exist any more, but this lasted less than 10 mins.

Proxmox software versions:

Code:

$ dpkg -l \*pve\*         
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name          Version      Architecture Description
+++-==============-============-============-=================================
ii  corosync-pve  1.4.7-1      amd64        Standards-based cluster framework
ii  fence-agents-p 4.0.10-2    amd64        fence agents for redhat cluster s
ii  libcorosync4-p 1.4.7-1      amd64        Standards-based cluster framework
ii  libopenais3-pv 1.1.4-3      amd64        Standards-based cluster framework
ii  libpve-access- 3.0-16      amd64        Proxmox VE access control library
ii  libpve-common- 3.0-24      all          Proxmox VE base library
ii  libpve-storage 3.0-31      all          Proxmox VE storage management lib
ii  novnc-pve      0.4-7        amd64        HTML5 VNC client
ii  openais-pve    1.1.4-3      amd64        Standards-based cluster framework
ii  pve-cluster    3.0-16      amd64        Cluster Infrastructure for Proxmo
ii  pve-firewall  1.0-18      amd64        Proxmox VE Firewall
ii  pve-firmware  1.1-3        all          Binary firmware code for the pve-
un  pve-kernel    <none>                    (no description available)
ii  pve-kernel-2.6 2.6.32-140  amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6 2.6.32-147  amd64        The Proxmox PVE Kernel Image
un  pve-kvm        <none>                    (no description available)
ii  pve-libspice-s 0.12.4-3    amd64        SPICE remote display system serve
ii  pve-manager    3.4-1        amd64        The Proxmox Virtual Environment
ii  pve-qemu-kvm  2.1-12      amd64        Full virtualization on x86 hardwa
un  pve-qemu-kvm-2 <none>                    (no description available)
ii  redhat-cluster 3.2.0-2      amd64        Red Hat cluster suite
ii  resource-agent 3.9.2-4      amd64        resource agents for redhat cluste


Viewing all articles
Browse latest Browse all 171679

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>