Netgate SG-1000 microFirewall

Author Topic: HA CARP - IPv6 Two masters  (Read 1000 times)

0 Members and 1 Guest are viewing this topic.

Offline aeburriel

  • Newbie
  • *
  • Posts: 1
  • Karma: +1/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #15 on: December 30, 2017, 03:48:37 pm »
I've been dealing with the same problem in my HA setup and it turned to be related to bug #6579
https://redmine.pfsense.org/issues/6579

The affected CARP IPv6 address was something like:
2001:aaaa:bbb:ccc:0d00:ffff:ffff:ffff
After removing leading zeros:
2001:aaaa:bbb:ccc:d00:ffff:ffff:ffff

CARP started to work reliably on that interface

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 10266
  • Karma: +1177/-313
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #16 on: December 30, 2017, 05:21:43 pm »
Nice catch.

Quote
LAN@213   fd57:187e:523f:0715::f    MASTER
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline IcePick

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #17 on: January 23, 2018, 07:06:13 pm »
I am having exactly the same as this since moving to 2.4 from 2.3.5
interesting only on 2 of the 4 IPv6 CARPs
they were the only 2 that could use :: in their address
I tried expanding to 0:0:0:
it did not help

I have confirmed by packet capture that packets to ff02::12 are seen on both systems



--------------
Ok I figured out how to get it to a normal state (all master on primary and all backup on secondary).
You need to reboot the backup firewall, and while its rebooting clear the firewall states on the primary.
Carp failover works perfectly when its like this but there is still an issue.

ANY configuration sync (manual/auto) from the primary to the backup causes the backup to become master on the two IPV6 carps.
------------
« Last Edit: January 23, 2018, 07:13:35 pm by IcePick »

Offline IcePick

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #18 on: January 23, 2018, 07:32:58 pm »
To reiterate I did not have this issue until upgrading from 2.3.5 to 2.4.2-RELEASE-p1, or at least it seemed to have gotten worse.

More testing:
changed from x::1 (ie X:0:0:0:1)  to x:1:1:1:1
on one of the CARP interfaces and the problem went away
Did not change the real interface IP

UPDATE: it worked for the first one, but broke both after I changed the second one.
Why are these 2 different then the other 2?
they connect to the same switch
I found a difference, one set of addresses used all lower case for the hex in the address, the none working ones had capitals.
I have changed all to lower and rebooted B unit and it came up all in backup, did not have to reset states on A firewall.
I'm not saying this is the issue - but giving people ideas of what I found
So in summary: using all lower case for hex and changed the addresses to ones that can not condense to ::




« Last Edit: January 24, 2018, 08:00:46 am by IcePick »

Offline anthonysomerset

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #19 on: March 08, 2018, 03:17:12 am »
Just want to add i appear to have hit this "bug" in one of our SG-4860 clusters

our IPv6 addresses are in their shortened form with no leading zeros, had to reboot secondary to clear this out, will keep an eye on things

Offline xciter327

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #20 on: March 08, 2018, 09:46:24 am »
I am also hitting something similar this in our office/test system.

Both devices are connected to a Cisco 3560G switch. IGMP snooping and ipv6 mld snooping are disabled. All ports are set to "portfast". There are no "loops" in the network. There are no topology changes.

You will notice that each one sees the others advertisements and their own.

Primary:
16:42:40.428976 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:42:42.597228 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
16:42:50.886692 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:42:52.607533 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
16:43:01.382988 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:43:02.612549 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

Backup:
16:42:09.212760 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:42:12.573960 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
16:42:19.608720 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:42:22.578900 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
16:42:30.015028 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
16:42:32.585911 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

This only happens for IPv6 CARP IPs.

Here are the interfaces, just to confirm the vhid:

Primary:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 0c:c4:7a:ac:82:1a
   hwaddr 0c:c4:7a:ac:82:1a
   inet6 fe80::ec4:7aff:feac:821a%igb0 prefixlen 64 scopeid 0x1
   inet6 xxxx:xxxx:1:2::3 prefixlen 124
   inet6 xxxx:xxxx:1:2::2 prefixlen 124 vhid 4
   inet yyy.yyy.233.108 netmask 0xfffffff0 broadcast yyy.yyy.233.111
   inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   carp: MASTER vhid 1 advbase 10 advskew 1
   carp: MASTER vhid 4 advbase 10 advskew 1

Backup:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 0c:c4:7a:ab:37:24
   hwaddr 0c:c4:7a:ab:37:24
   inet6 fe80::ec4:7aff:feab:3724%igb0 prefixlen 64 scopeid 0x1
   inet6 xxxx:xxxx:1:2::4 prefixlen 124
   inet yyy.yyy.233.109 netmask 0xfffffff0 broadcast yyy.yyy.233.111
   inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   carp: MASTER vhid 4 advbase 10 advskew 101
   carp: BACKUP vhid 1 advbase 10 advskew 101

So the CARP interface is correctly assigned to the primary node, but the backup one still claims its master in the dashboard and with "ifconfig igb0".
« Last Edit: March 08, 2018, 09:51:40 am by xciter327 »

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 10266
  • Karma: +1177/-313
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #21 on: March 08, 2018, 01:18:55 pm »
Why did you play with advbase/advskew?

Use 1/0 on the primary that will sync 1/10 to the secondary. Then just leave it alone.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline xciter327

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #22 on: March 12, 2018, 04:12:06 am »
Yes. I did try multiple base values between 0 - 20 for the base and 0 and 1 for skew. The settings are correctly(+100 for skew) transferred to the backup unit. Still backup thinks it's primary for IPv6.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 10266
  • Karma: +1177/-313
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #23 on: March 12, 2018, 04:55:40 am »
Are you 100% certain the case described in reply #15 ^ is not present?

Quote
Use 1/0 on the primary that will sync 1/10 to the secondary. Then just leave it alone.

Just do that. If changing it didn't correct it it is not the problem.

Packet capture on both nodes and see if you see the CARP going out the interface or in the interface. You can filter on CARP only in Diagnostics > Packet Capture.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline xciter327

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #24 on: March 12, 2018, 09:35:46 am »

1. Regarding post #15 solution. I tried both shorthand(no leading zeroes) and full notation with nothing omitted.
2. I included a tcpdump in my first post. It looks to me that they are both receiving each other's updates.

Offline IcePick

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #25 on: March 12, 2018, 09:45:45 am »
Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

Offline xciter327

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #26 on: March 12, 2018, 10:05:46 am »
Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

Yes I did. No difference.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 10266
  • Karma: +1177/-313
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #27 on: March 12, 2018, 10:32:48 am »
Did you put base/skew back to the default or not?
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline xciter327

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #28 on: March 13, 2018, 03:11:12 am »
Did you put base/skew back to the default or not?
Yes, I did.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 10266
  • Karma: +1177/-313
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #29 on: March 13, 2018, 04:09:19 am »
Well, cut loose with more. Screen shots, pcaps, whatever. IPv6 CARP works.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM