pfSense Gold Subscription

Author Topic: HA CARP - IPv6 Two masters  (Read 260 times)

0 Members and 1 Guest are viewing this topic.

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
HA CARP - IPv6 Two masters
« on: December 04, 2017, 08:58:15 am »
Hello,

I just setup two devices running pfsense 2.4.2 running in ha mode.

I have several carp interfaces however the ipv6 carp interfaces show master on each device and the ipv4 carp interfaces are working properly.

I have checked the broadcast domain for other vrrp devices and the vhid that the carp interfaces are using are not in use anywhere else.

Im really not sure why its not working. Again this is only affecting ipv6

Any help would be greatly appreciated.

--primary device--
CARP Interface   IP Address   Status
WAN@210   66.X.X.30    MASTER
WAN@211   2001:X:X:X::F    MASTER
LAN@212   172.26.8.65    MASTER
LAN@213   fd57:187e:523f:0715::f    MASTER
RFC_BACKEND@214   172.26.8.30    MASTER


--backup device--
CARP Interface   IP Address   Status
WAN@210   66.X.X.30    BACKUP
WAN@211   2001:X:X:X::F    MASTER
LAN@212   172.26.8.65    BACKUP
LAN@213   fd57:187e:523f:0715::f    MASTER
RFC_BACKEND@214   172.26.8.30    BACKUP

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #1 on: December 04, 2017, 01:42:21 pm »
Im seeing the following in the logs on the backup firewall

Dec 4 17:40:20   php-fpm   58958   /xmlrpc.php: The command '/sbin/ifconfig 'bce0.715' inet6 'fd57:187e:523f:0715::f' delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 4 17:40:20   php-fpm   58958   /xmlrpc.php: The command '/sbin/ifconfig 'bce0.210' inet6 '2001:X:X:X::F' delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 4 17:40:20   kernel      ifa_maintain_loopback_route: insertion failed for interface bce0.715: 17
Dec 4 17:40:20   php-fpm   58958   /xmlrpc.php: The command '/sbin/ifconfig bce0.715 inet6 'fd57:187e:523f:0715::f' prefixlen '64' alias vhid '213'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): File exists'
Dec 4 17:40:20   kernel      ifa_maintain_loopback_route: insertion failed for interface bce0.210: 17
Dec 4 17:40:20   php-fpm   58958   /xmlrpc.php: The command '/sbin/ifconfig bce0.210 inet6 '2001:X:X:X::F' prefixlen '64' alias vhid '211'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): File exists'

Offline awebster

  • Sr. Member
  • ****
  • Posts: 356
  • Karma: +54/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #2 on: December 04, 2017, 08:34:55 pm »
What are the actual real IPv6 IPs?
Are you assigning fd57 from the same subnet as the actual intefaces?
Why would you want to use unique local addresses on IPv6?  That's not the design philosophy of IPv6.
--A.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #3 on: December 04, 2017, 08:52:32 pm »
Your switching is probably not properly passing traffic to multicast destination ff02::12.

Diagnostics > Packet Capture on the primary:

Interface: One with an IPv6 CARP VIP
Address Family: IPv6-Only
Protocol: any (Capturing CARP here doesn't seem to work.. Problem for another day.)
Host Address: ff02::12
Count: 5

You should get something like this. Your source address will be different but should also start with fe80:

02:45:01.595176 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:45:02.601844 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:45:03.645118 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:45:04.652798 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:45:05.668150 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36

Do the same capture on the Secondary. You should see the same thing:

02:46:12.490962 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:46:13.550945 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:46:14.611020 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:46:15.670940 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36
02:46:16.728002 IP6 fe80::f092:faff:fe6a:3279 > ff02::12: ip-proto-112 36

You will probably not see that. You will probably see the secondary transmitting from its own link-local address because it is not receiving the multicasts from the primary and is, properly, treating that CARP VIP as down. If that is the case you need to fix your layer 2.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #4 on: December 05, 2017, 09:36:33 am »
I didn't include the actual IP addresses because I didn't want to expose the firewall but it's locked down so the point is moot.

This pair of firewalls will be the gateway for vpn users. I have another vpn appliance to handle that.
The VPN will give users RFC 1918 / 4193 (ULA) addresses and the firewall pair which is the gateway for those usesers will perform NAT / NPT to Globally routed addresses. I don't know if this is best practice but this is the solution I am trying to implement.

-- carp --
WAN@210   66.133.130.30    MASTER
WAN@211   2001:1960:20:D2::F    MASTER
LAN@212   172.26.8.65    MASTER
LAN@213   fd57:187e:523f:0715::f    MASTER
RFC_BACKEND@214   172.26.8.30    MASTER

-- primary --
bce0.210: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:89:11:f6
        inet6 fe80::224:81ff:fe89:11f6%bce0.210 prefixlen 64 scopeid 0xd
        inet6 2001:1960:20:d2::a prefixlen 64
        inet6 2001:1960:20:d2::f prefixlen 64 vhid 211
        inet 66.133.130.28 netmask 0xfffffff8 broadcast 66.133.130.31
        inet 66.133.130.30 netmask 0xfffffff8 broadcast 66.133.130.31 vhid 210
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 210 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 210 advbase 1 advskew 0
        carp: MASTER vhid 211 advbase 1 advskew 0
        groups: vlan
bce0.710: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:89:11:f6
        inet6 fe80::224:81ff:fe89:11f6%bce0.710 prefixlen 64 scopeid 0xe
        inet 172.26.8.28 netmask 0xfffffff8 broadcast 172.26.8.31
        inet 172.26.8.30 netmask 0xfffffff8 broadcast 172.26.8.31 vhid 214
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 710 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 214 advbase 1 advskew 0
        groups: vlan
bce0.715: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:89:11:f6
        inet6 fe80::224:81ff:fe89:11f6%bce0.715 prefixlen 64 scopeid 0xf
        inet6 fd57:187e:523f:715::a prefixlen 64
        inet6 fd57:187e:523f:715::f prefixlen 64 vhid 213
        inet 172.26.8.66 netmask 0xffffffc0 broadcast 172.26.8.127
        inet 172.26.8.65 netmask 0xffffffc0 broadcast 172.26.8.127 vhid 212
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 715 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 212 advbase 1 advskew 0
        carp: MASTER vhid 213 advbase 1 advskew 0
        groups: vlan


-- backup --

bce0.210: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:88:f1:06
        inet6 fe80::224:81ff:fe88:f106%bce0.210 prefixlen 64 scopeid 0xd
        inet6 2001:1960:20:d2::b prefixlen 64
        inet 66.133.130.29 netmask 0xfffffff8 broadcast 66.133.130.31
        inet 66.133.130.30 netmask 0xfffffff8 broadcast 66.133.130.31 vhid 210
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 210 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 211 advbase 1 advskew 100
        carp: BACKUP vhid 210 advbase 1 advskew 100
        groups: vlan
bce0.710: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:88:f1:06
        inet6 fe80::224:81ff:fe88:f106%bce0.710 prefixlen 64 scopeid 0xe
        inet 172.26.8.29 netmask 0xfffffff8 broadcast 172.26.8.31
        inet 172.26.8.30 netmask 0xfffffff8 broadcast 172.26.8.31 vhid 214
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 710 vlanpcp: 0 parent interface: bce0
        carp: BACKUP vhid 214 advbase 1 advskew 100
        groups: vlan
bce0.715: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:88:f1:06
        inet6 fe80::224:81ff:fe88:f106%bce0.715 prefixlen 64 scopeid 0xf
        inet6 fd57:187e:523f:715::b prefixlen 64
        inet 172.26.8.67 netmask 0xffffffc0 broadcast 172.26.8.127
        inet 172.26.8.65 netmask 0xffffffc0 broadcast 172.26.8.127 vhid 212
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 715 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 213 advbase 1 advskew 100
        carp: BACKUP vhid 212 advbase 1 advskew 100
        groups: vlan



-- primary --
15:29:54.265266 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36
15:29:55.088217 IP6 fe80::224:81ff:fe88:f106 > ff02::12: ip-proto-112 36
15:29:55.325010 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36
15:29:56.374974 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36
15:29:56.485201 IP6 fe80::224:81ff:fe88:f106 > ff02::12: ip-proto-112 36


-- backup --
5:34:50.696588 IP6 fe80::224:81ff:fe88:f106 > ff02::12: ip-proto-112 36
15:34:50.939315 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36
15:34:51.943702 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36
15:34:52.128312 IP6 fe80::224:81ff:fe88:f106 > ff02::12: ip-proto-112 36
15:34:52.953321 IP6 fe80::224:81ff:fe89:11f6 > ff02::12: ip-proto-112 36

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #5 on: December 05, 2017, 03:15:16 pm »
My secondary:

xn2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 4e:f6:47:4c:0e:df
   hwaddr 4e:f6:47:4c:0e:df
   inet6 fe80::4cf6:47ff:fe4c:edf%xn2 prefixlen 64 scopeid 0x7
   inet6 2001:beef:cafe:7e02::3 prefixlen 64
   inet6 2001:beef:cafe:7e02::1 prefixlen 64 vhid 240
   inet 172.25.237.3 netmask 0xffffff00 broadcast 172.25.237.255
   inet 172.25.237.1 netmask 0xffffff00 broadcast 172.25.237.255 vhid 237
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet manual
   status: active
   carp: BACKUP vhid 237 advbase 1 advskew 100
   carp: BACKUP vhid 240 advbase 1 advskew 100

Your secondary:

bce0.715: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 00:24:81:88:f1:06
        inet6 fe80::224:81ff:fe88:f106%bce0.715 prefixlen 64 scopeid 0xf
        inet6 fd57:187e:523f:715::b prefixlen 64
        inet 172.26.8.67 netmask 0xffffffc0 broadcast 172.26.8.127
        inet 172.26.8.65 netmask 0xffffffc0 broadcast 172.26.8.127 vhid 212
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 715 vlanpcp: 0 parent interface: bce0
        carp: MASTER vhid 213 advbase 1 advskew 100
        carp: BACKUP vhid 212 advbase 1 advskew 100
        groups: vlan

Note the absence of the CARP VIP on the interface itself.

It looks like the interface is confused. Not sure. Have you rebooted the secondary?

I was able to get mine into a strange state but only by manually issuing ifconfig commands in the shell. You probably want to make sure everything looks good in the VIP settings and reboot the secondary.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #6 on: December 05, 2017, 04:44:21 pm »
Yeah i noticed that as well when i was replying earlier.  I have rebooted the device and that doesnt clear the issue.
I'm pretty sure its not a L2 issue because the v4 carp vip works and is on the same vlan 210/715.

I am seeing this in the logs in the backup device.

Dec 5 20:18:45   php-fpm   71243   /xmlrpc.php: The command '/sbin/ifconfig 'bce0.715' inet6 'fd57:187e:523f:0715::f' delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 5 20:18:45   php-fpm   71243   /xmlrpc.php: The command '/sbin/ifconfig 'bce0.210' inet6 '2001:1960:20:D2::F' delete' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 5 20:18:45   kernel      ifa_maintain_loopback_route: insertion failed for interface bce0.715: 17
Dec 5 20:18:45   php-fpm   71243   /xmlrpc.php: The command '/sbin/ifconfig bce0.715 inet6 'fd57:187e:523f:0715::f' prefixlen '64' alias vhid '213'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): File exists'
Dec 5 20:18:45   kernel      ifa_maintain_loopback_route: insertion failed for interface bce0.210: 17
Dec 5 20:18:45   php-fpm   71243   /xmlrpc.php: The command '/sbin/ifconfig bce0.210 inet6 '2001:1960:20:D2::F' prefixlen '64' alias vhid '211'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): File exists'

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #7 on: December 05, 2017, 05:33:20 pm »
Yeah. I get that is what you are seeing.

There is nothing systemic regarding IPv6 and CARP. It has to be something in your config.

Make sure all the interfaces match exactly in order and in name.

Make sure all the IP aliases and other VIPs match exactly, except for the advskew.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #8 on: December 07, 2017, 02:14:24 pm »
Ok I figured out how to get it to a normal state (all master on primary and all backup on secondary).
You need to reboot the backup firewall, and while its rebooting clear the firewall states on the primary.
Carp failover works perfectly when its like this but there is still an issue.

ANY configuration sync (manual/auto) from the primary to the backup causes the backup to become master on the two IPV6 carps.


Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #9 on: December 07, 2017, 05:07:14 pm »
None of that is necessary in a "normal" environment. I reboot the VMs all the time. Just works.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #10 on: December 08, 2017, 08:26:41 am »
None of that is necessary in a "normal" environment. I reboot the VMs all the time. Just works.

I'm not really following you on this one. I'm not running these on VM's but that doesn't matter.
I don't want to have to reboot a physical machine or a VM every time I make a configuration change.

This is not a L2 issue. This smells like a bug.




Offline awebster

  • Sr. Member
  • ****
  • Posts: 356
  • Karma: +54/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #11 on: December 08, 2017, 10:19:15 am »
Works great for me... (IPs masked to protect the innocent), but I did find that increasing ADVBASE to 10 on the backup as opposed to default 1 helped alot (maybe its because these are running on ESXi), anyway that's my recipe and I'm sticking to it.
Consequently on the backup uncheck "virtual IPs" in the System / High Availability Sync page.


MASTER

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:43:51:32
        hwaddr 00:0c:29:43:51:32
        inet6 fe80::20c:29ff:fe43:5132%em0 prefixlen 64 scopeid 0x1
        inet AA.BB.CC.226 netmask 0xfffffff8 broadcast AA.BB.CC.231
        inet6 xxxx:xxxx::1c prefixlen 125
        inet6 xxxx:xxxx::1e prefixlen 125 vhid 244
        inet AA.BB.CC.225 netmask 0xfffffff8 broadcast AA.BB.CC.231 vhid 242
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 244 advbase 1 advskew 0
        carp: MASTER vhid 242 advbase 1 advskew 0
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:43:51:3c
        hwaddr 00:0c:29:43:51:3c
        inet6 fe80::20c:29ff:fe43:513c%em1 prefixlen 64 scopeid 0x2
        inet XX.YY.ZZ.251 netmask 0xffffff00 broadcast XX.YY.ZZ.255
        inet6 xxxx:xxxx:10:2800::2 prefixlen 64
        inet XX.YY.ZZ.254 netmask 0xffffff00 broadcast XX.YY.ZZ.255 vhid 240
        inet6 xxxx:xxxx:10:2800::1 prefixlen 64 vhid 241
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 240 advbase 1 advskew 0
        carp: MASTER vhid 241 advbase 1 advskew 0


BACKUP

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:4c:da:30
        hwaddr 00:0c:29:4c:da:30
        inet6 fe80::20c:29ff:fe4c:da30%em0 prefixlen 64 scopeid 0x1
        inet AA.BB.CC.227 netmask 0xfffffff8 broadcast AA.BB.CC.231
        inet6 xxxx:xxxx::1d prefixlen 125
        inet6 xxxx:xxxx::1e prefixlen 125 vhid 244
        inet AA.BB.CC.225 netmask 0xfffffff8 broadcast AA.BB.CC.231 vhid 242
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: BACKUP vhid 244 advbase 10 advskew 100
        carp: BACKUP vhid 242 advbase 10 advskew 100
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:4c:da:3a
        hwaddr 00:0c:29:4c:da:3a
        inet6 fe80::20c:29ff:fe4c:da3a%em1 prefixlen 64 scopeid 0x2
        inet XX.YY.ZZ.252 netmask 0xffffff00 broadcast XX.YY.ZZ.255
        inet6 xxxx:xxxx:10:2800::3 prefixlen 64
        inet XX.YY.ZZ.254 netmask 0xffffff00 broadcast XX.YY.ZZ.255 vhid 240
        inet6 xxxx:xxxx:10:2800::1 prefixlen 64 vhid 241
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: BACKUP vhid 240 advbase 10 advskew 100
        carp: BACKUP vhid 241 advbase 10 advskew 100
--A.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #12 on: December 08, 2017, 11:34:32 am »
Doesn't sound like a bug because far too many people are NOT seeing the issue. It is something specific to the way you have it configured or something in your environment.
« Last Edit: December 09, 2017, 01:13:02 pm by Derelict »
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline rhwendt

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #13 on: December 08, 2017, 12:27:33 pm »
Thank you all for helping.

I just factory rest both devices today and set them up from scratch again.
All the carp interfaces were working as expected except the IPV6 ULA CARP for the LAN (fd57:187e:523f:715::f/64)
It was exhibiting the same issues i was seeing prior to the factory reset, both primary and secondary both showing master.
The IPV6 GUA on the wan was working as expected
If I rebooted the secondary firewall all the carp interfaces would be in backup status. Anytime I synced the config from the primary it would cause the double master status.


I was able to find a solution based off what awebster said about unchecking the virtual ip in the HA sync.
I unchecked this option and rebooted the secondary firewall and now all the carp interfaces are showing the correct status and config syncing doesnt affect them.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: HA CARP - IPv6 Two masters
« Reply #14 on: December 09, 2017, 01:09:14 pm »
Just to be sure there wasn't something somewhere that misbehaved with ULA and CARP:

Primary:

xn5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether ee:c2:d9:d8:55:46
   hwaddr ee:c2:d9:d8:55:46
   inet6 fe80::ecc2:d9ff:fed8:5546%xn5 prefixlen 64 scopeid 0xd
   inet6 fda9:cfd8:f9f:1000::2 prefixlen 64
   inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
   inet 192.168.123.2 netmask 0xffffff00 broadcast 192.168.123.255
   inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet manual
   status: active
   carp: MASTER vhid 242 advbase 1 advskew 0
   carp: MASTER vhid 243 advbase 1 advskew 0

Secondary:

xn5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 6e:24:e4:84:f5:f9
   hwaddr 6e:24:e4:84:f5:f9
   inet6 fe80::6c24:e4ff:fe84:f5f9%xn5 prefixlen 64 scopeid 0xa
   inet6 fda9:cfd8:f9f:1000::3 prefixlen 64
   inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
   inet 192.168.123.3 netmask 0xffffff00 broadcast 192.168.123.255
   inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet manual
   status: active
   carp: BACKUP vhid 242 advbase 1 advskew 100
   carp: BACKUP vhid 243 advbase 1 advskew 100

Enter CARP Maintenance mode on Primary:

Primary:

xn5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=3<RXCSUM,TXCSUM>
   ether ee:c2:d9:d8:55:46
   hwaddr ee:c2:d9:d8:55:46
   inet6 fe80::ecc2:d9ff:fed8:5546%xn5 prefixlen 64 scopeid 0xd
   inet6 fda9:cfd8:f9f:1000::2 prefixlen 64
   inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
   inet 192.168.123.2 netmask 0xffffff00 broadcast 192.168.123.255
   inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet manual
   status: active
   carp: BACKUP vhid 242 advbase 1 advskew 254
   carp: BACKUP vhid 243 advbase 1 advskew 254

Secondary:

xn5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 6e:24:e4:84:f5:f9
   hwaddr 6e:24:e4:84:f5:f9
   inet6 fe80::6c24:e4ff:fe84:f5f9%xn5 prefixlen 64 scopeid 0xa
   inet6 fda9:cfd8:f9f:1000::3 prefixlen 64
   inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
   inet 192.168.123.3 netmask 0xffffff00 broadcast 192.168.123.255
   inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet manual
   status: active
   carp: MASTER vhid 242 advbase 1 advskew 100
   carp: MASTER vhid 243 advbase 1 advskew 100

Fails back fine, too.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM