The pfSense Store

Author Topic: Default route lost when primary is restored  (Read 133 times)

0 Members and 1 Guest are viewing this topic.

Offline Cortland

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Default route lost when primary is restored
« on: December 28, 2017, 09:49:21 am »
I have a CARP setup using local addresses on the WAN interfaces and a single public IP as the WAN CARP VIP.  All is well on startup and on failover, but when the primary gets restored after failover, it no longer has a default route.
  • On startup: primary and secondary have a default route to my specified default gateway; clients have internet and all is well.
  • On failover: secondary only has a default route to the gateway; clients have internet and is well.
  • On restore: secondary only has a default route to the gateway; clients have no internets.

After restore, I can manually ssh in to the primary and `route add default {my gateway}` and everything is fine.  I hear the devil whispering "cronjob" but that don't sit right with me.

I've seen similar posts with this issue.  The solution seems to be, make sure you have a NAT outbound rule from "this firewall" to the WAN CARP VIP.  I got that.  Issue persists.  Any ideas?  Thanks!


Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9589
  • Karma: +1089/-309
    • View Profile
Re: Default route lost when primary is restored
« Reply #1 on: December 28, 2017, 10:53:53 am »
Your single address and gateway is configured as static on the interface?



There is no longer a base OS requirement that a CARP VIP be in the same subnet as the interface.

That does not mean that your configuration is a supported one for HA.

If it is worth HA it is worth doing right.

Get more addresses from your ISP.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Cortland

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: Default route lost when primary is restored
« Reply #2 on: December 28, 2017, 03:49:08 pm »
Yes, the single public IP and gateway are configured as static.

My provider gave me a /30 network for 2Gbps fiber service.  I can certainly go back and ask for a /29, but if anybody has any ideas I'd like to understand why the single IP is breaking my gateway on CARP restoration.  Thanks again.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9589
  • Karma: +1089/-309
    • View Profile
Re: Default route lost when primary is restored
« Reply #3 on: December 28, 2017, 04:38:09 pm »
The public IP address has to be the CARP VIP (not the interface address) or HA has no chance of working.

What are the interface addresses configured on the WAN interfaces on both nodes? What are the gateways?

What is the WAN CARP VIP?

HA works really well when done correctly. When people try to "game" it, not so much.

You would have to:

Assign a private addressing scheme to the WAN interfaces.

Coerce the system to accept a default gateway outside that subnet (The ISP side of the /30). That generally involves checking the "Use non-local gateway" checkbox in the advanced settings of the gateway.

Set the CARP VIP on WAN to your side of the public /30)

Ensure that ALL traffic egressing WAN intended for the internet is outbound NAT to the CARP VIP address. (Note that this is where the process usually breaks down because this is impossible on the node that is currently CARP BACKUP which means it can't resolve DNS or anything without extreme creativity)

The HA code is not designed to work around this (because it is not possible). If it were me I would forgo HA in lieu of a cold spare and Auto-Config Backup or get a /29 and do it right.


Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Cortland

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: Default route lost when primary is restored
« Reply #4 on: December 28, 2017, 05:47:39 pm »
Yes, my public IP is my CARP VIP.  Everything *is* working with CARP enabled, and also when CARP fails over.  It's only after the primary node transitions back to master that it goes awry and only because of the default route issue.  If I put a shell script on the primary to check for a default route every few seconds and add it if one is missing, then everything is functionally perfect.  But, of course, that's pretty lame.

WAN1 IP: 10.251.0.10
WAN2 IP: 10.251.0.11
WAN CARP VIP: 216.12.33.X

My default gateway on both nodes is the same and set up for the non-local gateway option. The NAT Address on my NAT Outbound rules is set for the VIP and not the WAN address.

I understand your point regarding three public IPs, as well as the limitation of the nodes not being able to access the internet while in BACKUP status.  I'm not trying to half-ass this -- but everything I want to achieve with this HA setup appears to be working great, except for the issue of the primary node losing its default route when it goes back to master.

Online BeingMoody

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Default route lost when primary is restored
« Reply #5 on: December 28, 2017, 09:03:32 pm »
I've worked thru this the past several days. here's how I worked thru it. My business cable modem supports NAT as well as Public IP's. so I assigned a private IP to the WAN port on each node with a default GW. Then created an additional GW (not default) to the public IP address of the cable modem. No need to NAT the firewall to the VIP, I tried this at first and it never worked. Just change NAT to manual and change the destination to the VIP.

One word of info, my public range is /29 and I initially created the outbound VIP as .219 and wanted to change it to .221. Changing the Firewall -> VIP from .219 to .221 wasn't completely successful. Actually had to change the individual NAT's to the new VIP.

Hopefully this helps you out.