The pfSense Store

Author Topic: CARP with multi-wan [SOLVED]  (Read 139 times)

0 Members and 1 Guest are viewing this topic.

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1786
  • Karma: +87/-11
    • View Profile
CARP with multi-wan [SOLVED]
« on: November 08, 2017, 02:26:20 pm »
I'm trying to configure CARP with multi-wan and can't figure out what I'm making wrong but, basically, it doesn't work as expected.



Modem                             Modem 1                              Modem 2           
mode                               PPPoE                                   PPPoE           
public IP                       xxx.xxx.xxx.xxx                   yyy.yyy.yyy.yyy           
IP (internal side)         192.168.253.14/29           192.168.253.6/29           
forward to                   192.168.253.9/29             192.168.253.1/29


pfSense                             igb0                                   igb1                                   igb2                           igb3                             igb5
interface                            WAN1                                 WAN2                               LAN                            DMZ                             SYNC
WAN master                  192.168.253.10/29           192.168.253.2/29        192.168.0.252/24        192.168.200.252         192.168.253.17/29
WAN slave                     192.168.253.11/29           192.168.253.3/29        192.168.0.253/24        192.168.200.253         192.168.253.18/29
VIP                                 192.168.253.9/29             192.168.253.1/29       192.168.0.254/24         192.168.200.254/24            none
WHID                                   2                                           1                                      3                                 4                              none
dft GW                           192.168.253.14                192.168.253.6                       none                          none                             none

WAN1 & WAN2 are both member of "WAN" group, both tier-1, none is designed as "default gateway"
WAN is configured as gateway in all rules at LAN interface level when flow is reaching internet. This works fine.

CARP itself is almost OK (some problems with DHCP because some clients receive DHCP offer from each DHCP server with different IP) but the real problem I'm facing is with in-coming flow.

If I try to reach internal mail server from internet using modem 1 public IP, this works.
When I try to reach same server using modem 2 public IP, I can see packets reaching WAN2, even going out from LAN to mail server but it looks like packets are back using route through VIP with WHID 2 instead of WHID 1.

I did read this https://doc.pfsense.org/index.php/Bypassing_Policy_Routing and applied it without any success. (and I don't understand how this applies to non-local IPs)
I also have multiple IPsec tunnels, for the time being hardcoded to connect to either WAN1 or WAN2, depending on remote network and this works fine (yes I'm facing the issue with IPSec status preventing to display is correctly).

I, of course, changed outbound NAT rules, in manual mode, to target WAN VIPs

Well, I'm quite confused.
I feel this is related to routing, perhaps bypass policy but I've to admit that, for the time being, I'm lost with poor understanding (and not real capability to closely investigate because, to make it worst, I'm configuring everything remotely through OpenVPN access  ;D ;D ;D :o

Any idea?
« Last Edit: November 09, 2017, 03:28:26 pm by chris4916 »

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: CARP with multi-wan
« Reply #1 on: November 09, 2017, 01:44:25 pm »
DHCP is probably giving you trouble because it sounds like you didn't configure the failover addresses for each DHCP interface. DHCP should be coordinating leases between the two units provided that failover is setup correctly in the DHCP settings (also, make sure to set a CARP VIP as the gateway and DNS server in the DHCP settings)

As for the reply traffic, it sounds like maybe your routing or gateways are not correct. Check that:

1. On WAN and WAN2 you have the gateway for the interface selected on Interfaces > WAN and Interfaces > WAN2
2. On your firewall rules, make sure you DO NOT use interface groups/rules for inbound WAN traffic. Traffic must use rules on the WAN interface tabs or else it cannot be properly tagged with reply-to so the firewall can remember the proper return path for traffic
3. Make sure that on your WAN interface rules, you DO NOT set a gateway on any rules on WAN tabs, this will definitely break things.

Somewhat related but not to that, make sure you do have at least one gateway marked default under System > Routing, and you do not need to manually set a gateway on every LAN rule if it's taking the default path. Only set a gateway group or alternate gateway if you need it to take a non-default path.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1786
  • Karma: +87/-11
    • View Profile
Re: CARP with multi-wan
« Reply #2 on: November 09, 2017, 03:23:00 pm »
Thank you Jimp for the anwser  ;)

1 - Regarding DHCP, I configured failover peer IP address, setting here physical IP address of slave pfSense. BTW, on slave pfSense, DHCP has been automatically configured with master physical IP as failover peer IP. I also defined, in DHCP server, CARP VIP, on the LAN side, as defaut gateway and DNS. This works. I mean that CARP work well. If I stop master pfSense, everything switches smoothly to slave server

And DHCP works, but not always. Here is what I collect in my rsyslog server:


Nov  9 06:51:54 192.168.0.253 dhcpd: DHCPREQUEST for 192.168.0.3 from 7e:d8:99:8f:32:a4 via igb2
Nov  9 06:51:54 192.168.0.253 dhcpd: DHCPACK on 192.168.0.3 to 7e:d8:99:8f:32:a4 via igb2
Nov  9 06:51:54 192.168.0.253 dhcpd: Unable to add forward map from webserver.rmo.ci to 192.168.0.3: REFUSED
Nov  9 06:52:30 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov  9 06:52:28 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:29 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov  9 06:52:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:35 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:35 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:39 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:43 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:43 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:02 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:59 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:02 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:52:59 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov  9 06:53:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:38 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:36 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:36 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:44 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:44 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:53:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:54:00 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:54:03 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:54:00 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov  9 06:54:03 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2


This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work  ;)
Notice that some devices are receiving same IP from each DHCP server.

2 - About FW rules:
I don't have any single gateway defined for WAN, WAN1, WAN2 or floating rules. everything uses "default".
Also, I just checked twice: both WAN1 & WAN2 have their own interface defined (modem to which interface is attached)
Well... everything looks clean thus I read your answer again and again... tada  ;D ;D

Problem was with FW rules for incoming flow on the WAN "group" interface.
Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.
Thanks a lot.

I'll focus now on DHCP  ;)

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: CARP with multi-wan
« Reply #3 on: November 09, 2017, 03:27:53 pm »
This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work  ;)
Notice that some devices are receiving same IP from each DHCP server.

That is normal. They will both offer since they are both active, but whichever lease the client accepts will be shared between the two systems.

Problem was with FW rules for incoming flow on the WAN "group" interface.
Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.

Great!
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!