Netgate m1n1wall

Author Topic: Carp Failover and bridged Wan  (Read 3619 times)

0 Members and 1 Guest are viewing this topic.

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Carp Failover and bridged Wan
« on: June 16, 2012, 04:10:37 pm »
What is the current situation with pfSense and getting failover with 2 firewalls and bridged Wans?

It seems there have been plenty of people trying to acheive this basic functionality over the years but no real solution as of yet.
Has anyone managed to get this to work reliably without using STP to acheive it? Maybe someone could write a package with scripts to accurately monitor the Carp interfaces and bring up the bridge on the failover firewall to avoid layer 2 loops? I think this function would be welcomed by alot of pfSense users.

Any input would be apreciated because this is something i really need to get working soon or I will have to oabandon pfsense and go for a commercial solution ( and I REALLY love pfSense and don't want to change).

Offline podilarius

  • Hero Member
  • *****
  • Posts: 1768
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #1 on: June 19, 2012, 08:03:21 am »
I don't see how it can be done without STP. That is the bridge way of not setting up a loop. I guess CARP would work the same way if you could stop the loop with STP.

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #2 on: June 19, 2012, 09:42:43 am »
Unfortunately I have set it up with STP but it just doesnt work reliable enough. Either STP fails to witch routes to the failover firewall or a loop crashes the whole network.

I read somewhere on these forums that someone had written a script that would monitor the carp interface and if the main firewall failed then the script would bring up the bridge and this would prevent a loop from occuring. Apparently it worked fairly ok but not sure of the exact instructions to get it set up.


Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6333
  • Karma: +0/-0
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: Carp Failover and bridged Wan
« Reply #3 on: June 19, 2012, 08:06:15 pm »
I wouldn't rely on STP in such cases, too many possibilities for failure. Instructions here http://forum.pfsense.org/index.php/topic,4984.msg87793.html#msg87793

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #4 on: June 21, 2012, 03:34:00 am »
Thats exactly whati'm looking for. Thanks a mil.
Are there any improvements to this method that anyone is aware of?

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6333
  • Karma: +0/-0
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: Carp Failover and bridged Wan
« Reply #5 on: June 21, 2012, 03:11:09 pm »
The link I posted is the best you can do. I think we're the only firewall in the world that accommodates such a setup (short of rolling your own) because of the risks and complications inherent in redundant bridging firewalls. It's best to re-engineer your network so you don't need bridging if you want redundant firewalls, though a lot of people do exactly as in that link with no issue.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #6 on: June 24, 2012, 01:29:20 pm »
Note that in those instructions, on 2.0.x you'll need to match on "vip" as the subsystem, and on 2.1 the subsystem should be "[a-z]+[0-9]+_vip[0-9]+"

As someone who ran a CARP+Bridge setup for several years, let me say it was always a headache. I moved that to a completely routed setup and never looked back. It was well worth the time it took to transition.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #7 on: June 26, 2012, 01:35:35 pm »
When you say moved to a completely routed setup, I don't know if that's possible for my setup. I have a /24 ip pool from my Data Centre but the only way I thought I could get my web servers to be firewalled and still use these public IPs as their primary interface was with a transparent bridge.

Is there another way? I don't want to switch to a Nat setup.

Offline podilarius

  • Hero Member
  • *****
  • Posts: 1768
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #8 on: June 26, 2012, 01:51:20 pm »
Why would you want your Web servers to have real IP addresses. I run all my web servers behind a NATed firewall. It is safer IMO and also works better in more solutions. I also changed over to split horizon DNS and made sure my web pages used DNS name or relative path for self referrals.
Like Jim said, it would be worth the time to transition to that. I am sure there is another solution out there as well. There are many ways to tackle this problem.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #9 on: June 26, 2012, 01:55:13 pm »
Beg for an additional /29 from your DC.

Use the /29 for your CARP WANs and the DC/ISP gear

Use the /24 directly (you can route, not nat) on the LAN side

That's the typical datacenter style deployment for a firewall.

Things in your /24 use the firewall's CARP VIP IP in the /24 as their gateway. Your DC routes that /24 to your firewall's CARP VIP inside the /29.

No bridging required, much more reliable, none of the headaches.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #10 on: July 06, 2012, 06:14:48 pm »
I've managed to get a /29 subnet and here is what ive set up so far.

firewall 1:
Wan 200.200.200.2 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.1 (ip of existing /24 subnet)
Carp 172.17.1.1
Wan VIP 200.200.200.4
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on Wan

firewall 2:
Wan 200.200.200.3 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.2 (ip of existing /24 subnet)
Carp 172.17.1.2
Wan VIP 200.200.200.4
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on Wan
 
So I set up a server and gave it an ip of 100.100.100.5 and a gateway of 100.100.100.3 (Lan Carp VIP)

I can get on the net and it works well for outgoing traffic. I have rebooted firewall 1 and firewall 2 switches to master in a few seconds. Internet still works and all looks good. Firewall 1 reboots and switches back to master and internet still works.

But then I realise there is a problem. Going to whatismyip.com shows my IP as 200.200.200.4 instead of 100.100.100.5
This is a problem for all my webservers that need to have their own ip display for outbound traffic.

What have I done wrong?
I want this to be a fully routed setup and not just a Nat setup. How do I get my /24 ips to route correctly for outgoing traffic?

Thanks

« Last Edit: July 06, 2012, 06:20:26 pm by craggy »

Offline podilarius

  • Hero Member
  • *****
  • Posts: 1768
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #11 on: July 08, 2012, 09:49:42 pm »
Sounds like you still have NAT on or atleast natting to the wrong IP. Check your outbound NAT. You might have to turn off auto and go from there.

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #12 on: July 09, 2012, 03:03:23 am »
I've checked outbound Nat and it's definitely on Manual. I used the default rule and set the translation interface to the vid ip as suggested by the docs.
Maybe this is where I'm going wrong?

I've tried all the options from the translation drop down list and I even tried deleting all rules under manual outbound Nat in the hope it would route instead of Nat if no Nat rules were present but no luck. With no rules the test server can't even get out to the Internet.

Offline podilarius

  • Hero Member
  • *****
  • Posts: 1768
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #13 on: July 09, 2012, 06:27:15 am »
Since you are using internet route-able IPs, you don't need to NAT at all. You just have to make sure that the IP address your ISP routes to for your /24 addresses is your CARP VIP interface. Why do you have a CARP of 172.17.1/24, this is a private IP. It should probably be something like:

Firewall1:
WAN 200.200.200.3
WAN CARP VIP: 200.200.200.2
LAN: 100.100.100.2
LAN CARP VIP: 100.100.100.1
AON disabled and all rules removed.
If you are using DHCP, you are going to have to change the default setting so that it uses the LAN CARP ip for Gateway and such.

Firewall 2:
WAN: 200.200.200.4
WAN CARP: pulled from Primary
LAN: 100.100.100.3
LAN CARP: pulled from Primary
NAT: pulled from Primary

You will tell your ISP to route your /24 addresses to 200.200.200.2. Once that is complete, your should not have any problems.

Offline craggy

  • Jr. Member
  • **
  • Posts: 73
  • Karma: +0/-0
    • View Profile
Re: Carp Failover and bridged Wan
« Reply #14 on: July 09, 2012, 09:41:17 am »
Why do you have a CARP of 172.17.1/24, this is a private IP.


MAybe i explained my setup wrong. THe CARP ips i refer to is the private sync interface i.e a crossover cable between the two firewalls.

I have a CARP VIP added and it is one of the 3 ips from my /29 subnet.

firewall 1:
Wan 200.200.200.2 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.1 (ip of existing /24 subnet)
Carp 172.17.1.1 (this is private sync interface)
Wan VIP 200.200.200.4 (this is CARP IP on WAN from /29 range)
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on Wan

I havent yet asked the DC to route my /24 subnet to this CARP ip in the /29 because I wanted to be sure everything is wouking and i dont want any downtime.

The /29 and /24 are currently being presented on the same interface from the DC.
« Last Edit: July 09, 2012, 09:47:28 am by craggy »