Netgate SG-1000 microFirewall

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Heimire

Pages: [1] 2 3 4 5 ... 8
General Questions / Re: OpenVPN failover with multi-WAN & CARP
« on: February 15, 2018, 05:59:13 pm »
CAnt you just set OpenVPN to use ANY.
Then setup DDNS to update the IP externally  and use the FQDN for the vpn client?

General Questions / Gateway switching
« on: February 15, 2018, 10:47:23 am »
We have a setup that does not work unless we turn on gateway switching.

We use HA in with 2 WAN connections.
All LAN subnets are using a failover gateway group for the rules.
If we do a carp fail over it works fine.

If we shut down primary circuit by killing the switch port the BGP fails over normally but we lose all traffic.
If we turn on gateway switching it works.

Turned off HA and shut down secondary firewall.
So its now only running multi wan.
Same problem.  Its like it will not use the other gateway.

WAN1 gateway is default and tier 1.
WAN2 gateway is tier 2.

Can someone explain the potential drawbacks to the gateway switching?

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 07, 2018, 04:47:45 pm »
There are no physical connection between any of the 3 switches.
2 of them is dumb netgear switches I purchased for this testing.
The hub is something very old and retired.

Only physical connection between them is via pfsense so I am a loss when you are saying its not pfsense.

Thank you for all your help, I really appreciate your input even if I am a bit confused.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 07, 2018, 04:20:21 pm »
Hint: It's not pfSense.

Please explain how it can't be pfsense.
Look at my network drawing.

2 dumb switches not connected to each other.
2 pfsense boxes
1 dumb hub.

The switches not connect to each other.
pfsense boxes are connected to switches via 2 wan interfaces.
pfsense boxes are connected to the dumb hub with 2 lan interfaces.

Only other devices involved is a laptop connected to the hub.

Since the switches/hub has NO connection between them, how can it not be pfsense?

Only thing that connects all the devices together is pfsense boxes.

primary pfsense is connected to switch 1 via wan
primary pfsense is connected to switch 2 via wan
primary pfense is connected to hub via lan.

backup pfsense is connected to switch 1 via wan
backup pfsense is connected to switch 2 via wan
backup pfense is connected to hub via lan.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 07, 2018, 02:45:58 pm »
Why is that MAC address in a pcap on WAN then? Seems you have some sorting out to do there. inside MAC addresses should never be on the WAN layer 2.

Thats what all of this has been about.
Trying to sort that out.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 07, 2018, 01:49:51 pm »
Thats the VM that we use for this testing.
Its a Windows 2012 server that we just changed the IP to create this situation.
It is sitting on the Hyper-V cluster.

The problem is that if we by accident types in 172.222.x.x instead of 172.22.x.x it creates the storm.
We discovered that when one of the guys made exactly that mistake.

It does not happen if we turn off backup firewall, or disable the LAN side of the backup firewall.
The way we make the storm stop is to correct the IP and disable the WAN NIC on primary firewall.

I changed the rules for the LAN sections to only allow 172.22.22.x/23, that is the subnet we are using.
I also removed the policy by pass rule that you see in the image attached before.  Not sure why its there and if its needed.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 05, 2018, 02:21:20 pm »
So here is what I did.

Tested this and captured on LAN interfaces, then on WAN.
Those are attached, just remove .txt if you want to view them.

Turned off the policy route but that made no difference for this problem.

Then I changed the LAN rule to only accept traffic from our network.
That seems to have fixed it.

But I still have the policy routes disabled but not sure I really need them.
I thought I got that from the HA setup but cant recall where and why.

So I think this is a temporary fix but not sure of the consequence of disabling the rule yet.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 03, 2018, 04:15:02 pm »
Nothing bridged on the network.

Attached is the test network layout.
Its pretty simple.

I have the RFC1918 in the NAT_LAN rule set and I even notice I used the words Bypass policy route.
I can't recall why I included that one and I can't make sense out of it.
You can see it in the previous post image.

So that could be it but I am not in a position to test it remotely.

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: February 03, 2018, 02:45:39 pm »
So we broke everything down.

We have 2 separate dumb switches for the WAN interfaces.
Not connected to each other.

Sync cable is direct.

We connected each LAN interface into a dumb hub and we ONLY connected a single laptop.
On the laptop we change the IP from 172.22.22.xx to 172.222.22.xx and it creates a strom within seconds.

It still happens..

If I turn off the backup firewall the problem goes away. 
If I disconnect the LAN connection on the backup firewall the problem goes away.
So the only loop here is having the LAN interfaces connected to the same switch but thats how its designed.

So its not a loop in our switches.

I can't find a policy route that could do this.

I did follow the steps to set this up using the hangout presentation on 2.4 HA.
George (pfsense) did not see anything that looks funky when he looked but he did not look for this problem.

I have included screenshots of the lan rules and nat mode screen.
It shows we have several LANs but only one connected during testing is the NAT_LAN.  We also have a VPN connection going to another location but that IP is 192.168.30.xx

So at this point I am not sure where to go.
What kind of info do I need to provide to get some help?

General Questions / Re: 2.4.2 in HA mode NBNS storm kills wan
« on: January 30, 2018, 12:17:32 pm »
Thank you so much for answering.

I am pretty sure its not a policy route in place but I did not setup the switches.
The person who did said he doubled checked but we will go over it again.

I am not sure if spanning tree is enabled on the HP 5406 switches. I think they might be off by default.

Again, thank you so much for taking the time to respond.


General Questions / 2.4.2 in HA mode NBNS storm kills wan
« on: January 29, 2018, 09:08:11 pm »
We are running 2.42 in HA mode.
We have 6 nics with different subnets, etc.

One subnet is using 172.22.22.x
If we by mistake enter or 172.2.22.xx (or any other combo for the second octet but not tested) it will create a packet storm that makes our ISP shut us down.
They auto kill at 25% of total bandwidth for the interface and its a 1 gb port for storms.

So they kill our wan connections but our connections are connected to a switch so they stay up on our side.

After we shut down the server with the mistake on, the firewalls continued the storm.  I know this since I did a packet capture.
I captured probably 10-15 minutes after the mistake was corrected and the server was shut down but the packet storm is still there.

I had to disable the WAN interface on the primary firewall for it to stop.

So this makes a simple typo shut down our circuits.

We really need to figure out how to prevent pfSense from doing this and we are desperate for suggestions.

so if you have any suggestion, please let us know.

General Questions / Re: allow user to choose gateway 'on the fly'
« on: January 16, 2018, 12:35:11 pm »
"The problem is that with destination ip based rules I'm unable to choose which gateway should be used to connect to any specific site unless I login to pfSense and manually change the firewall rule for that destination ip."

Based on that line I assumed he had a firewall rule that was destination spesific.
If thats the case he should be able to assign a different gateway other than the default one in the advanced settings for the rule.

So that should work I think unless he wants to use for that rule different gateway based on what he needs at that one moment but why have a destination based rule?

General Questions / Re: allow user to choose gateway 'on the fly'
« on: January 16, 2018, 10:58:39 am »
So if you go to the rule and expand advanced options under Extra options settings at the bottom.
you find a gateway option there.

You can't use that to select the gateway to use?

Maybe I  not understanding the issue?

General Questions / Added limiter resulted in spontaneous reboots
« on: January 16, 2018, 10:51:24 am »
pfsense 2.4.2 in HA mode.

Steps taken to create this mess.
On primary.
Added traffic limiter by:
Firewall/traffic shaper
Added new
Name: l3df
bandwidth 15mb
mask: source address
Rest default

Then added to a rule
edit rule
Selected the limiter for In pipe.

Hit save.

It made the primary firewall reboot.
Come up for about 15 seconds then reboot.
This continued none stop.

It replicated the settings to the backup firewall.
The backup firewall did the same thing but it crashed the file system and never came back up at all.

I managed to get into the firewall and disable the limiter and that fixed the primary. (took over an hour).
On the backup firewall I had to fix the file system and then it came backup.

Its pretty scary that a simple mistake like this will shut down both your primary and secondary.

It would be nice to have a delay in replicating firewall rules that can kill your primary. 

I assume there are no way to delay firewall rules/settings replication to prevent situations like this.

Just did it and noticed I got a 29ms response time on one of the pings.
First time I see that. 

Ran it again and this time I see a 234ms ping. 


--- ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.188/23.642/234.569/70.309 ms

Well, that would certainly explain things. This could arise from a few things, but the most likely guess is the target device handles ICMP as a very low priority. You can confirm this by using a monitor address that is a little further out into the world.

As a general rule you want to use a monitor address that is physically on the other side of your WAN link. Some people use public addresses such as Google's DNS servers. For my monitoring, I use one of my ISPs regional concentrators.

You can use the mtr package to help you choose a suitable target. Run mtr with a target of and look at the hops along the way.

I think you hit it on the head.
This is still being setup and we have no live traffic there yet.
We are moving in there and just seen weird things we did not expect.

I will find some points to monitor outside the data center.

Thank you so much for your input.
Very helpful and I also realize I jumped to conclusion.
Should have done more than 3 ping when tested but they came back perfect every time.
I think when i did the testing earlier when i set the ping to 10 and ran it several times, I saw high numbers in probably 60-70% of the time.
Should have dug a bit deeper before posting.


Pages: [1] 2 3 4 5 ... 8