Netgate SG-1000 microFirewall

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Heimire

Pages: [1] 2 3
General Questions / Gateway switching
« on: February 15, 2018, 10:47:23 am »
We have a setup that does not work unless we turn on gateway switching.

We use HA in with 2 WAN connections.
All LAN subnets are using a failover gateway group for the rules.
If we do a carp fail over it works fine.

If we shut down primary circuit by killing the switch port the BGP fails over normally but we lose all traffic.
If we turn on gateway switching it works.

Turned off HA and shut down secondary firewall.
So its now only running multi wan.
Same problem.  Its like it will not use the other gateway.

WAN1 gateway is default and tier 1.
WAN2 gateway is tier 2.

Can someone explain the potential drawbacks to the gateway switching?

General Questions / 2.4.2 in HA mode NBNS storm kills wan
« on: January 29, 2018, 09:08:11 pm »
We are running 2.42 in HA mode.
We have 6 nics with different subnets, etc.

One subnet is using 172.22.22.x
If we by mistake enter or 172.2.22.xx (or any other combo for the second octet but not tested) it will create a packet storm that makes our ISP shut us down.
They auto kill at 25% of total bandwidth for the interface and its a 1 gb port for storms.

So they kill our wan connections but our connections are connected to a switch so they stay up on our side.

After we shut down the server with the mistake on, the firewalls continued the storm.  I know this since I did a packet capture.
I captured probably 10-15 minutes after the mistake was corrected and the server was shut down but the packet storm is still there.

I had to disable the WAN interface on the primary firewall for it to stop.

So this makes a simple typo shut down our circuits.

We really need to figure out how to prevent pfSense from doing this and we are desperate for suggestions.

so if you have any suggestion, please let us know.

General Questions / Added limiter resulted in spontaneous reboots
« on: January 16, 2018, 10:51:24 am »
pfsense 2.4.2 in HA mode.

Steps taken to create this mess.
On primary.
Added traffic limiter by:
Firewall/traffic shaper
Added new
Name: l3df
bandwidth 15mb
mask: source address
Rest default

Then added to a rule
edit rule
Selected the limiter for In pipe.

Hit save.

It made the primary firewall reboot.
Come up for about 15 seconds then reboot.
This continued none stop.

It replicated the settings to the backup firewall.
The backup firewall did the same thing but it crashed the file system and never came back up at all.

I managed to get into the firewall and disable the limiter and that fixed the primary. (took over an hour).
On the backup firewall I had to fix the file system and then it came backup.

Its pretty scary that a simple mistake like this will shut down both your primary and secondary.

It would be nice to have a delay in replicating firewall rules that can kill your primary. 

I assume there are no way to delay firewall rules/settings replication to prevent situations like this.

I have no idea why this is taking place and its really bugging me.

pfSense 2.4.2_P1 in HA setup.
In a data center so our gateways are in the same room pretty much.
Its a 1gb LAN connection.

The dashboard shows RTT to between 2 and 12ms and RTTsd can be over 20ms.
But if I ping the gateway from the firewall its less than 1ms.  Same from backup firewall.

If I ping from a server on the LAN side to the same gateway I am seeing sub ms too.
So pinging appears to be normal but dashboard is showing different numbers.

The data center is using VSF I am sure since the gateways can't be pinged if we are not connected.

I took a simple pfsense box and plugged it in to the same port as the primary and it shows the same high RTT/RTTsd.
Loaded 2.3.5 on the box but nothing changed, I was thinking it could be a bug, or something.

Moved that same firewall to another connection in the same data center (onsite tech workstation area) and it shows normal sub ms RTT and normal RTTsd.

Any idea why this is taking place. 
No idea why this is bugging me so much but I might need mental help to get over this :)


Packages / FRR BGP Config example request.
« on: January 03, 2018, 05:27:54 pm »

I am trying to make FRR BGP work for us.
It works but I think we are seeing a long fail over time.

If I reboot the primary it can take 2 minutes before we get connectivity again.

Or if I disable CARP on primary the connection goes down for 8 seconds then comes backup for some seconds, goes back down for 9 seconds then comes up again.
The seconds vary.

We are in a data center with 2 connections to the cabinet.
A /29 for each firewall. 
FRR is running on both.

I am not sure if I can do anything about speeding up the fail over.

I feel like I am missing the obvious but not sure where to look.
Any suggestions?

BGP configuration primary.
##################### DO NOT EDIT THIS FILE! ######################
# This file was created by an automatic configuration generator.  #
# The contents of this file will be overwritten without warning!  #
password Super.1346
log syslog

# BGP Config
router bgp 18599
  bgp log-neighbor-changes
  bgp router-id
  timers bgp 6 20
  address-family ipv4 unicast

  # BGP Neighbors
  neighbor remote-as 3900
  neighbor description Primary Datafoundry
  address-family ipv4 unicast
    neighbor activate
    no neighbor send-community
    neighbor next-hop-self
    neighbor soft-reconfiguration inbound
  neighbor remote-as 3900
  neighbor description Backup Datafoundry
  address-family ipv4 unicast
    neighbor activate
    no neighbor send-community
    neighbor next-hop-self
    neighbor soft-reconfiguration inbound

General Questions / duplicate echo reply received
« on: December 23, 2017, 09:37:32 am »
We are seeing this on both firewalls. 
They are located in a data center so the gateway is just a hop a way on a gig connection.

We have 2 different circuits using BGP and CARP.

Time   Process   PID   Message
Dec 23 08:09:33   dpinger      WANGW duplicate echo reply received
Dec 23 07:57:35   dpinger      WAN2GW duplicate echo reply received
Dec 23 07:03:20   dpinger      WANGW duplicate echo reply received
Dec 23 06:36:04   dpinger      WANGW duplicate echo reply received
Dec 23 06:35:40   dpinger      WANGW duplicate echo reply received
Dec 23 05:37:53   dpinger      WAN2GW duplicate echo reply received
Dec 23 03:30:37   dpinger      WAN2GW duplicate echo reply received
Dec 23 03:14:43   dpinger      WAN2GW duplicate echo reply received
Dec 23 02:07:58   dpinger      WANGW duplicate echo reply received
Dec 23 00:39:56   dpinger      WAN2GW duplicate echo reply received
Dec 22 22:40:07   dpinger      WANGW duplicate echo reply received
Dec 22 22:27:43   dpinger      WAN2GW duplicate echo reply received

Routing and Multi WAN / FRR configuration
« on: December 22, 2017, 10:46:23 am »
Does anyone have a simple quide on how to configure FRR to work with BGP?

Routing and Multi WAN / 2.4.2 BGP working correctly?
« on: December 18, 2017, 04:19:04 pm »
We have a HA setup in one data center running 2.26.
We are using BGP with no problems.

In the new data center we are running another HA setup running 2.4.2.

We have 2 connections, we are using CARP and BGP.

The weird thing we are dealing with is that when we tell the primary firewall to disable CARP BOTH firewalls are closing the session so it takes a very long time to fail over.

This is what the provider sent me.
Dec 18 10:50:46 CST: %BGP-SW2-5-NBR_RESET: Neighbor reset (Peer closed the session) Dec 18 10:50:46 CST: %BGP-SW2-5-NBR_RESET: Neighbor reset (Peer closed the session) Dec 18 10:50:46 CST: %BGP-SW2-3-NOTIFICATION: received from neighbor 6/2 (Administrative Shutdown) 0 bytes Dec 18 10:50:46 CST: %BGP-SW2-3-NOTIFICATION: received from neighbor 6/2 (Administrative Shutdown) 0 bytes Dec 18 10:50:46 CST: %BGP-SW2-5-ADJCHANGE: neighbor Down Peer closed the session Dec 18 10:50:46 CST: %BGP_SESSION-SW2-5-ADJCHANGE: neighbor IPv4 Unicast topology base removed from session  Peer closed the session Dec 18 10:50:46 CST: %BGP-SW2-5-ADJCHANGE: neighbor Down Peer closed the session Dec 18 10:50:46 CST: %BGP_SESSION-SW2-5-ADJCHANGE: neighbor IPv4 Unicast topology base removed from session  Peer closed the session

Is it possible this is a bug or do I have something screwed up.  This is also the same setup where we see 2-8ms on the dashboard gateway screens but when you ping the gateways from the firewall or laptop its sub 1ms.

Our BGP config.
# This file was created by the package manager. Do not edit!

AS 18599
fib-update yes
holdtime 20
listen on
neighbor {
descr "WAN1 BGP"
remote-as 3900
set nexthop self
neighbor {
descr "WAN2 BGP"
remote-as 3900
set nexthop self
set prepend-self 2
deny from any
deny to any
allow from
allow to
allow from
allow to

webGUI / Dashboard gateway showing very high ping time
« on: December 14, 2017, 11:16:07 am »
Take a look at the images.
Its from a 2.4.2 firewall.

Notice how high those ping times to the gateways are when they are viewed from the dashboard.

Look how much less (normal) they are when I ping the gateways.

Does anyone know if its a bug in this version?

NAT / Bridging data centers
« on: December 04, 2017, 10:17:59 am »
We are moving to a new data center now.

We have a pfSense box at the old data center (2.2.6) and another in the new center (2.4.2).

Both sides will have the same LAN subnets.
We want to bridge a few of the LAN subnets for the purpose of the migration.

So it look like this:

Old center LAN Ė 192.168.20.xx bridged to New center LAN.
Old center NAT_LAN 192.168.30.xx bridged to new center NAT_LAN

So currently we have a server using IP, we want to move that to the new data center and make it appear on the old data center LAN.
We do not have any available interfaces on the firewalls.

Is there a way to do this?
If yes, please post up some suggestions.

We are currently using pfSense in HA mode with BGP.

We want to change both the firewalls to new hardware and we do have a new block of IP addresses.
The old firewalls are running 2.26 and I donít really want to do anything with those.

So my plan is to do this.
Install pfsense 2.26 on the new firewalls.
Export config from old firewalls.
edit the config.xml file.
Change the interface names and IP addresses.
Then import to new firewalls.
Upgrade those to 2.3 then upgrade to 2.4.x

Does this make sense or I am missing something here.

Installation and Upgrades / Pre upgrade questions HA with BGP
« on: August 06, 2017, 05:17:24 pm »
We are currently running 2.2.6 per node.
We have 2 nodes in a HA setup using BGP.

I plan to upgrade backup node first then if that works fine upgrade primary.

Is there anything to be concerned about or something thats not compatible with the latest version in this setup?


General Questions / Log analyzer for Snort/pfsense
« on: July 05, 2017, 10:12:47 am »

Can anyone recommend a log analyzer for Snort.
Want to see attacks and patterns.

Open source or free would be good.



Routing and Multi WAN / Full internet routes
« on: November 17, 2016, 05:24:40 pm »
One suggestion to solve a black hole problem is to receive full internet routes on the pfsense firewalls we have with BGP.

is pfsense capable of doing so.
Each firewall has 8gb of memory.

I have no idea of what it means, consequence or requirements doing so.
I have no idea how to make it happen either.

Looking for info to decide if this is a possible solution or not.



Routing and Multi WAN / BGP with routing/monitoring question
« on: November 17, 2016, 05:20:13 pm »
We have 2 firewalls running CARP with BGP.
Diverse path in the same data center.

One circuit hits a Houston router, the other hits a Dallas router.

Each firewall has 2 WAN interface with a /29 on each.

We have a /25 thatís announced via BGP.
One circuit is considered the primary.
Fail over the secondary works fine if we reboot the primary or physically pull the cable.

Monday the data center made a mistake and added a policy that pretty much black holed the BGP traffic.  The firewalls did not fail over to the second circuit.

So how can we make that happen?

If I set the monitoring IP to lets say for both gateways on the firewalls then set packet loss thresholds to lets say 50% or other metric.  So if the primary firewall cant ping that IP it will consider the route to be down.

Is it correct that the firewalls will update the BGP announcement to be the secondary circuit if that happens?

Pages: [1] 2 3