next »

[mevans336]

Hello all,

I'm seeing legitimate traffic being blocked from LAN -> WAN, even though I have all LAN -> WAN traffic allowed. Here is the firewall log entry:

May 14 11:30:27     LAN     192.168.3.97:48663     147.249.x.x:80     TCP

I have pfSense in transparent bridge mode and the 192.168.3.97 server is behind an Apache load balancer, which actually performs NAT just like a router. So, the 192.168.3.97 box SHOULD be using the Apache Load Balancer (192.168.3.81) as it's gateway. The apache load balancer has a public IP address assigned to it's WAN port. An ifconfig in the OS of the 192.168.3.97 box confirms it is using 192.168.3.81 as it's gateway and it can get to the public internet through the load balancer just fine.

It appears this may be some type of threshold trigger as this is a query for our web application that may send 50-100 requests simultaneously.

Any ideas why is pfSense picking up this traffic and blocking it?

[mevans336]

Anyone?

If we can't figure out why this legitimate traffic is being blocked I'm going to have to spend money on a commercial firewall product. Please don't make me waste my money on that!

[GruensFroeschli]

To be honest: your text-explanation of your network might be clear to you, but to anyone that reads it, it's pretty incomprehensible without diagrams.

You could try to set under advanced in the rule config a value for the maximum simultaneous connection attemps from client.
I'm not exactly sure what the default values are.

[mevans336]

Thanks, here's a brief diagram that will hopefully help:

192.168.3.x Network (LAN) <------> NAT Gateway (192.168.3.81 Gateway, 207.99.x.x Public IP) <-------> pfSense in Bridged Mode (207.99.x.x IPs) <-------> Data Center Gateway of 207.99.x.x <-------> Internet


When a couple of servers on the 192.168.3.x network send several requests to a server across the internet, the firewall (pfSense in bridged mode) picks up the traffic from the IP of the server, from LAN to WAN, and sometimes blocks it. I have a couple LAN to WAN rules, one which is to allow all, and a few to not log local LAN broadcasts to a specific port.

Here is the entry logged in the firewall log: May 14 11:30:27     LAN     192.168.3.97:48663     147.249.x.x:80     TCP

The 147.249.x.x is the destination address across the internet. The 192.168.3.97 is one of the servers, behind the NAT Gateway on the 192.168.3.x network, which is having the issue. The issue seems to be a threshold limit, as it doesn't always happen.

Does this help at all or is it still as clear as mud?

[GruensFroeschli]

So your pfSense is at the WAN of your other router that does NAT.

But then it would make absolutely no sense why you're seeing traffic from your private space.

Are you sure that your NAT Gateway does the NAT as it should?

[mevans336]

So your pfSense is at the WAN of your other router that does NAT.

But then it would make absolutely no sense why you're seeing traffic from your private space.

Are you sure that your NAT Gateway does the NAT as it should?

That is exactly what I was thinking!

There are other servers that do not flow through the NAT boxes, as the NAT boxes are HTTP load balancers for our web infrastructure. So the pfSense server is not installed at the WAN of the NAT boxes, but rather a common switch that the servers which aren't behind the NAT boxes, along with the WAN of the NAT boxes, are all plugged into.

The servers behind the NAT boxes have their gateway as the LAN IP of the NAT box, 192.168.3.81, so I can't imagine why they'd attempt to talk directly to the pfSense server.

Any ideas?

[GruensFroeschli]

Maybe you should, to troubleshoot this problem, segment your network physically.
One physical segment for servers that access the balancer.
One for those that dont.

Then see again if the problem persists.

If it does you know for sure that your balancer is the problem.

[mevans336]

Maybe you should, to troubleshoot this problem, segment your network physically.
One physical segment for servers that access the balancer.
One for those that dont.

Then see again if the problem persists.

If it does you know for sure that your balancer is the problem.

The only problem with that is I am in NC and the servers are in NJ.

If I did this, I'd need 3 switches right? Or I'd need to use the VLAN capabilities of the two switches I have in there now.

One for the load balanced segment
One for the non-load balanced segment
One that connects back to both switches and the pfSense box?

Is there anything else I can look at, even if I have to drop to a shell on the pfSense box, that may give me more info?

[GruensFroeschli]

I meant something like this:


balanced subnet------- balancer ---------  pfsense------------upstream gateway
                                                          /
non balanced subnet ------------------- /


Your pfSense would have 3 NIC's
balanced,
nonbalanced,
WAN.



buuuut: what do you use the balancer for?
Can it do something the balancer in pfSense cannot do?

[Perry]

Could it be a service on the webserver telling 147.249.x.x to use the local ip instead of the public?

[mevans336]

I meant something like this:


balanced subnet------- balancer ---------  pfsense------------upstream gateway
                                                          /
non balanced subnet ------------------- /


Your pfSense would have 3 NIC's
balanced,
nonbalanced,
WAN.



buuuut: what do you use the balancer for?
Can it do something the balancer in pfSense cannot do?

We purchased the load balancers 2 years ago before pfSense was ever in the picture unfortunately. When I installed pfSense a few months ago, I kicked myself because I noticed right there, http load balancing. You don't even want to know what the High Availability pair of devices cost.

Our pfSense box has the capability for a 3rd NIC, so that is definitely an option, and a cheaper one than a Cisco ASA device. Are you fairly confident that would resolve the issue? Is there anything I can look at first? Installing a 3rd NIC will require me to fly to NJ, take our entire infrastructure offline, install the NIC and reconfigure pfSense. We're a very small company, so it's a pretty big deal for us.

[mevans336]

Could it be a service on the webserver telling 147.249.x.x to use the local ip instead of the public?

Can you elaborate a little here?

The webservers have two IP addresses assigned to them on physically different adapters. One on the 192.168.2.x subnet for management purposes, backups and SSH access, but no gateway, and their traffic adapter on the 192.168.3.x network, with a gateway of 192.168.3.81 for all inbound/outbound public traffic.

As far as I know, I haven't bound the webservers to a specific IP and they listen on all available IP addresses.

[Perry]

I can try
You say that server with ip 192.168.3.97 get block in firewall when trying to send to ip 147.249.x.x
and the problem it's only with ip 147.249.x.x
As you don't have a host directly connected to pfSense with the ip 192.168.3.97 something must publish that ip to pfSense.