pfSense Gold Subscription

Author Topic: EMERGENCY: pfsense not blocking ip  (Read 206 times)

0 Members and 1 Guest are viewing this topic.

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
EMERGENCY: pfsense not blocking ip
« on: August 12, 2017, 10:35:55 am »
Hi , we currently under DDoS attack to our apache2 web servers. In front of those servers there is a pfsense firewall with haproxy that load balances traffic.

We are adding a reject rule for the relevant ip but we observe that requests are still getting through!

Could this be cause the connection might be already established ?
Are firewall rules also applied on established connections?









Thanks
« Last Edit: August 12, 2017, 11:22:07 am by puter »

Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2073
  • Karma: +177/-11
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #1 on: August 12, 2017, 11:00:27 am »
I assume you added the reject source IP on your WAN and cleared all states pertaining to that IP?

On a side note. If it's a volumetric attack, there is nothing you can do. Rejecting the traffic doesn't stop it from reaching you in the first place. If it's a resource attack, design your system better.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8193
  • Karma: +956/-279
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #2 on: August 12, 2017, 11:06:36 am »
You will probably have to post your rules so someone can see what you did wrong.

Must be a block rule
must block the SOURCE address
Must be above the rule that passes the normal traffic in the rule set
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #3 on: August 12, 2017, 11:24:45 am »
I assume you added the reject source IP on your WAN and cleared all states pertaining to that IP?

On a side note. If it's a volumetric attack, there is nothing you can do. Rejecting the traffic doesn't stop it from reaching you in the first place. If it's a resource attack, design your system better.

clear all states pertraining to that IP? How do i do that?
Regarding the side note, there isnt any issues with the bandwidth, the problem is that connection reach application servers and generate high load   

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #4 on: August 12, 2017, 11:25:21 am »
You will probably have to post your rules so someone can see what you did wrong.

Must be a block rule
must block the SOURCE address
Must be above the rule that passes the normal traffic in the rule set

Thanks for the quick replay. I have included some screenshots of the rule.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8193
  • Karma: +956/-279
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #5 on: August 12, 2017, 11:48:12 am »
Diagnostics > States
Filter on the source IP address you are trying to block
Kill them

Probably not pertinent to your current issue but you really should upgrade to 2.3.4 at some point.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #6 on: August 12, 2017, 11:58:22 am »
Diagnostics > States
Filter on the source IP address you are trying to block
Kill them

Probably not pertinent to your current issue but you really should upgrade to 2.3.4 at some point.

I filtered with the source ip 45.77.40.157 and it didnt return anything...strange.

Could this has something to do with haproxy listening on port 80?

Offline chrcoluk

  • Full Member
  • ***
  • Posts: 293
  • Karma: +14/-50
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #7 on: August 12, 2017, 12:13:00 pm »
this for me is hard to diagnose as the only thing to work with is a screenshot of "part" of your rules configured in the GUI.

Ideally need logs, and full cli diagnosis.

Filtering on a software firewall helps if the attack is saturating resources behind the firewall such as e.g. available apache connections, it doesnt help if e.g. saturating the actual connection or PF states.

As said above

Make sure ip is not allowed by any other rules that might be processed ahead of reject rule.
Clear all states.
Make sure you covering the right protocol and connection flags.
pfSense 2.4
Braswell N3150 - 4 gig Kingston 1333 C11 DDR3L
60 gig kingston ssd - Jetway mini pcie 2x intel i350 lan

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8193
  • Karma: +956/-279
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #8 on: August 12, 2017, 01:04:17 pm »
As long as the block rule is above the rule that passes the traffic to HAproxy, no, it is just a connection on port 80.

Perhaps you should describe exactly what you are seeing in this "DDoS".
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!

Offline chpalmer

  • Hero Member
  • *****
  • Posts: 1693
  • Karma: +88/-1
    • View Profile
    • Home of Cablenut
Re: EMERGENCY: pfsense not blocking ip
« Reply #9 on: August 12, 2017, 01:25:51 pm »
Remember that if you make a rule during an "event" (connection attempt) as soon as you hit submit, clear the states ect..  any unsolicited attempts matching that firewall rule will be blocked/rejected..

But solicited attempts will not be. Your server will still be answering the earlier attempts and will still connecting outbound, now the traffic is solicited and will be allowed.

If I have an event which I want to stop then I tend to disable my servers outbound rule first, then the inbound, then clear the states.   

Do you still have an allow all rule for your servers outbound?  Maybe time to tighten that up a little..   You actually do not need any outbound rules for your server to answer a connection attempt from an inbound rule.


P.S. statements made by me are not necessarily condoned by the management of this fine organization.  http://badmodems.com

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #10 on: August 12, 2017, 06:23:54 pm »
Remember that if you make a rule during an "event" (connection attempt) as soon as you hit submit, clear the states ect..  any unsolicited attempts matching that firewall rule will be blocked/rejected..

But solicited attempts will not be. Your server will still be answering the earlier attempts and will still connecting outbound, now the traffic is solicited and will be allowed.

If I have an event which I want to stop then I tend to disable my servers outbound rule first, then the inbound, then clear the states.   

Do you still have an allow all rule for your servers outbound?  Maybe time to tighten that up a little..   You actually do not need any outbound rules for your server to answer a connection attempt from an inbound rule.


As long as the block rule is above the rule that passes the traffic to HAproxy, no, it is just a connection on port 80.

Perhaps you should describe exactly what you are seeing in this "DDoS".

I Finally figured out what was happening, thankfully before i got mad! I still need your help though.

This sneaky guy was crafting the "X-Forwarded-For" header and was using a fake ip address hence why when was blocking this ip, traffic was still shown in weblogs  (from this ip).

We tried a different approach by counting established connections and we quickly started figuring out ips that caused the DDoS. Adding those ips in firewall rules did the trick.


For clarification reasons, we did a "curl GET /" with an ip of "137.137.136.136" and indeed thats the ip (fake one) we show in our weblogs

Now, in another cluster where we use nginx as reverse proxy, changing/crafting the X-Forwarded-For Header doesnt work. Meaning , even if you enter a random ip in the specific header nginx will still pass the correct ip to the backend webserver?
 
Does this has to do in general with haproxy or is this a pfsense haproxy issue?
Anyone had such an issue ?

Once again, thanks for your help

Offline puter

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: EMERGENCY: pfsense not blocking ip
« Reply #11 on: August 13, 2017, 06:22:27 am »
Regarding X-Forwarded-For, this is a haproxy issue. In order to sort this out you must add the following parameter on haproxy frontend service in pfsense.

reqidel ^X-Forwarded-For:.*