pfSense Support Subscription

Author Topic: pfBlocker  (Read 183501 times)

0 Members and 1 Guest are viewing this topic.

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #300 on: December 30, 2011, 06:31:22 pm »
I mean that.  Sorry, I didn't saw the "[one at a time]"

No problem and guess no more lists for me :(

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #301 on: December 30, 2011, 06:42:40 pm »
The are some file hacks to increase php memory limit, but it is not supported or recommended by core team.

DO IT AT YOUR OWN RISK

edit ini_set("memory_limit","128M"); to 500M in /etc/inc/config.inc file

and include ini_set('memory_limit', '490M'); on pfblocker.inc

Do not increase it to a value greater then 510M, suhosin.memory_limit is set to 512M
« Last Edit: December 31, 2011, 06:54:09 am by marcelloc »

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #302 on: December 30, 2011, 07:13:19 pm »
edit ini_set("memory_limit","128M"); to 500M in /etc/inc/config.inc file

and include ini_set('memory_limit', '490M'); on pfblocker.inc

Do not increase it to a value greater then 510M, suhosin.memory_limit is set to 512M

I take it a reboot is needed for these changes to take affect?

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #303 on: December 30, 2011, 07:20:33 pm »
Not at all. There is no changes on php.ini

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #304 on: December 30, 2011, 07:31:29 pm »
Hmm, I edited the following to test:

/etc/inc/config.inc  and changed  ini_set("memory_limit","128M"); to 500M

/usr/local/pkg/pfblocker.inc  and added  ini_set('memory_limit', '490M');

Then tried to enable the TBG list, but still get the Fatal error. Did I do something wrong?

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #305 on: December 30, 2011, 08:34:30 pm »
Not sure, I did it on a x64 virtual machine and I do not get php limit errors.

All files are uploaded.

EDIT

With all these large files, my virtual machine gets 100% cpu for more then a minute and even with a very large table size, it reach virtual machine memory limit (384Mb).
« Last Edit: December 30, 2011, 08:38:53 pm by marcelloc »

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #306 on: December 30, 2011, 09:53:18 pm »
Something interesting about table-entries limit:

I've set up maximum table entries to a very high value 99999999999 at gui and could see it reflected on /tmp/rules.debug

BUT

on pfctl -sa I get this:
LIMITS:
states        hard limit  1000000
src-nodes     hard limit  1000000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit 1215752191


I saw my virtual machine freezing console with 99999999999 value, so I reduced it on gui to 5000000

EDIT

After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot

I get no erros with both large lists applied

as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc
« Last Edit: December 30, 2011, 10:18:25 pm by marcelloc »

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #307 on: December 30, 2011, 11:26:31 pm »
Something interesting about table-entries limit:

I've set up maximum table entries to a very high value 99999999999 at gui and could see it reflected on /tmp/rules.debug

BUT

on pfctl -sa I get this:
LIMITS:
states        hard limit  1000000
src-nodes     hard limit  1000000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit 1215752191


I saw my virtual machine freezing console with 99999999999 value, so I reduced it on gui to 5000000

EDIT

After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot

I get no erros with both large lists applied

as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc


Interesting, and  yes when I edited /etc/inc/config.inc I saw it already had  ini_set("memory_limit","256M"); . I have a x64 install. Nevertheless when I get a chance I will try your last settings again and reboot. Thanks for the info.

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #308 on: December 31, 2011, 08:58:04 pm »
EDIT

After 2G RAM, Firewall Maximum Table Entries set to 5000000 and a reboot

I get no erros with both large lists applied

as x64 version already has ini_set("memory_limit","256M");, I've just included ini_set('memory_limit', '250M'); on line two of /usr/local/pkg/pfblocker.inc


Yes! :) I tested the above and no more errors, sort of; I'll explain below.

I did the following:

Edited /usr/local/pkg/pfblocker.inc adding the following on the first line where the comments end[from the top]. Right after this line:
*/                                         <- this is line number 31
ini_set("memory_limit", "250M");  <- this would be line number 32

Then set the Firewall Maximum Table Entries: 5000000 and did a reboot. But received these errors:
Dec 31 18:13:44   php: : There was an error while parsing the package filter rules for /usr/local/pkg/squid.inc.

Dec 31 18:13:44   php: : The command '/sbin/pfctl -nf /tmp/rules.test.packages' returned exit code '1', the output was 'no IP address found for 550.164.3.166/32 /tmp/rules.test.packages:17: file "/var/db/aliastables/pfBlockerBluetack.txt" contains bad data'

After some testing those errors are due to using the following list:
http://list.iblocklist.com/?list=bt_proxy&fileformat=p2p&archiveformat=gz

NOTE: I tried several times removing and adding that list, with the same result.

Once I removed that list I got this:
Dec 31 18:46:28   php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was 'no IP address found for 75.35.185. /tmp/rules.debug:19: file "/var/db/aliastables/pfBlockerTBG.txt" contains bad data no IP address found for ÿ0.0.0.0/8 /tmp/rules.debug:95: file "/etc/bogons" contains bad data pfctl: Syntax error in config file: pf rules not loaded'

More testing and the error above was due to this list:
http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gz

But this time after removing the list above and re-adding the error went away. I guess there are some issues with those lists. Will try to contact the author to see if that can be fixed.

So in the end, no more errors and Squid works as a transparent proxy using pfBlocker as well now :)

For reference these are the lists that I'm currently using error free:

</pfblocker>
      <pfblockerlists>
         <config>
            <aliasname>Bluetack</aliasname>
            <description><![CDATA[Bluetack: level1-3, ads, spyware, badpeers, hijacked, dshield and bogon]]></description>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_level1&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_level2&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_level3&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_ads&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_spyware&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_templist&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_hijacked&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_dshield&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=bt_bogon&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <action>Deny_Inbound</action>
            <cron>EveryDay</cron>
            <custom/>
         </config>
         <config>
            <aliasname>TBG</aliasname>
            <description><![CDATA[TBG: Primary Threats, General Corporate Ranges, Hijacked and Bogon]]></description>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=ijfqtofzixtwayqovmxn&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=ecqbsykllnadihkdirsh&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=tbnuqfclfkemqivekikv&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <row>
               <format>gz</format>
               <url>http://list.iblocklist.com/?list=ewqglwibdgjttwttrinl&amp;fileformat=p2p&amp;archiveformat=gz</url>
            </row>
            <action>Deny_Inbound</action>
            <cron>EveryDay</cron>
            <custom/>
         </config>
      </pfblockerlists>


I would like to Thank marcelloc for his time looking into this. A donation was made[yesterday, before marcelloc posted the fix]; will donate again when I can. Hope this change is added to the package for all to benefit, after a few more days of testing. To make sure this change does not affect anything else.

EDIT: Happy New Year :)
« Last Edit: December 31, 2011, 09:16:13 pm by taryezveb »

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #309 on: December 31, 2011, 09:18:00 pm »
Thanks  ;D. Did you mentioned these packages developers on donation?
« Last Edit: December 31, 2011, 10:53:19 pm by marcelloc »

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #310 on: December 31, 2011, 11:01:56 pm »
Did you mentioned these packages developers on donation?

Yes, I broke it down on how the donation should broken up. Like so: $XX for the X package maintainer(s).

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9995
  • Karma: +3/-0
    • View Profile
Re: pfBlocker
« Reply #311 on: December 31, 2011, 11:06:45 pm »
I'll include this ini_set('memory_limit', '250M') patch when pfblocker detects a x64 version.

There are other places to edit it, maybe a redmine for that as diagnostics-> tables also has a 128Mb limit even on x64.


Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #312 on: December 31, 2011, 11:14:22 pm »
I'll include this ini_set('memory_limit', '250M') patch when pfblocker detects a x64 version.

There are other places to edit it, maybe a redmine for that as diagnostics-> tables also has a 128Mb limit even on x64.

Great and yes good point, since Diagnostics -> Tables gives a similar: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/www/diag_tables.php on line 89

But at least that does not affect things like the previous "Fatal error: Allowed memory size..." did.
« Last Edit: December 31, 2011, 11:45:20 pm by taryezveb »

Offline tommyboy180

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 976
  • Karma: +0/-0
    • View Profile
    • TomSchaefer.org
Re: pfBlocker
« Reply #313 on: January 02, 2012, 08:44:46 pm »
countryipblocks.net is testing their latest IPv4 Network Aggregation scripts and would like to have your help. They are in two formats:

PG2 and IP Range.

If you get a chance, please test them with us and give feedback so we can pass on to countryipblocks.

www.countryipblocks.net
-Tom Schaefer
SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM
TomSchaefer.org/pfsense
Please support Countryblock | IP-Blocklist | File Browser | Strikeback Here

Offline RonpfS

  • Full Member
  • ***
  • Posts: 129
  • Karma: +0/-0
    • View Profile
Re: pfBlocker
« Reply #314 on: January 03, 2012, 03:01:47 pm »
I ran into an issue with pfBlocker when trying to copy the configuration onto a new install.

I am using aliases only, a Block in rule on WAN, a Reject out rule on LAN.
I have the following packages: Backup 0.1.5, Cron 0.1.5, Filemanager 0.1.1, pfBlocker 1.0.1, Snort 2.9.1 v 2.0.2
Running pfsense 2.0.1 x86

1) I saved the configuration including RRD data

2) I restored the configuration to a fresh 2.0.1 install on the same machine

It rebooted and started to install packages, but pfsense could not access the internet because of the pfBlocker was not running and the missing aliases break the firewall.
I could not recover, removing rules didn' work. I was spending too much time trying things and failing, I ended up cloning the old disk instead.

What is the way to recover from the pfBlocker aliases problem?
just removing the rules in the GUI didn't work. I didn't try to remove the table or alias.
I remove the rules in the config.xml, to no available.
Today I see that I should have removed the alias. Is this enough?
I do not have a spare machine to do testing.

What is the best way to copy the configuration ?
Was I supposed to install all the packages before restoring the configuration?
This might have worked if the Restore config script did the package install before the reboot.
(or download the package now before the reboot to prepare for package install)

I guess removing the rules and disabling pfBlocker before saving the configuration should work.


« Last Edit: January 03, 2012, 05:36:08 pm by RonpfS »