Netgate m1n1wall

Author Topic: pfBlocker  (Read 182415 times)

0 Members and 1 Guest are viewing this topic.

Offline kilthro

  • Full Member
  • ***
  • Posts: 174
    • View Profile
Re: pfBlocker
« Reply #390 on: February 02, 2012, 08:15:40 am »
The first try is always to increase max_table_entries on system advanced.

Most of Cannot allocate memory from  /tmp/rules.debug are related to this.

Ok I will try that and see if that helps. I didnt want to touch anything until i asked. Its just strange, that its been fine for a couple of months with the same tables ect. Then just started happening. Then if i delete the list/table and rebuilt it was fine for a little while. I appreciate the feedback. This evening I will increase it and see if that removes the problem.

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9930
    • View Profile
Re: pfBlocker
« Reply #391 on: February 02, 2012, 08:18:57 am »
This evening I will increase it and see if that removes the problem.

It does not affect firewall function at all, you can apply it any time.

Offline kilthro

  • Full Member
  • ***
  • Posts: 174
    • View Profile
Re: pfBlocker
« Reply #392 on: February 02, 2012, 08:52:20 am »
This evening I will increase it and see if that removes the problem.

It does not affect firewall function at all, you can apply it any time.

Good to know, Thanks. I will not be back at that location until this evening. I just get email notifications when something is working right. :-)

Offline kilthro

  • Full Member
  • ***
  • Posts: 174
    • View Profile
Re: pfBlocker
« Reply #393 on: February 02, 2012, 08:37:49 pm »
Well the default was set to 200k so i increased to 900k and have refreshed a few times and no more errors.. Strange how this just started happening recently with out anything else changing on the firewall/setup.

Oh well.. Thanks again for the suggestion.


Offline tommyboy180

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 976
    • View Profile
    • TomSchaefer.org
Re: pfBlocker
« Reply #394 on: February 02, 2012, 09:04:23 pm »
Well the default was set to 200k so i increased to 900k and have refreshed a few times and no more errors.. Strange how this just started happening recently with out anything else changing on the firewall/setup.

Oh well.. Thanks again for the suggestion.



Any packages recently installed? What about any system restarts?
-Tom Schaefer
SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM
TomSchaefer.org/pfsense
Please support Countryblock | IP-Blocklist | File Browser | Strikeback Here

Offline kilthro

  • Full Member
  • ***
  • Posts: 174
    • View Profile
Re: pfBlocker
« Reply #395 on: February 03, 2012, 07:21:17 am »
Well the default was set to 200k so i increased to 900k and have refreshed a few times and no more errors.. Strange how this just started happening recently with out anything else changing on the firewall/setup.

Oh well.. Thanks again for the suggestion.



Any packages recently installed? What about any system restarts?

Nope nothing installed. Snort is blinking in the package widget as there is an update for that but I haven't done it yet since I typically have to reset all of the stuff back up in it when it updates.. It runs flawlessly but when I upgrade it, it always seems to flaky so I have put that off for now lol..  The system has been up for 50 days+ straight with no restarts. Honestly I have configured it, and for the most part forget about it unless I get an email stating something errors out ect.. Every now and then I may log into it to check for updates and status on things but that's not regular.

When i was increasing the amount I did tried 400k and 600k with the same error. 900k was the first where it didn't error out so that what I stuck with. This morning when everything updated I didn't get an error so it seems like that amount is ok for now.
« Last Edit: February 03, 2012, 07:30:21 am by kilthro »

Offline Amarth

  • Jr. Member
  • **
  • Posts: 30
    • View Profile
Re: pfBlocker
« Reply #396 on: February 11, 2012, 07:16:08 am »
This has probably been covered but I could not find any info on it. There seems to be a formatting issue on the pfBlocker Dashboard Widget.

Line: 69
Code: [Select]
print "<pre>";

Causes the column labels to offset to the right. ie: Alias ends up over the CIDR count.  I saw no closing so commented it out and the formatting now works.
« Last Edit: February 11, 2012, 07:19:06 am by Amarth »

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9930
    • View Profile
Re: pfBlocker
« Reply #397 on: February 11, 2012, 09:18:07 am »
It's a missing debug cmd prior to var_dump.

If It's on 1.0.1 package version, I'll remove on next release.

Offline vlurk

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: pfBlocker
« Reply #398 on: February 12, 2012, 11:51:34 pm »
It looks like I am having the "cannot allocate memory" issue as well, running a 2.0.1 (i386) release, nanobsd 2G and pfBlocker 1.0.1. I increased the max table size, read page 21 as well but it just wouldn't cut it with my small platform.

Since I REALLY wanted to get the Bluetack's level1 blocklist, I needed to try something else and eventually found another solution...

What I did is write a simple perl script with that will read the list, and output a new CIDR list splitted into many files, actually 100,000 entries per file.

I then created a small cron job to download the list and execute the script on a linux server running Apache.  Finally, I configured three different blocklists (aliases) under pfblocker (not just a big one with three URLs...).  And it works!  Maybe having such a mechanism (splitting big files) built into pfblocker could be useful for some.

In case someone is interested by the perl script, it looks like this:
Code: [Select]
#!/usr/bin/perl
use Net::CIDR::Lite;

my $filenum = "0" x 4; # if you ++ a string, it keeps the padding

sub new_file {
        $filenum ++;
        my $name = "splat_$filenum.lis";
        open OUT, ">$name" or die "canne open $name cap'n:$!\n";
        warn "writing to:$name\n";
}


my $cidr = Net::CIDR::Lite->new;
open (MYFILE, $ARGV[0]);
while (<MYFILE>) {
  chomp;
  $_ =~ /[^:]+:(.*)/;
  my $range = $1; #extracted IP Range, verify it is IPv4
  if ( $range =~ m/\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}/i ) {
        $cidr->add_range($range);
  }
}
close (MYFILE);
my $index = 0;
my @cidr_list = $cidr->list;
foreach my $block ( @cidr_list ) {
        if ( $index % 100000 == 0 ) {
                new_file;
        }
        print OUT $block,"\n";
        $index++;
}
close (OUT);

It receives the unzipped list in input, and will output the files in the CWD. Simple as that.

Have a nice day.

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9930
    • View Profile
Re: pfBlocker
« Reply #399 on: February 13, 2012, 04:30:46 am »
Good contributon. Thank you. :)

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9930
    • View Profile
Re: pfBlocker
« Reply #400 on: February 19, 2012, 08:22:35 pm »
Just updated pfBlocker to 1.0.2 with:

  • Fix on array check error at line 368 when there is no alias defined on pfSense
  • reduce duplicate cases on automatic rules when using multiple interfaces as inbound and/or outbound
  • Increase php memory limit to 250Mb when x64 pfSense is detected(DO AT YOUR OWN RISK PATCH applied to code ;))
  • Updated country ip lists
« Last Edit: February 21, 2012, 01:41:37 pm by marcelloc »

Offline tommyboy180

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 976
    • View Profile
    • TomSchaefer.org
Re: pfBlocker
« Reply #401 on: February 21, 2012, 06:46:22 am »
Awesome!
-Tom Schaefer
SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM
TomSchaefer.org/pfsense
Please support Countryblock | IP-Blocklist | File Browser | Strikeback Here

Offline taryezveb

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: pfBlocker
« Reply #402 on: February 21, 2012, 12:24:43 pm »

Offline vlurk

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: pfBlocker
« Reply #403 on: February 21, 2012, 07:46:46 pm »
Thanks for the update: I am gonna give it a shot like right after I post this.

 I noticed that my script would fail when some blocklist would include multiple colons on one line. Here is my updated script, which now accept a number of lines as the second argument.

Code: [Select]
#!/usr/bin/perl
use Net::CIDR::Lite;

my $filenum = "0" x 4; # if you ++ a string, it keeps the padding
my $limit = 100000; # default max number of lines

sub new_file {
$filenum ++;
my $name = "splat_$filenum.lis";
open OUT, ">$name" or die "canne open $name cap'n:$!\n";
        warn "writing to:$name\n";
}


my $cidr = Net::CIDR::Lite->new;
open (MYFILE, $ARGV[0]);
if ( defined($ARGV[1])) {
if ( $ARGV[1] =~  m/^\d{2,6}$/ ) {
$limit = int($ARGV[1]);
}
}

while (<MYFILE>) {
  chomp;
  my @line = split(/:+/);
  my $range = $line[-1]; #get IP Range, verify it is IPv4
  if ( $range =~ m/^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$/ ) {
$cidr->add_range($range);
  }
}
close (MYFILE);
my $index = 0;
my @cidr_list = $cidr->list;
foreach my $block ( @cidr_list ) {
if ( $index % $limit == 0 ) {
close (OUT);
new_file;
}
print OUT $block,"\n";
$index++;
}
close (OUT);

Since my platform is not x64, and only have 256MB of RAM, I am not sure the new patch will fix the memory allocation issue for me... I am running with 60% memory used on average. Right now I am using a 60,000 lines as my maximum. 100,000 would seem to fail on some occasions.

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 9930
    • View Profile
Re: pfBlocker
« Reply #404 on: February 21, 2012, 07:58:07 pm »
Vlurk,

The memory patch is only for am64.
As we try to avoid file hacks, I've applied a value that is defined on config.inc but not reflected on gui.
As you have few memory available, the best option is the way you are doing or of course a hardware upgrade.

Thank's for the script update.