pfSense Support Subscription

Author Topic: pfBlockerNG v2.1 w/TLD  (Read 33744 times)

0 Members and 1 Guest are viewing this topic.

Offline RonpfS

  • Hero Member
  • *****
  • Posts: 684
  • Karma: +96/-2
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #60 on: August 13, 2016, 09:25:32 pm »
You have the URL that generate the errors so I can reproduce here?
2.3.5-RELEASE (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_1/Dev, suricata 4.0.1

Offline reg1982

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #61 on: August 14, 2016, 10:52:17 am »
I have been surfing the web to find one. Just cause i'm trying I am having a hard time.

This site did it once on my desktop but didn't do it on my phone.

https://www.instantssl.com/ssl-certificate-products/https.html

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #62 on: August 21, 2016, 09:36:11 pm »
Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.

Rob

Hi Rob,

The memory issue will be fixed with v2.1.1_3, however, you don't want to reverse the "Registered" vs the "Represented" entries.  Please refer to the link in the GeoIP tabs "Whats new in GeoIP2" to help you understand the difference between those two types.
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #63 on: August 21, 2016, 09:52:48 pm »
Here is a link to PR # 175 for pfBlockerNG v2.1.1_3      (This PR first needs to be reviewed and merged by the pfSense Devs)

The 2.1 release was beta tested for several months with approx a dozen testers with varying hardware (1GB-16GB, i386-AMD64). The MaxMind database is updated the first Tuesday of each month.

After reviewing the latest MaxMind IPv6 database, you can see below that the IPv6 line count increased 5 fold vs the previous month. This is a significant increase and as such the package required more PHP memory to be able to process the updated MaxMind database. The two Countries that changed significantly are US and DE, so until MaxMind has resolved this issue, you might consider not using those two IPv6 GeoIP lists.

This month:

1,147,813 US_v6.txt
1,137,159 DE_v6.txt

Last Month:

222,937 US_v6.txt
205,571 DE_v6.txt

I have contacted MaxMind support, to get some clarity on this issue, with the following response:
Quote
Thank you for contacting support. We did also observe a significant increase in IPv6 mappings, due to more specific blocks being mapped, starting with the 2016-07-05 release, and we are currently investigating what may be causing such an increase in the recent releases.

We do indeed aim to list the IP networks as efficiently as possible to help keep CSV file sizes down, so ideally the file sizes should not continue to increase dramatically once a fix is deployed. However, for the time being, the additional mappings shouldn't adversely affect the lookup results.

Thank you for the additional information; I've passed along your observations to our developers. At this time, we unfortunately do not have an ETA on a fix, but when I do receive any news, I'll be in touch.

I have re-factored the code to be able to handle this change in database size. This will reduce the overall PHP memory required. Its not recommended to "Block the world"; however, should your configuration follow this approach, then you may need to increase the pfSense Advanced "Firewall Maximum Table Entries" to 4M (or higher depending on the other Table entry size).

In my absense (vacation), forum user RonpfS steped up and helped convey some temporary workarounds and help users who were affected by this issue. I would personally like to extend my appreciation for all of his efforts. Its what "Open Source" is all about, and I encourage more people to get involved.

Everyone needs to bump his Karma! Thanks again!


Additional Changes:
  • Added a 'placeholder' for undefined MaxMind 'Represented Countries'. This is necessary as month-to-month MaxMind Updates seem to have fluctuations that can cause a list to become undefined/redefined.
  • Improved DNSBL Firewall Permit Rule options (Added OpenVPN, IPsec and Interface group options)
  • Improved removal of DNSBL VIP address mapping when DNSBL is disabled.
  • Added DNSBL parser for Alienvault OTX pulses. This will only collect "Domains". You can add the same feed into the IPv4 tab to collect "IPs".
  • Added a "Disabled" option to the CRON update options.
  • Additions to the DNSBL TLD suffixes
  • Fixed issue with widget not clearing DNSBL packet counts
« Last Edit: August 22, 2016, 10:00:51 am by BBcan177 »
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline hulleyrob

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +1/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #64 on: August 22, 2016, 04:15:34 pm »
Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.

Rob

Hi Rob,

The memory issue will be fixed with v2.1.1_3, however, you don't want to reverse the "Registered" vs the "Represented" entries.  Please refer to the link in the GeoIP tabs "Whats new in GeoIP2" to help you understand the difference between those two types.

Not sure if we are talking about the same thing there, i was referring to the North American IPV6 tab which my selection from a few countries which i picked changed to unselecting those ones and selecting all the ones i had not picked.

This is what i meant by it had reversed the selection and seemed to be the cause of all my memory use as you have pointed out its is now several million.

Rob

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #65 on: August 23, 2016, 09:23:56 am »
pfBlockerNG v2.1.1_3 has been approved and merged and is now available to be installed/Upgraded.

I have noticed that some of the installation log messages are not appearing in the pkg install window. I am investigating that; however, the installation is still occurring in the background.

The MaxMind conversion will take a few mins to process, so wait for it to complete.

UPDATE

I pushed a fix for this just now. The pfBlockerNG version is now 2.1.1_4
« Last Edit: August 23, 2016, 10:38:54 am by BBcan177 »
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline Qinn

  • Full Member
  • ***
  • Posts: 135
  • Karma: +5/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #66 on: August 23, 2016, 10:55:10 am »
pfBlockerNG v2.1.1_3 has been approved and merged and is now available to be installed/Upgraded.

I have noticed that some of the installation log messages are not appearing in the pkg install window. I am investigating that; however, the installation is still occurring in the background.

The MaxMind conversion will take a few mins to process, so wait for it to complete.

UPDATE

I pushed a fix for this just now. The pfBlockerNG version is now 2.1.1_4

Thanks I noticed that the log while updating to 2.1.1_3 didn't gave sign it finished, after updating to 2.1.1_4 all seems well ;)

btw I would like to test a php /usr/local/www/pfblockerng/pfblockerng.php dc but as I have dramatically changed the hardware I cannot compare it to when the memory issues occurred (see https://forum.pfsense.org/index.php?topic=102470.750 )!


Cheers Qinn
« Last Edit: August 23, 2016, 11:02:08 am by Qinn »

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #67 on: August 23, 2016, 11:11:42 am »
Thanks I noticed that the log while updating to 2.1.1_3 didn't gave sign it finished, after updating to 2.1.1_4 all seems well ;)

It was still running in the background (v2.1.1_3). The issue was that it wasn't printing the log messages to the installation window. So if you would have left it running for a min or so, it would have completed. Its now fixed in v2.1.1_4

Quote
btw I would like to test a php /usr/local/www/pfblockerng/pfblockerng.php dc but as I have dramatically changed the hardware I cannot compare it to when the memory issues occurred (see https://forum.pfsense.org/index.php?topic=102470.750 )!

The code was re-factored to not use as much PHP memory, so hopefully no one else runs into those issues :) Still hoping that MaxMind fixes the issues that caused those two Countries IPv6 entries to explode 5 fold....
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline Qinn

  • Full Member
  • ***
  • Posts: 135
  • Karma: +5/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #68 on: August 23, 2016, 12:35:46 pm »
Thanks I noticed that the log while updating to 2.1.1_3 didn't gave sign it finished, after updating to 2.1.1_4 all seems well ;)

It was still running in the background (v2.1.1_3). The issue was that it wasn't printing the log messages to the installation window. So if you would have left it running for a min or so, it would have completed. Its now fixed in v2.1.1_4

Quote
btw I would like to test a php /usr/local/www/pfblockerng/pfblockerng.php dc but as I have dramatically changed the hardware I cannot compare it to when the memory issues occurred (see https://forum.pfsense.org/index.php?topic=102470.750 )!

The code was re-factored to not use as much PHP memory, so hopefully no one else runs into those issues :) Still hoping that MaxMind fixes the issues that caused those two Countries IPv6 entries to explode 5 fold....

Thanks for the quick relpy! Yeah, that's the hardest part of coding, making it idiot proof for both users and resources. In this case it was the latter ;)
« Last Edit: August 23, 2016, 01:08:32 pm by Qinn »

Offline Qinn

  • Full Member
  • ***
  • Posts: 135
  • Karma: +5/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #69 on: August 23, 2016, 12:39:48 pm »
Maybe it's n=1 and is it just me, but after update to 2.1.1_4 unbound won't come up, I did a reboot let's wait and see.
« Last Edit: August 23, 2016, 12:45:29 pm by Qinn »

Offline jrdnlc

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #70 on: August 23, 2016, 10:47:24 pm »
Anyone have issues with pfBlocker and Playstation 4 online gaming? While playing online games lag a lot that the only fix was to disable pfb.
The logs show nothing of what exactly is blocking it.

Is there a way to exclude the PS4 to not use the service?

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #71 on: August 23, 2016, 10:54:28 pm »
Anyone have issues with pfBlocker and Playstation 4 online gaming? While playing online games lag a lot that the only fix was to disable pfb.
The logs show nothing of what exactly is blocking it.

Is there a way to exclude the PS4 to not use the service?

Did you review the pfBlockerNG Alerts Tab? If its being blocked via an IP List, it will show in the logs.  For DNSBL it should also show in the Alerts Tab. For DNSBL there are some further instructions listed in the DNSBL tab which can be seen when you click on the blue infoblock icon in the INFO section.  If it is being blocked by DNSBL and you can't find the Domain thats being blocked, you can set the DNS settings of the LAN device to a different DNS server to bypass DNSBL.
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline brandur

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +4/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #72 on: August 24, 2016, 08:44:12 am »
Hi @BBcan177
I just wanted to inform you that the info link/icon (to the right of the update ion), links to the wrong forum thread.

It's pointing to (pfBlockerNG v2.0 w/DNSBL): https://forum.pfsense.org/index.php?topic=102470.0
When it should be pointing to (pfBlockerNG v2.1 w/TLD): https://forum.pfsense.org/index.php?topic=115357.0
SG-4860 w/128GB SSD & 8GB RAM

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2554
  • Karma: +797/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #73 on: August 24, 2016, 09:36:40 am »
Hi @BBcan177
I just wanted to inform you that the info link/icon (to the right of the update ion), links to the wrong forum thread

Thanks good catch! Will change that when I submit the next release :)   

Wow! This thread had over 1000 views since last night  ;) 
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline Qinn

  • Full Member
  • ***
  • Posts: 135
  • Karma: +5/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #74 on: August 24, 2016, 11:45:09 am »
Hi @BBcan177
I just wanted to inform you that the info link/icon (to the right of the update ion), links to the wrong forum thread

Thanks good catch! Will change that when I submit the next release :)   

Wow! This thread had over 1000 views since last night  ;)

That's high 1k over night! For now 2.1.1_4 is running 22 hours without a flaw ;) . I did a php /usr/local/www/pfblockerng/pfblockerng.php dc and all went right (took about 25 min, but that is to be expected as of the dramatic raise of the resources of MaxMind).