pfSense Support Subscription

Author Topic: pfBlockerNG v2.1 w/TLD  (Read 33524 times)

0 Members and 1 Guest are viewing this topic.

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
pfBlockerNG v2.1 w/TLD
« on: July 18, 2016, 07:34:30 pm »
PR # 156/157 have been posted for pfBlockerNG v2.1.1

CHANGELOG:

MaxMind GeoLite2

New Changes here:
    https://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/

Highlights:
  • GeoLite2 data is already in CIDR format, so should be faster to process then the previous GeoLite data which was in Range format.
  • GeoLite2 data now includes "Represented IPs" along with "Registered IPs"... So the options now include Countries with "_rep".
  • Asia and Europe have an "Undefined" Network list which is now available to be used.
  • Localized Language options are available... See General Tab.
  • Add Antarctica Tab.
  • Downloads via HTTPS MaxMind URLs
  • Top 20 Spammers Tab is now auto-generated (as other GeoIP Tabs)

DNSBL TLD (Beta Feature)

DNSBL TLD is a new feature to determine if all Sub-Domains should be blocked for each listed Domain. TLD is more memory intensive and is not recommended for low performance/Low-Memory installations. TLD will limit the number of Domains that can be processed, Once the TLD Domain limit below is exceeded, the balance of the Domains will be listed as-is. IE: Blocking only the listed Domain (Not Sub-Domains).

    TLD Domain Limit Restrictions:

    < 1.0GB RAM - Max 100k Domains
    < 1.5GB RAM - Max 150k Domains
    < 2.0GB RAM - Max 200k Domains
    < 2.5GB RAM - Max 250k Domains
    < 3.0GB RAM - Max 400k Domains
    < 4.0GB RAM - Max 600k Domains
    < 5.0GB RAM - Max 1.0M Domains
    < 6.0GB RAM - Max 1.5M Domains
    < 7.0GB RAM - Max 2.5M Domains
    > 7.0GB RAM - > 2.5M Domains

When enabled and after all downloads for DNSBL Feeds have completed; TLD will process the Domains. TLD uses a predetermined list of TLDs, to determine if the listed Domain should be configured to block all Sub-Domains. The predetermined TLD list can be found in [    /usr/local/pkg/pfblockerng/dnsbl_tld    ]

Options to Blacklist whole TLDs with a provision to Whitelist specific Domain/Sub-Domains in these TLD Blacklists. With the TLD Whitelist option, Alerts will not populate, as the Domains are in a "Static" Resolver zone and as such DNS resolution is via NXDOMAIN.

Options to exclude certain TLDs and/or Domains from the TLD Process.

    Lists of worst TLDs:

    https://www.spamhaus.org/statistics/tlds/
    http://toolbar.netcraft.com/stats/tlds

    The TLD feature has so far been tested by approximately a dozen beta testers.


Other Improvements
  • Improve OpenVPN Auto-Rule options
  • Add IPSec Auto-Rule options
  • Add Malware Corpus Tracker to the DNSBL parser www.h3x.eu
  • DNSBL and Alexa Whitelisting has been improved to remove all Sub-Domains. This is accomplished by prefixing a "dot" before the Domain name in the Custom Whitelist.
  • Fix issue with the "XMLRPC Sync" tab - Disable Sync option of "General tab settings" was previously reversed
  • DNSBL Alerts Tab- The Whitelisting User Input popup has been improved.
  • Alerts Tab - Added an "Icon Legend" to the bottom of the page.
  • Escape Log Browser data before printing to screen.
  • Escape Update Tab log before printing to screen.
  • Add additional Alerts Tab Threat Lookups
    • Intel - Threat Intelligence (Formerly McAfee)
    • Threat Miner
    • Threat Crowd
    • Ransomware Tracker
    • Google Safe-Browsing
    • NetCraft Site Report
    • hpHosts
    • mnemonic Passive DNS
  • Other under-the-hood improvements

« Last Edit: July 18, 2016, 08:06:16 pm by BBcan177 »
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline luckman212

  • Hero Member
  • *****
  • Posts: 726
  • Karma: +59/-0
    • View Profile
    • @luckman212 - github
Re: pfBlockerNG v2.1 w/TLD
« Reply #1 on: July 18, 2016, 07:54:13 pm »
This sounds like an awesome update!  Thanks for your hard work.

I am trying to better understand what the new TLD feature enables us to do. Would it e.g. allow a captive portal to be set up which allows *.facebook.com (to enable Facebook logins) for example?

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #2 on: July 18, 2016, 08:24:32 pm »
The TLD feature is used by pfBlockerNG DNSBL for Domain blocking via the Unbound DNS Resolver.

When "TLD" is enabled... It checks each Domain to see what the TLD (Top-Level domain) is for each listed Domain in the DNSBL Blacklist Feeds... then if there is one more level, it will block the whole Domain since its the root Domain name....

When Feeds do not post the Full-Domain, then only the listed Sub-Domains are blocked... 
ie: ads.yahoo.com   Will ony block that Sub-Domain and not yahoo.com

Example 1:
download.101com.com

com                                - Top-Level Domain
101com.com                  - Second-level Domain
download.101com.com  - Third-level Domain

Example 2:
example.uk.com

uk.com                           - Top-Level Domain
example.uk.com             - Second-Level Domain


The DNSBL database is located at    /var/unbound/pfb_dnsbl.conf

When TLD is enabled, in that conf file you will see "transparent" zones, which means its only blocking the actual Domains listed...  None of the Sub-Domains are blocked...

Scroll down that file, and look for lines that have "redirect" zones, which means its blocking the full Domain and all Sub-Domains...

You can also block whole TLD(s) like   | cn  |  ru  | pw  |  xyz  | etc.... Option also exist to Whitelist specific Domains when the whole TLD is being blocked.

The following Unbound documentation has additional detail on the "Local-zone" configuration:
    https://unbound.net/documentation/unbound.conf.html
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline someuser123

  • Newbie
  • *
  • Posts: 17
  • Karma: +9/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #3 on: July 19, 2016, 09:45:43 am »
YeY ;D :D :) ;) huge update, awesomeness cant wait for this update, thanks for all your hard work....

Offline brandur

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +4/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #4 on: July 21, 2016, 04:40:21 am »
Sounds like a very exciting update :D
Thank you
SG-4860 w/128GB SSD & 8GB RAM

Offline Pippin

  • Full Member
  • ***
  • Posts: 241
  • Karma: +22/-3
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #5 on: July 21, 2016, 04:44:14 am »

Very nice addition, chapeau 8)
2.3.2-RELEASE (amd64) - GB N3150N-D3V
"There must be someone with intelligence in the party"
"Well, that rules you out Pippin"

Offline lpallard

  • Full Member
  • ***
  • Posts: 274
  • Karma: +3/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #6 on: July 23, 2016, 02:42:57 pm »
Cant wait for this update to roll out!!!!!

Offline zerodamage

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +2/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #7 on: July 26, 2016, 08:34:25 pm »
Is this available to install now?  I am only showing version 2.0.17 for update.  I uninstalled hoping maybe I would then see the updated version but it's still not there.

As a matter of fact, it says "Not Ready" in the update window when trying to install or update it now.  I now do not have it installed and am not able to install it but my system says it is installed.

It actually did install but only to version 2.0.17.  Is that right?
« Last Edit: July 26, 2016, 08:39:45 pm by zerodamage »

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #9 on: July 28, 2016, 10:33:06 pm »
The latest version of pfBlockerNG   v2.1.1_1  has been reviewed and merged into pfSense 2.3.3 Dev. If your on the 2.3.3 Snapshots, its available to be installed now.

I believe that the Devs will merge it for pfSense 2.3.2 shortly, so stay tuned for the update.

If you have any questions or Feedback, please let me know....

Please Read the instructions in the DNSBL tab for the new TLD feature before enabling it.
Once enabled, follow that with a "Force Reload - DNSBL".

Review any MaxMind GeoIP settings, since there have been significant changes with the upgrade to GeoLite2.

Note: If you have less than 5GB of RAM and you have added the Bambenek DGA DNSBL Feed, please move that to the last entry in the DNSBL Feeds. Since that feed is quite large (700k+ Domains), its best to allow TLD to process the other Feeds first before hitting the max TLD Domain limit.  (http://osint.bambenekconsulting.com/feeds/dga-feed.gz).
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline f34rinc

  • Jr. Member
  • **
  • Posts: 50
  • Karma: +21/-0
    • View Profile
    • legoclan
Re: pfBlockerNG v2.1 w/TLD
« Reply #10 on: July 29, 2016, 09:51:58 am »
Nice work BBcan177  :D  setup blocking of .ru as a test and it works.

Offline DownloadDeviant

  • Newbie
  • *
  • Posts: 18
  • Karma: +2/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #11 on: July 29, 2016, 12:46:39 pm »
THANKS! Can't wait! Good stuff....great work...and thanks for helping us dumb dumbs  :P here and over at Reddit!

PS - is there a quick n dirty way to test PFBNG to be sure you've generally set it up correctly? Like going to a website and not seeing ads, etc.?
System: pfSense 2.3.4p1 CPU: AMD Athlon 5350 (Kabini) MOBO: ASRock AM1H-ITX HD: 60GB SSD Patriot Inferno RAM: G.SKILL Sniper 2x4GB DDR3 2133 NIC: Intel I350-T2 CASE: Antec ISK 310-150 PS: Lite-On 75W AC Adapter PACKAGES: Cron
NAS: Synology DS415+

Offline mauroman33

  • Full Member
  • ***
  • Posts: 175
  • Karma: +23/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #12 on: July 29, 2016, 04:12:59 pm »
Thank you so much for this fantastic work!!!

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #13 on: July 29, 2016, 05:27:50 pm »
THANKS! Can't wait! Good stuff....great work...and thanks for helping us dumb dumbs  :P here and over at Reddit!

PS - is there a quick n dirty way to test PFBNG to be sure you've generally set it up correctly? Like going to a website and not seeing ads, etc.?

Thanks... Are you on the latest 2.1.1_1 version?   Haven't heard much feedback yet, so not sure if many have installed it yet...

Not sure what sites are the worst for ADs... but yahoo is probably up there....


Thank you so much for this fantastic work!!!
Thanks!
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline DownloadDeviant

  • Newbie
  • *
  • Posts: 18
  • Karma: +2/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #14 on: July 29, 2016, 07:02:30 pm »
Thanks... Are you on the latest 2.1.1_1 version?   Haven't heard much feedback yet, so not sure if many have installed it yet...

Not sure what sites are the worst for ADs... but yahoo is probably up there....

I'm still on 2.0.17. I've slowed down my updating a bit since I've had some snags and had to rebuild 3 times in the past 7 weeks. Two were my fault...lol I thought I had router plugged into the battery port on the UPS but didn't...storm hit...lost power...pf went corrupt. Sooooooooo, I'm a bit worn out on tampering right now. lol That said, I'll probably upgrade it this weekend.

Yahoo it is then. I'm very new to pfBNG so I need to learn it and get comfortable. I don't want to get  too aggressive. I just want it to serve as a companion for my Firefox plugins and to help keep my girlfriend protected.
System: pfSense 2.3.4p1 CPU: AMD Athlon 5350 (Kabini) MOBO: ASRock AM1H-ITX HD: 60GB SSD Patriot Inferno RAM: G.SKILL Sniper 2x4GB DDR3 2133 NIC: Intel I350-T2 CASE: Antec ISK 310-150 PS: Lite-On 75W AC Adapter PACKAGES: Cron
NAS: Synology DS415+