The pfSense Store

Author Topic: pfBlockerNG v2.1 w/TLD  (Read 33519 times)

0 Members and 1 Guest are viewing this topic.

Offline Andrew453

  • Full Member
  • ***
  • Posts: 121
  • Karma: +5/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #30 on: July 31, 2016, 05:08:43 am »
I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher.  But it seems itself to contain some second level domains?

Hi Andrew453,

If I only used the TLD, it would be a simple process of looking at any listed Domain and seeing if it had only a second-level Domain (SLD) then block the entire Domain. However, there are suffixes like "uk.com" which is what I would call the TLD that is used to determine if there is one more level. So all of the TLDs (suffixes) in that file are known TLDs which is used in the determination process. Most of the file was taken from the "Public Suffix Registry".


Yes ok.  That's exactly what I thought the file was for.   (i.e. some eTLDs are longer than others, so you need a list e.g. .com vs .co.uk to work out what to treat as an eTLD)

The thing that was confusing me was there were some domains in the list that looked a bit odd, e.g.

myactivedirectory.com
mydrobo.com
mysecuritycamera.com
myshopblocks.com
myvnc.com

I think all you're saying is that pfblockerng will treat those as eTLDs even though, strictly speaking, they aren't ... which is fine.

p.s. a big thank you for implementing this.  It was on my wish-list as I recall - https://forum.pfsense.org/index.php?topic=106534


« Last Edit: July 31, 2016, 05:25:38 am by Andrew453 »

Redyr

  • Guest
Re: pfBlockerNG v2.1 w/TLD
« Reply #31 on: July 31, 2016, 06:32:14 am »
Hi BBcan177,

I can't update h3x feed from available feeds list in pfBlockerNG v2.1.

It show below.

Code: [Select]
[ h3x ] Downloading update .. 200 OK
 Remote timestamp missing
 No Domains Found
Same here

And I can't let TLD Exclusion List working. Can you give a example or check it works?
Did you do a Force Reload after changing the list ?

I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.

I tried with Update, Cron , Reload.

[ malw_corpus ]       Downloading update .. 200 OK
  Remote timestamp missing
 No Domains Found
« Last Edit: July 31, 2016, 06:48:50 am by Redyr »

Offline CiscoX

  • Newbie
  • *
  • Posts: 23
  • Karma: +2/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #32 on: July 31, 2016, 10:12:22 am »
Hi,
Thank you for your hard work on this package :)

After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packets
 
« Last Edit: July 31, 2016, 10:17:32 am by CiscoX »
500/500Mbps Fiber

Offline RonpfS

  • Hero Member
  • *****
  • Posts: 684
  • Karma: +96/-2
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #33 on: July 31, 2016, 12:25:45 pm »
I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.

I tried with Update, Cron , Reload.

[ malw_corpus ]       Downloading update .. 200 OK
  Remote timestamp missing
 No Domains Found

Each URL contains sites that were active in the last period (month, week, day or hour).

If you look at the 1hour or the 1day csv file, they only have one comment. The 1week and 1month have entries.

You should only choose one of the feeds according to your need. I guess most will pick the 1month URL.


2.3.5-RELEASE (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_1/Dev, suricata 4.0.1

Offline lpallard

  • Full Member
  • ***
  • Posts: 274
  • Karma: +3/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #34 on: July 31, 2016, 01:29:25 pm »
Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package.  It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information".  This happens for the "Installed Packages as well as "Available Packages" tabs!

On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".

Tried disabling both pfblockerNG and DNSBL to no avail.  Snort is disabled and the blocked hosts list is empty.

Now I cannot update, install or uninstall packages....  How do I remedy to this?

Offline Andrew453

  • Full Member
  • ***
  • Posts: 121
  • Karma: +5/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #35 on: July 31, 2016, 01:33:50 pm »
I've had that trouble before too.  It happened when I was trying to change from the development thread for updates to the stable thread.  I couldn't update anything.  I eventually found some instructions to reinstall the main pfsense components from the command line.  I ended up still on the development thread and didn't venture to try to change it back after that.

Offline lpallard

  • Full Member
  • ***
  • Posts: 274
  • Karma: +3/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #36 on: July 31, 2016, 01:56:48 pm »
I've had that trouble before too.  It happened when I was trying to change from the development thread for updates to the stable thread.  I couldn't update anything.  I eventually found some instructions to reinstall the main pfsense components from the command line.  I ended up still on the development thread and didn't venture to try to change it back after that.

Not sure I understand that.  I am not playing with development stuff, nor that I am configured to retrieve packages from development repos..  Just a vanilla pfsense install with pfblockerNG, snort and thats it.  Not normal all of a sudden I lose connection to repos..

Also after a reboot I see these warnings in the main page:

Code: [Select]
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #37 on: July 31, 2016, 04:43:34 pm »

Code: [Select]
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00
There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11
There were error(s) loading the rules: /tap/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20

These all seem to be related to the MaxMind IPv6 database. Looks like you will need to bump the pfSense max aliastable entries limit from 2M to 4M. If you enable aggregation in the general tab, it should condense the CIDRs and reduce the overall IP count. This changed due to using the new MaxMind Geolite2 database which seems to have smaller subsets of the data listed causing more IP entries to be added.
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: pfBlockerNG v2.1 w/TLD
« Reply #38 on: July 31, 2016, 04:46:14 pm »
After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packets

I am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline java007md

  • Newbie
  • *
  • Posts: 18
  • Karma: +4/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #39 on: July 31, 2016, 08:28:40 pm »
Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package.  It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information".  This happens for the "Installed Packages as well as "Available Packages" tabs!

On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".

Tried disabling both pfblockerNG and DNSBL to no avail.  Snort is disabled and the blocked hosts list is empty.

Now I cannot update, install or uninstall packages....  How do I remedy to this?

From the following thread:

https://forum.pfsense.org/index.php?topic=116019.0

I followed the ssh command line execution steps:

pkg update -f
pkg upgrade -f

and the same problem was resolved.

Offline CiscoX

  • Newbie
  • *
  • Posts: 23
  • Karma: +2/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #40 on: August 01, 2016, 03:33:49 am »
After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
The DNSBL_EasyList won't delete the packets

I am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.

Hi, No problem. Have a nice Holiday :)
500/500Mbps Fiber

Offline Qinn

  • Full Member
  • ***
  • Posts: 135
  • Karma: +5/-1
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #41 on: August 01, 2016, 03:57:14 am »
Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.

Offline RonpfS

  • Hero Member
  • *****
  • Posts: 684
  • Karma: +96/-2
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #42 on: August 01, 2016, 11:16:15 am »
Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.
Here are the original pfBlockerNG thread https://forum.pfsense.org/index.php?topic=86212.0
and the pfBlockerNG v2.0 w/DNSBL thread https://forum.pfsense.org/index.php?topic=102470
2.3.5-RELEASE (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_1/Dev, suricata 4.0.1

Offline Heimire

  • Jr. Member
  • **
  • Posts: 90
  • Karma: +6/-0
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #43 on: August 01, 2016, 12:20:37 pm »
I am getting this error when I try to use the Spamhaus list in this tread.


===[  DNSBL Process  ]================================================

[ EasywoElements ]    exists.
[ SpamHouse_TLDS ]    Downloading update .. 200 OK
  Remote timestamp missing .
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # Alexa    Final               
  ----------------------------------------------------------------------
  3        3          0          0          0          3                   
  ----------------------------------------------------------------------

[ DNSBL FAIL ] [ Skipping : SpamHouse_TLDS ]

[1470071701] unbound-checkconf[87654:0] error: error parsing local-data at 38 '(xmlhttp.readystate 60 IN A 10.10.10.1': Syntax error, could not parse the RR
[1470071701] unbound-checkconf[87654:0] error: Bad local-data RR (xmlhttp.readystate 60 IN A 10.10.10.1
[1470071701] unbound-checkconf[87654:0] fatal error: failed local-zone, local-data configuration
[ Malware_1month ]    Downloading update [ 08/01/16 12:15:01 ] .. 200 OK
  Remote timestamp missing .
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # Alexa    Final               
  ----------------------------------------------------------------------
  1221     956        0          0          0          956                 
  ----------------------------------------------------------------------

[ Malware_1week ]    Downloading update [ 08/01/16 12:15:04 ] .. 200 OK
  Remote timestamp missing .
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # Alexa    Final               
  ----------------------------------------------------------------------
  526      487        487        0          0          0                   
  ----------------------------------------------------------------------

[ Malware_1day ]    Downloading update [ 08/01/16 12:15:05 ] .. 200 OK
  Remote timestamp missing .
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # Alexa    Final               
  ----------------------------------------------------------------------
  48       47         47         0          0          0                   
  ----------------------------------------------------------------------

[ Malware_1hour ]    Downloading update .. 200 OK
  Remote timestamp missing
 No Domains Found

------------------------------------------
Assembling database... completed
Executing TLD
TLD analysis. completed
Finalizing TLD...  completed
 ----------------------------------------
 Original    Matches    Removed    Final     
 ----------------------------------------
 6062        5530       1          6061     
 -----------------------------------------
Validating database... completed [ 08/01/16 12:15:08 ]
Reloading Unbound.... completed
DNSBL update [ 6061 | PASSED  ]... completed


Offline RonpfS

  • Hero Member
  • *****
  • Posts: 684
  • Karma: +96/-2
    • View Profile
Re: pfBlockerNG v2.1 w/TLD
« Reply #44 on: August 01, 2016, 03:42:05 pm »
Which Spamhaus URL are you using ?
this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.

as for the H3X, only one is needed
https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896

And do a Force Reload after making the modifications.
« Last Edit: August 01, 2016, 03:45:45 pm by RonpfS »
2.3.5-RELEASE (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_1/Dev, suricata 4.0.1