The pfSense Store

Author Topic: DNS Blacklist, New Package! Check it out.  (Read 54704 times)

0 Members and 1 Guest are viewing this topic.

Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
DNS Blacklist, New Package! Check it out.
« on: October 04, 2009, 07:35:57 am »
Okay, so yesterday we released DNS Blacklist which is now added into the Packages list for pfSense.

The idea of the package is to use a freely available url/host/ip blacklist database to filter out sites you do not want visited from your network.  Using every category listed in the DNS Blacklist will require roughly 256mb free memory on its own.  This works by using dnsmasq and adding the categories you select into the dnsmasq config to reject the unwanted host and forward to Google.  This is somewhat similar to OpenDNS in a sense, and does work really well.

I started this project a little over a month ago and with the help of mcrane I was able to get it finished and submitted.  Right now you're limited to the hosts within the database but if the project continues on I will set it to allow custom entries from the DNS Blacklist control panel, and the option to edit the categories yourself.

Any and all feedback is welcome, or you can find me in ##pfsense or ##pfsense-dnsblacklist or just message me on freenode.   And a big thanks to mcrane for his help, and putting up with me :P


---

Edit:  I do realize that since combining the porn/adult categories I forgot to edit the text on the control panel.  Also you are not required to restart dnsmasq manually like it says in the "savemsg"... That was fixed by adding services_dnsmasq_configure() into the source.  And one last thing I can think of... You do not need to hit "APPLY" in the savemsg dialog to make the changes stick.  I'll get all of this taken care of later today.   Just in a hurry to get the package released we didn't get the control panel cleaned up properly.
« Last Edit: October 04, 2009, 08:00:42 am by xa0z »

Offline mikesamo

  • Full Member
  • ***
  • Posts: 225
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #1 on: October 04, 2009, 08:20:15 am »
thank you for the package I will check this =)

Online Supermule

  • Hero Member
  • *****
  • Posts: 1494
  • Karma: +2/-1
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #2 on: October 04, 2009, 09:45:12 am »
Cannot be disabled and uninstalled in webgui....
Kind regards Brian


Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #3 on: October 04, 2009, 11:39:45 am »
Cannot be disabled and uninstalled in webgui....

I noticed this earlier after the package was submitted.  Not sure what happened between the version I submitted and the version being supplied but it does seem to have an issue here and there that I'll take care of later today.  For a temporary fix, you can disable "DNS Blacklist", but you must leave a category selected while the "Enabled" checkbox is deselected.

I don't have access to the "DNS Blacklist" repo on pfSense to fix things so I have to wait for mcrane to come online, then submit the changes to him and then he will submit them to pfSense.  

Online Supermule

  • Hero Member
  • *****
  • Posts: 1494
  • Karma: +2/-1
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #4 on: October 04, 2009, 11:50:59 am »
Nice....Not superurgent, but a small feedback to you.

:)

I suppose one can update the package via an uninstall....
Kind regards Brian


Offline jigpe

  • Sr. Member
  • ****
  • Posts: 371
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #5 on: October 04, 2009, 09:32:29 pm »
Thank you xa0z! Great job :)

Thanks for the heads up!

jigp

Offline keeper

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #6 on: October 05, 2009, 06:20:25 am »
thanks, more power to your team guys   :)

Offline vreid473

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #7 on: October 05, 2009, 10:52:00 am »
Hello,

Thanks for implementing this package.  I'm trying it out , and I think it has a nice feature set and is super easy to configure.  I have a few suggestions for additions:

1.  The ability to add or remove domains and url's to a whitelist and to a blacklist.
2.  The ability to specify the web page url to which users are redirected for each category.
3.  The ability to specify the blacklist to use.  Or, alternatively, some documentation about what's on the existing blocklist.

I noticed that the dns blacklist package mentions porn in the notes at the top of the page, but there appears to be not porn category (at least on the embedded machine on which I'm doing my initial testing).  Is this an error or a feature to reduce the footprint on embedded devices?

I appreciate the time and effort that you have placed into creating this package.  I think it definitely fits the need for an additional filtering plugin for PFSense.  :)

Vaughn Reid III

Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #8 on: October 05, 2009, 11:55:24 am »
 
  Thanks for the comments.   The next version of this script does allow users to add their own custom blacklist entries.  As of right now there is no way to specify the URL that a blacklisted item is sent to, that is why it is forwarding to a Google IP.  Doing the method you are requesting requires a proxy.  This is being looked into for the future.

As for being able to select your own blacklist database, I have already thought of this and think it would be a great idea but right now the scripting is static ...it was static, it is dynamic now... to use preset categories/directories for the database.  In the near future I would like to have this set to load the categories from a variable after reading the available directories containing the blacklists.

In a beta version of DNS Blacklist we had two separate categories... "Porn" and "Adult".  That blacklist database is not what we're using right now so the categories are different and now contained under the same category.   The configuration page still lists the two categories in the NOTE because I forgot to remove it, just a self error.

We're going to release a "fixed" version here soon, but a major release with additions will be a little later.
« Last Edit: October 05, 2009, 12:09:47 pm by xa0z »

Offline mcrane

  • Sr. Member
  • ****
  • Posts: 495
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #9 on: October 05, 2009, 06:23:54 pm »
Did some improvements to the package today.
1. wording changes.
2. uninstall has been fixed
3. all categories can be deselected.

You can edit categories by using pfSense's command page:
Diagnostics -> Command

ls /usr/local/www/packages/dnsblacklist/blacklists

Lets say you wanted to edit the 'adult' list, you can download the list using:
File to download -> /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

You would then edit the file using an editor that deals well with large files.
If you are on windows don't use the standard notepad.exe instead use pspad or notepad++

To upload the file from the command page use:
File to upload

The file will be uploaded to the /tmp directory.

Then use 'Execute Shell command' to cp the file to the correct directory.
cp /tmp/domains /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

If you follow the folder structure consistently you can add additional categories.
You also would need to carefully edit the /usr/local/www/packages/dnsblacklist/blacklists/global_usage file which list the categories and provides the description for the categories. You can edit the global_usage file using Diagnostics -> Edit File



Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #10 on: October 05, 2009, 07:29:03 pm »
Tomorrow I will begin working on our own Blacklist Database.  It will work in the same manor as current...  We will have a blacklist directory, and within that directory we will have a directory with the respective category name, and then within that directory we have the "domains" file which contains the domains that will be added to the active blacklist when that category is selected.

Today mcrane did a great job in updating the code to make sure everything was working as normal as we can.  Remember we're still in BETA so give it some time, make your requests for addons, report any bugs you might see and we will do our best to make sure things go as smooth as possible.

As for right now the only concern I have is making sure we can get a really good, clean, host-database for the blacklisting.

Offline mcrane

  • Sr. Member
  • ****
  • Posts: 495
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #11 on: October 05, 2009, 07:37:56 pm »
The current list comes from here:
http://cri.univ-tlse1.fr/blacklists/index_en.php

Its just not as large as xa0z would like.

Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #12 on: October 05, 2009, 08:35:19 pm »
The size isn't what bothers me... I think that a proper database should contain categories that are more limited to specific matching items, and better defined categories at that.

That was the whole point of me starting this project.

Offline mcrane

  • Sr. Member
  • ****
  • Posts: 495
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #13 on: October 06, 2009, 12:13:25 am »
Here is the original blacklist that was going to be used.
http://urlblacklist.com/

I didn't use it in the final version of the package because I didn't feel that redistribution of that list was the ethical thing to do when the website that provides the list is trying to sell it. They provide the entire list for download and payment is on an honor basis as stated on their website. Their intent is really to sell it for a subscription. I did not notice this when I started building this package. When I noticed they were trying to sell the list they had compiled I could not with good conscious redistribute urlblacklist.com's list. If someone wants to use that list and pay for the subscription they can do so by editing the domain lists and adding the categories as I described in an earlier post.
http://urlblacklist.com/?sec=subscribe

Here is a few more choices of DNS blacklists.
http://www.squidguard.org/blacklists.html

Anyone can edit the list that the package comes with. In order for an alternative customized list to be used in the DNS blacklist package it must be compiled ethically and legally or I will not commit it to the package. For example it wouldn't be ethical to use lists from urlblacklist.com or other lists that don't offer the list in freely. Unless you obtain permission directly from them to use the list for such a purpose.

The list from http://cri.univ-tlse1.fr/blacklists/index_en.php says on their website 'can be used with many commercial or free software' therefore it is ethical to use their list for the DNS Blacklist package.
« Last Edit: October 06, 2009, 02:34:47 am by mcrane »

Offline xa0z

  • Jr. Member
  • **
  • Posts: 30
  • Karma: +0/-0
    • View Profile
Re: DNS Blacklist, New Package! Check it out.
« Reply #14 on: October 06, 2009, 09:19:52 am »
Hmm, so I guess there isn't a need for me to continue on with the project.  It is just pointless for me to try and do one thing and then have it all changed to not be the way it was when I started it.

I already said I was going to build my own database list but what's the point, I don't need to supply something that anyone can just grab from all over the net to use.

I'm glad for the help I got when I started writing the project but if I am being phased out, which it so seems I am, then I guess my input, or work is useless.