Development/Documentation > Documentation

Guide to filtering web content (http and https) with pfsense 2.3

(1/28) > >>

aGeekHere:
Guide to filtering web content (http and https) with pfsense 2.3 updated 09 June 2017

After seeing a lot of new users asking how to set up web filtering with pfsense I decided to create an extensive guide.

This document is going to be broken down into 3 main parts

1 Host overrides with DNS resolver
2 Squid and squidguard filtering  Transparent vs Non Transparent proxy
3 wpad

Lets begin
Enable DNS resolver
Services/DNS/Resolver/General Settings
Tic enable
Save

Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

UPDATED
Check that the new DNS rule is above the Default allow LAN to any rule in Firewall\Rules\LAN

Now we are going to create some host overrides, the goal for the host overrides is to force google and bing to use there safe search feature.

Click add under Host overrides
Host = www
Domain = bing.com
IP =  204.79.197.220
Description = bing
Save

Now bing is using safe search

Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

Now for google, because google has many different domains it would take a very long time to fill them all in, so we are going to create a short cut.

Ssh into the router
type 8
cd /
cd var/unbound
vi forecegoogle.conf
leave blank for now
save (wq)

Go to Diagnostics/Edit File
click browse
click var
click unbound
now you should see a file called forecegoogle.conf, click it

enter the following


--- Code: ---local-data: "www.google.ad A 216.239.38.120"
local-data: "www.google.ae A 216.239.38.120"
local-data: "www.google.com A 216.239.38.120"
local-data: "www.google.com.af A 216.239.38.120"
local-data: "www.google.com.ag A 216.239.38.120"
local-data: "www.google.com.ai A 216.239.38.120"
local-data: "www.google.al A 216.239.38.120"
local-data: "www.google.am A 216.239.38.120"
local-data: "www.google.co.ao A 216.239.38.120"
local-data: "www.google.com.ar A 216.239.38.120"
local-data: "www.google.as A 216.239.38.120"
local-data: "www.google.at A 216.239.38.120"
local-data: "www.google.com.au A 216.239.38.120"
local-data: "www.google.az A 216.239.38.120"
local-data: "www.google.ba A 216.239.38.120"
local-data: "www.google.com.bd A 216.239.38.120"
local-data: "www.google.be A 216.239.38.120"
local-data: "www.google.bf A 216.239.38.120"
local-data: "www.google.bg A 216.239.38.120"
local-data: "www.google.com.bh A 216.239.38.120"
local-data: "www.google.bi A 216.239.38.120"
local-data: "www.google.bj A 216.239.38.120"
local-data: "www.google.com.bn A 216.239.38.120"
local-data: "www.google.com.bo A 216.239.38.120"
local-data: "www.google.com.br A 216.239.38.120"
local-data: "www.google.bs A 216.239.38.120"
local-data: "www.google.bt A 216.239.38.120"
local-data: "www.google.co.bw A 216.239.38.120"
local-data: "www.google.by A 216.239.38.120"
local-data: "www.google.com.bz A 216.239.38.120"
local-data: "www.google.ca A 216.239.38.120"
local-data: "www.google.cd A 216.239.38.120"
local-data: "www.google.cf A 216.239.38.120"
local-data: "www.google.cg A 216.239.38.120"
local-data: "www.google.ch A 216.239.38.120"
local-data: "www.google.ci A 216.239.38.120"
local-data: "www.google.co.ck A 216.239.38.120"
local-data: "www.google.cl A 216.239.38.120"
local-data: "www.google.cm A 216.239.38.120"
local-data: "www.google.cn A 216.239.38.120"
local-data: "www.google.com.co A 216.239.38.120"
local-data: "www.google.co.cr A 216.239.38.120"
local-data: "www.google.com.cu A 216.239.38.120"
local-data: "www.google.cv A 216.239.38.120"
local-data: "www.google.com.cy A 216.239.38.120"
local-data: "www.google.cz A 216.239.38.120"
local-data: "www.google.de A 216.239.38.120"
local-data: "www.google.dj A 216.239.38.120"
local-data: "www.google.dk A 216.239.38.120"
local-data: "www.google.dm A 216.239.38.120"
local-data: "www.google.com.do A 216.239.38.120"
local-data: "www.google.dz A 216.239.38.120"
local-data: "www.google.com.ec A 216.239.38.120"
local-data: "www.google.ee A 216.239.38.120"
local-data: "www.google.com.eg A 216.239.38.120"
local-data: "www.google.com.et A 216.239.38.120"
local-data: "www.google.fi A 216.239.38.120"
local-data: "www.google.com.fj A 216.239.38.120"
local-data: "www.google.fm A 216.239.38.120"
local-data: "www.google.fr A 216.239.38.120"
local-data: "www.google.ga A 216.239.38.120"
local-data: "www.google.ge A 216.239.38.120"
local-data: "www.google.gg A 216.239.38.120"
local-data: "www.google.com.gh A 216.239.38.120"
local-data: "www.google.com.gi A 216.239.38.120"
local-data: "www.google.gl A 216.239.38.120"
local-data: "www.google.gm A 216.239.38.120"
local-data: "www.google.gp A 216.239.38.120"
local-data: "www.google.gr A 216.239.38.120"
local-data: "www.google.com.gt A 216.239.38.120"
local-data: "www.google.gy A 216.239.38.120"
local-data: "www.google.com.hk A 216.239.38.120"
local-data: "www.google.hn A 216.239.38.120"
local-data: "www.google.hr A 216.239.38.120"
local-data: "www.google.ht A 216.239.38.120"
local-data: "www.google.hu A 216.239.38.120"
local-data: "www.google.co.id A 216.239.38.120"
local-data: "www.google.ie A 216.239.38.120"
local-data: "www.google.co.il A 216.239.38.120"
local-data: "www.google.im A 216.239.38.120"
local-data: "www.google.co.in A 216.239.38.120"
local-data: "www.google.iq A 216.239.38.120"
local-data: "www.google.is A 216.239.38.120"
local-data: "www.google.it A 216.239.38.120"
local-data: "www.google.je A 216.239.38.120"
local-data: "www.google.com.jm A 216.239.38.120"
local-data: "www.google.jo A 216.239.38.120"
local-data: "www.google.co.jp A 216.239.38.120"
local-data: "www.google.co.ke A 216.239.38.120"
local-data: "www.google.com.kh A 216.239.38.120"
local-data: "www.google.ki A 216.239.38.120"
local-data: "www.google.kg A 216.239.38.120"
local-data: "www.google.co.kr A 216.239.38.120"
local-data: "www.google.com.kw A 216.239.38.120"
local-data: "www.google.kz A 216.239.38.120"
local-data: "www.google.la A 216.239.38.120"
local-data: "www.google.com.lb A 216.239.38.120"
local-data: "www.google.li A 216.239.38.120"
local-data: "www.google.lk A 216.239.38.120"
local-data: "www.google.co.ls A 216.239.38.120"
local-data: "www.google.lt A 216.239.38.120"
local-data: "www.google.lu A 216.239.38.120"
local-data: "www.google.lv A 216.239.38.120"
local-data: "www.google.com.ly A 216.239.38.120"
local-data: "www.google.co.ma A 216.239.38.120"
local-data: "www.google.md A 216.239.38.120"
local-data: "www.google.me A 216.239.38.120"
local-data: "www.google.mg A 216.239.38.120"
local-data: "www.google.mk A 216.239.38.120"
local-data: "www.google.ml A 216.239.38.120"
local-data: "www.google.com.mm A 216.239.38.120"
local-data: "www.google.mn A 216.239.38.120"
local-data: "www.google.ms A 216.239.38.120"
local-data: "www.google.com.mt A 216.239.38.120"
local-data: "www.google.mu A 216.239.38.120"
local-data: "www.google.mv A 216.239.38.120"
local-data: "www.google.mw A 216.239.38.120"
local-data: "www.google.com.mx A 216.239.38.120"
local-data: "www.google.com.my A 216.239.38.120"
local-data: "www.google.co.mz A 216.239.38.120"
local-data: "www.google.com.na A 216.239.38.120"
local-data: "www.google.com.nf A 216.239.38.120"
local-data: "www.google.com.ng A 216.239.38.120"
local-data: "www.google.com.ni A 216.239.38.120"
local-data: "www.google.ne A 216.239.38.120"
local-data: "www.google.nl A 216.239.38.120"
local-data: "www.google.no A 216.239.38.120"
local-data: "www.google.com.np A 216.239.38.120"
local-data: "www.google.nr A 216.239.38.120"
local-data: "www.google.nu A 216.239.38.120"
local-data: "www.google.co.nz A 216.239.38.120"
local-data: "www.google.com.om A 216.239.38.120"
local-data: "www.google.com.pa A 216.239.38.120"
local-data: "www.google.com.pe A 216.239.38.120"
local-data: "www.google.com.pg A 216.239.38.120"
local-data: "www.google.com.ph A 216.239.38.120"
local-data: "www.google.com.pk A 216.239.38.120"
local-data: "www.google.pl A 216.239.38.120"
local-data: "www.google.pn A 216.239.38.120"
local-data: "www.google.com.pr A 216.239.38.120"
local-data: "www.google.ps A 216.239.38.120"
local-data: "www.google.pt A 216.239.38.120"
local-data: "www.google.com.py A 216.239.38.120"
local-data: "www.google.com.qa A 216.239.38.120"
local-data: "www.google.ro A 216.239.38.120"
local-data: "www.google.ru A 216.239.38.120"
local-data: "www.google.rw A 216.239.38.120"
local-data: "www.google.com.sa A 216.239.38.120"
local-data: "www.google.com.sb A 216.239.38.120"
local-data: "www.google.sc A 216.239.38.120"
local-data: "www.google.se A 216.239.38.120"
local-data: "www.google.com.sg A 216.239.38.120"
local-data: "www.google.sh A 216.239.38.120"
local-data: "www.google.si A 216.239.38.120"
local-data: "www.google.sk A 216.239.38.120"
local-data: "www.google.com.sl A 216.239.38.120"
local-data: "www.google.sn A 216.239.38.120"
local-data: "www.google.so A 216.239.38.120"
local-data: "www.google.sm A 216.239.38.120"
local-data: "www.google.sr A 216.239.38.120"
local-data: "www.google.st A 216.239.38.120"
local-data: "www.google.com.sv A 216.239.38.120"
local-data: "www.google.td A 216.239.38.120"
local-data: "www.google.tg A 216.239.38.120"
local-data: "www.google.co.th A 216.239.38.120"
local-data: "www.google.com.tj A 216.239.38.120"
local-data: "www.google.tk A 216.239.38.120"
local-data: "www.google.tl A 216.239.38.120"
local-data: "www.google.tm A 216.239.38.120"
local-data: "www.google.tn A 216.239.38.120"
local-data: "www.google.to A 216.239.38.120"
local-data: "www.google.com.tr A 216.239.38.120"
local-data: "www.google.tt A 216.239.38.120"
local-data: "www.google.com.tw A 216.239.38.120"
local-data: "www.google.co.tz A 216.239.38.120"
local-data: "www.google.com.ua A 216.239.38.120"
local-data: "www.google.co.ug A 216.239.38.120"
local-data: "www.google.co.uk A 216.239.38.120"
local-data: "www.google.com.uy A 216.239.38.120"
local-data: "www.google.co.uz A 216.239.38.120"
local-data: "www.google.com.vc A 216.239.38.120"
local-data: "www.google.co.ve A 216.239.38.120"
local-data: "www.google.vg A 216.239.38.120"
local-data: "www.google.co.vi A 216.239.38.120"
local-data: "www.google.com.vn A 216.239.38.120"
local-data: "www.google.vu A 216.239.38.120"
local-data: "www.google.ws A 216.239.38.120"
local-data: "www.google.rs A 216.239.38.120"
local-data: "www.google.co.za A 216.239.38.120"
local-data: "www.google.co.zm A 216.239.38.120"
local-data: "www.google.co.zw A 216.239.38.120"
local-data: "www.google.cat A 216.239.38.120"

--- End code ---
save

Go to Services/DNS/Resolver/General Settings
in custom option enter


--- Code: ---server:
include: /var/unbound/forecegoogle.conf
--- End code ---

save
now google should be using safe mode.

Part 2
Install squid and squidguard in System/PackageManager/Available Packages

Now we are going to talk about transparent proxy vs non transparent proxy.
https://doc.pfsense.org/index.php/Setup_Squid_as_a_Transparent_Proxy

Transparent proxy for http is very easy to set up, you just enable Transparent HTTP Proxy in squid (and install the blacklist in squidguard but I will get to that later). Now all traffic should be going to your proxy server on port 3128. However, if you want to filter https then this is where it gets complicated, you have to enable SSL Man In the Middle Filtering and create Certificates and even after that you may get connection errors and all sorts of issues.

UPDATE
You can try setting up MITM by setting the SSL/MITM Mode to splice all, that way you do not need to create a certificate for each device on the network. (you still need to create a main certificate though)

So in this guide we are going to use a Non Transparent with wpad which will filter http and https content.
Update
I found that we can use both a transperrent proxy for port 80 and a wpad for 443 https content (UPDATE or you can use splice all in MITM), the wpad will be setup to use port 80 and 443. The transperrent proxy is going to catch every thing that the wpad misses, enable transperrent proxy in squid once you have the wpad setup.



First we are going to setup squidguard
Update
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).

In Package/Proxy filter SquidGuard: General settings/General settings
click blacklist
enter http://www.shallalist.de/Downloads/shallalist.tar.gz
download
wait to finish

Now we are going to create a new target category.
click Target categories (Do not skip this step).
This will be a white list.
add
name whitelist
description whitelist

Because google and bing are the only search engines (as of writing) that can force safes search we are going to block all other search engines except google and bing, white list google and bing
Domain list

NOTE NOT ALL ADDED YET FOR GOOGLE
Trying to fix google domains like play.google.com accounts.google.com mail.google.com and sites like www.google.com/contacts from getting blocked Fixed


--- Code: ---google.ac google.ad google.ae google.al google.am google.as google.at google.az google.ba google.be google.bf google.bg google.bi google.bj google.bs google.bt google.by google.ca google.cat google.cd google.cf google.cg google.ch google.ci google.cl google.cm google.cn google.co.ao google.co.bw google.co.ck google.co.cr google.co.hu google.co.id google.co.il google.co.in google.co.je google.co.jp google.co.ke google.co.kr google.co.ls google.com google.co.ma google.com.af google.com.ag google.com.ai google.com.ar google.com.au google.com.bd google.com.bh google.com.bn google.com.bo google.com.br google.com.bz google.com.co google.com.cu google.com.cy google.com.do google.com.ec google.com.eg google.com.et google.com.fj google.com.gh google.com.gi google.com.gr google.com.gt google.com.hk google.com.jm google.com.kh google.com.kw google.com.lb google.com.ly google.com.mm google.com.mt google.com.mx google.com.my google.com.na google.com.nf google.com.ng google.com.ni google.com.np google.com.om google.com.pa google.com.pe google.com.pg google.com.ph google.com.pk google.com.pr google.com.py google.com.qa google.com.sa google.com.sb google.com.sg google.com.sl google.com.sv google.com.tj google.com.tr google.com.tw google.com.ua google.com.uy google.com.vc google.com.vn google.co.mz google.co.nz google.co.th google.co.tz google.co.ug google.co.uk google.co.uz google.co.ve google.co.vi google.co.za google.co.zm google.co.zw google.cv google.cz google.de google-directory.co.uk google.dj google.dk google.dm google.dz google.ee google.es google.fi google.fm google.fr google.ga google.ge google.gg google.gl google.gm google.gp google.gr google.gy google.hn google.hr google.ht google.hu google.ie google.im google.iq google.is google.it google.je google.jo google.kg google.ki google.kz google.la google.li google.lk google.lt google.lu google.lv google.md google.me google.mg google.mk google.ml google.mn google.ms google.mu google.mv google.mw google.ne google.nl google.no google.nr google.nu google.off.ai googlepirate.com google.pl google.pn google.ps google.pt google.ro google.rs google.ru google.rw google.sc google.se google.sh google.si google.sk google.sm google.sn google.so google.sr google.st google.td google.tg google.tk google.tl google.tm google.tn google.to google.tt google.uz google.vg google.vu google.ws bing.com

--- End code ---

save

click Common ACL
click the plus button
target categories whitelist access whitelist
[blk_BL_searchengines] access deny
Default access [all] allow

To block ads (including on android and ios)
[blk_BL_adv] access deny

To block proxy sites
[blk_BL_anonvpn] access deny
Read though all the other categories and deny the ones you want

next click Do not allow IP-Addresses in URL (If this causes issues deselect it)
use safe search engines no longer works however you can click it as well.
Save
click General settings
click Apply
click Save

If you want you can do a quick test by setting up your pc to use the proxy and see how thing are working.

Part 3
Now we are going to set up a wpad read more here about wpad https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid
ssh in to pfsense
8
cd /
create the wpad.da file
vi /usr/local/www/wpad.da
wq

Create two new symbolic link files

--- Code: ---ln -s /usr/local/www/wpad.da /usr/local/www/wpad.dat
ln -s /usr/local/www/wpad.da /usr/local/www/proxy.pac
--- End code ---


Then go Diagnostics /Edit File
click browse
user
local
www
click wpad.da
add


--- Code: ---function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.1.1:3128";
}
--- End code ---

save


If you connect to a VPN you need to go direct for the VPN instead of the proxy, Remember you need to add the correct network class for the VPN  either A, B or C


--- Code: ---function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

        if (isInNet(dnsResolve(host), "1.0.0.0",  "255.0.0.0" ))
        { return "DIRECT"; }
 
    return "PROXY 192.168.1.1:3128";
}
--- End code ---

save
Go to Configure DNS Resolver add new host overrides
Host: wpad
Domain: mylocaldomain.local
IP Address: 192.168.1.1
Description: WPAD Autoconfigure Host
save
Next go to Services: DHCP server under Additional BOOTP/DHCP Options
add

--- Code: ---number: 252 type: string value: "http://192.168.1.1/wpad.dat"
number: 252 type: string value: "http://192.168.1.1/wpad.da"
number: 252 type: string value: "http://192.168.1.1/proxy.pac"
--- End code ---
save

set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save

Set your system to automatically detect settings (for windows it is in internet options connections lan settings).

You also have to set up the proxy setting for each program that cant connect (firefox, graphics drive software, vlc etc)

If you have programs that cannot connect and have no proxy setting you need to setup a firewall aliases
 Firewall/Aliases/IP
and add the destination server ip (use wire shark to help find the blocked Ips or in your firewall block rule enable Log packets that are handled by this rule, use http://ip-lookup.net/index.php to check what it is and add to the Aliases. If it is part of a domain add the domain)
now create a new firewall lan rule
IPv4 TCP * * * passAliases 80- 443 * pass rule.

Save

A note on smart phones (android, IOS, etc)
With android (not sure on other smart phones OS) you can not set it so that all the apps on the device use the proxy (not without rooting and other hacks), web browsers (google) will work fine using the proxy (if set in wireless connection options) but not apps or things like google play, so unless there is an option to use proxy for all apps on the device the most practical option here is just to allow smart phones to use port 80 and 443.
 
UPDATE 24 JUNE 2016
I have found that if you have connection issues using auto config for android or other smart phones try manually setting the proxy, now opening port 80 and 443 is not needed.

Now we should have pfsense all set up for web filtering. I hope this has been helpful and thanks to everyone on the forum who has help me in creating this guide.

Just a note for any specific issues with squid, squidguard or dns please create a new topic in the correct areas of the forum and link it here if needed

stilez:
One step you missed:


--- Quote ---make a symbolic link between the file
Then go Diagnostics /Edit File
--- End quote ---

You didn't say which file (and which target) to make the symbolic link, or the command you use for it. Some people might need to know.

aGeekHere:
Thanks, will update it soon.

Done

stilez:
Is "click wpad.da" and other "wpad.da" a typo?
If it's correct, it might be worth commenting after it that you do mean "da" not "dat", because having two files called wpad.da and wpad.dat might not be noticed, looks like a typo, is confusing, etc :)

aGeekHere:
Hi, there are 3 wpad files
wpad.dat
wpad.da
proxy.pac
https://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol

I made the wpad.da the main file you edit and made a symbolic link for wpad.dat and proxy.pac (so all you need to do is just edit the wpad.da file).
If this is still confusing me know and I will update the guide.

Navigation

[0] Message Index

[#] Next page

Go to full version