pfSense Forum

Development/Documentation => Documentation => Topic started by: aGeekHere on May 23, 2016, 12:06:54 am

Title: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on May 23, 2016, 12:06:54 am
Guide to filtering web content (http and https) with pfsense 2.3 updated 09 June 2017

After seeing a lot of new users asking how to set up web filtering with pfsense I decided to create an extensive guide.

This document is going to be broken down into 3 main parts

1 Host overrides with DNS resolver
2 Squid and squidguard filtering  Transparent vs Non Transparent proxy
3 wpad

Lets begin
Enable DNS resolver
Services/DNS/Resolver/General Settings
Tic enable
Save

Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

UPDATED
Check that the new DNS rule is above the Default allow LAN to any rule in Firewall\Rules\LAN

Now we are going to create some host overrides, the goal for the host overrides is to force google and bing to use there safe search feature.

Click add under Host overrides
Host = www
Domain = bing.com
IP =  204.79.197.220
Description = bing
Save

Now bing is using safe search

Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

Now for google, because google has many different domains it would take a very long time to fill them all in, so we are going to create a short cut.

Ssh into the router
type 8
cd /
cd var/unbound
vi forecegoogle.conf
leave blank for now
save (wq)

Go to Diagnostics/Edit File
click browse
click var
click unbound
now you should see a file called forecegoogle.conf, click it

enter the following

Code: [Select]
local-data: "www.google.ad A 216.239.38.120"
local-data: "www.google.ae A 216.239.38.120"
local-data: "www.google.com A 216.239.38.120"
local-data: "www.google.com.af A 216.239.38.120"
local-data: "www.google.com.ag A 216.239.38.120"
local-data: "www.google.com.ai A 216.239.38.120"
local-data: "www.google.al A 216.239.38.120"
local-data: "www.google.am A 216.239.38.120"
local-data: "www.google.co.ao A 216.239.38.120"
local-data: "www.google.com.ar A 216.239.38.120"
local-data: "www.google.as A 216.239.38.120"
local-data: "www.google.at A 216.239.38.120"
local-data: "www.google.com.au A 216.239.38.120"
local-data: "www.google.az A 216.239.38.120"
local-data: "www.google.ba A 216.239.38.120"
local-data: "www.google.com.bd A 216.239.38.120"
local-data: "www.google.be A 216.239.38.120"
local-data: "www.google.bf A 216.239.38.120"
local-data: "www.google.bg A 216.239.38.120"
local-data: "www.google.com.bh A 216.239.38.120"
local-data: "www.google.bi A 216.239.38.120"
local-data: "www.google.bj A 216.239.38.120"
local-data: "www.google.com.bn A 216.239.38.120"
local-data: "www.google.com.bo A 216.239.38.120"
local-data: "www.google.com.br A 216.239.38.120"
local-data: "www.google.bs A 216.239.38.120"
local-data: "www.google.bt A 216.239.38.120"
local-data: "www.google.co.bw A 216.239.38.120"
local-data: "www.google.by A 216.239.38.120"
local-data: "www.google.com.bz A 216.239.38.120"
local-data: "www.google.ca A 216.239.38.120"
local-data: "www.google.cd A 216.239.38.120"
local-data: "www.google.cf A 216.239.38.120"
local-data: "www.google.cg A 216.239.38.120"
local-data: "www.google.ch A 216.239.38.120"
local-data: "www.google.ci A 216.239.38.120"
local-data: "www.google.co.ck A 216.239.38.120"
local-data: "www.google.cl A 216.239.38.120"
local-data: "www.google.cm A 216.239.38.120"
local-data: "www.google.cn A 216.239.38.120"
local-data: "www.google.com.co A 216.239.38.120"
local-data: "www.google.co.cr A 216.239.38.120"
local-data: "www.google.com.cu A 216.239.38.120"
local-data: "www.google.cv A 216.239.38.120"
local-data: "www.google.com.cy A 216.239.38.120"
local-data: "www.google.cz A 216.239.38.120"
local-data: "www.google.de A 216.239.38.120"
local-data: "www.google.dj A 216.239.38.120"
local-data: "www.google.dk A 216.239.38.120"
local-data: "www.google.dm A 216.239.38.120"
local-data: "www.google.com.do A 216.239.38.120"
local-data: "www.google.dz A 216.239.38.120"
local-data: "www.google.com.ec A 216.239.38.120"
local-data: "www.google.ee A 216.239.38.120"
local-data: "www.google.com.eg A 216.239.38.120"
local-data: "www.google.com.et A 216.239.38.120"
local-data: "www.google.fi A 216.239.38.120"
local-data: "www.google.com.fj A 216.239.38.120"
local-data: "www.google.fm A 216.239.38.120"
local-data: "www.google.fr A 216.239.38.120"
local-data: "www.google.ga A 216.239.38.120"
local-data: "www.google.ge A 216.239.38.120"
local-data: "www.google.gg A 216.239.38.120"
local-data: "www.google.com.gh A 216.239.38.120"
local-data: "www.google.com.gi A 216.239.38.120"
local-data: "www.google.gl A 216.239.38.120"
local-data: "www.google.gm A 216.239.38.120"
local-data: "www.google.gp A 216.239.38.120"
local-data: "www.google.gr A 216.239.38.120"
local-data: "www.google.com.gt A 216.239.38.120"
local-data: "www.google.gy A 216.239.38.120"
local-data: "www.google.com.hk A 216.239.38.120"
local-data: "www.google.hn A 216.239.38.120"
local-data: "www.google.hr A 216.239.38.120"
local-data: "www.google.ht A 216.239.38.120"
local-data: "www.google.hu A 216.239.38.120"
local-data: "www.google.co.id A 216.239.38.120"
local-data: "www.google.ie A 216.239.38.120"
local-data: "www.google.co.il A 216.239.38.120"
local-data: "www.google.im A 216.239.38.120"
local-data: "www.google.co.in A 216.239.38.120"
local-data: "www.google.iq A 216.239.38.120"
local-data: "www.google.is A 216.239.38.120"
local-data: "www.google.it A 216.239.38.120"
local-data: "www.google.je A 216.239.38.120"
local-data: "www.google.com.jm A 216.239.38.120"
local-data: "www.google.jo A 216.239.38.120"
local-data: "www.google.co.jp A 216.239.38.120"
local-data: "www.google.co.ke A 216.239.38.120"
local-data: "www.google.com.kh A 216.239.38.120"
local-data: "www.google.ki A 216.239.38.120"
local-data: "www.google.kg A 216.239.38.120"
local-data: "www.google.co.kr A 216.239.38.120"
local-data: "www.google.com.kw A 216.239.38.120"
local-data: "www.google.kz A 216.239.38.120"
local-data: "www.google.la A 216.239.38.120"
local-data: "www.google.com.lb A 216.239.38.120"
local-data: "www.google.li A 216.239.38.120"
local-data: "www.google.lk A 216.239.38.120"
local-data: "www.google.co.ls A 216.239.38.120"
local-data: "www.google.lt A 216.239.38.120"
local-data: "www.google.lu A 216.239.38.120"
local-data: "www.google.lv A 216.239.38.120"
local-data: "www.google.com.ly A 216.239.38.120"
local-data: "www.google.co.ma A 216.239.38.120"
local-data: "www.google.md A 216.239.38.120"
local-data: "www.google.me A 216.239.38.120"
local-data: "www.google.mg A 216.239.38.120"
local-data: "www.google.mk A 216.239.38.120"
local-data: "www.google.ml A 216.239.38.120"
local-data: "www.google.com.mm A 216.239.38.120"
local-data: "www.google.mn A 216.239.38.120"
local-data: "www.google.ms A 216.239.38.120"
local-data: "www.google.com.mt A 216.239.38.120"
local-data: "www.google.mu A 216.239.38.120"
local-data: "www.google.mv A 216.239.38.120"
local-data: "www.google.mw A 216.239.38.120"
local-data: "www.google.com.mx A 216.239.38.120"
local-data: "www.google.com.my A 216.239.38.120"
local-data: "www.google.co.mz A 216.239.38.120"
local-data: "www.google.com.na A 216.239.38.120"
local-data: "www.google.com.nf A 216.239.38.120"
local-data: "www.google.com.ng A 216.239.38.120"
local-data: "www.google.com.ni A 216.239.38.120"
local-data: "www.google.ne A 216.239.38.120"
local-data: "www.google.nl A 216.239.38.120"
local-data: "www.google.no A 216.239.38.120"
local-data: "www.google.com.np A 216.239.38.120"
local-data: "www.google.nr A 216.239.38.120"
local-data: "www.google.nu A 216.239.38.120"
local-data: "www.google.co.nz A 216.239.38.120"
local-data: "www.google.com.om A 216.239.38.120"
local-data: "www.google.com.pa A 216.239.38.120"
local-data: "www.google.com.pe A 216.239.38.120"
local-data: "www.google.com.pg A 216.239.38.120"
local-data: "www.google.com.ph A 216.239.38.120"
local-data: "www.google.com.pk A 216.239.38.120"
local-data: "www.google.pl A 216.239.38.120"
local-data: "www.google.pn A 216.239.38.120"
local-data: "www.google.com.pr A 216.239.38.120"
local-data: "www.google.ps A 216.239.38.120"
local-data: "www.google.pt A 216.239.38.120"
local-data: "www.google.com.py A 216.239.38.120"
local-data: "www.google.com.qa A 216.239.38.120"
local-data: "www.google.ro A 216.239.38.120"
local-data: "www.google.ru A 216.239.38.120"
local-data: "www.google.rw A 216.239.38.120"
local-data: "www.google.com.sa A 216.239.38.120"
local-data: "www.google.com.sb A 216.239.38.120"
local-data: "www.google.sc A 216.239.38.120"
local-data: "www.google.se A 216.239.38.120"
local-data: "www.google.com.sg A 216.239.38.120"
local-data: "www.google.sh A 216.239.38.120"
local-data: "www.google.si A 216.239.38.120"
local-data: "www.google.sk A 216.239.38.120"
local-data: "www.google.com.sl A 216.239.38.120"
local-data: "www.google.sn A 216.239.38.120"
local-data: "www.google.so A 216.239.38.120"
local-data: "www.google.sm A 216.239.38.120"
local-data: "www.google.sr A 216.239.38.120"
local-data: "www.google.st A 216.239.38.120"
local-data: "www.google.com.sv A 216.239.38.120"
local-data: "www.google.td A 216.239.38.120"
local-data: "www.google.tg A 216.239.38.120"
local-data: "www.google.co.th A 216.239.38.120"
local-data: "www.google.com.tj A 216.239.38.120"
local-data: "www.google.tk A 216.239.38.120"
local-data: "www.google.tl A 216.239.38.120"
local-data: "www.google.tm A 216.239.38.120"
local-data: "www.google.tn A 216.239.38.120"
local-data: "www.google.to A 216.239.38.120"
local-data: "www.google.com.tr A 216.239.38.120"
local-data: "www.google.tt A 216.239.38.120"
local-data: "www.google.com.tw A 216.239.38.120"
local-data: "www.google.co.tz A 216.239.38.120"
local-data: "www.google.com.ua A 216.239.38.120"
local-data: "www.google.co.ug A 216.239.38.120"
local-data: "www.google.co.uk A 216.239.38.120"
local-data: "www.google.com.uy A 216.239.38.120"
local-data: "www.google.co.uz A 216.239.38.120"
local-data: "www.google.com.vc A 216.239.38.120"
local-data: "www.google.co.ve A 216.239.38.120"
local-data: "www.google.vg A 216.239.38.120"
local-data: "www.google.co.vi A 216.239.38.120"
local-data: "www.google.com.vn A 216.239.38.120"
local-data: "www.google.vu A 216.239.38.120"
local-data: "www.google.ws A 216.239.38.120"
local-data: "www.google.rs A 216.239.38.120"
local-data: "www.google.co.za A 216.239.38.120"
local-data: "www.google.co.zm A 216.239.38.120"
local-data: "www.google.co.zw A 216.239.38.120"
local-data: "www.google.cat A 216.239.38.120"
save

Go to Services/DNS/Resolver/General Settings
in custom option enter

Code: [Select]
server:
include: /var/unbound/forecegoogle.conf

save
now google should be using safe mode.

Part 2
Install squid and squidguard in System/PackageManager/Available Packages

Now we are going to talk about transparent proxy vs non transparent proxy.
https://doc.pfsense.org/index.php/Setup_Squid_as_a_Transparent_Proxy

Transparent proxy for http is very easy to set up, you just enable Transparent HTTP Proxy in squid (and install the blacklist in squidguard but I will get to that later). Now all traffic should be going to your proxy server on port 3128. However, if you want to filter https then this is where it gets complicated, you have to enable SSL Man In the Middle Filtering and create Certificates and even after that you may get connection errors and all sorts of issues.

UPDATE
You can try setting up MITM by setting the SSL/MITM Mode to splice all, that way you do not need to create a certificate for each device on the network. (you still need to create a main certificate though)

So in this guide we are going to use a Non Transparent with wpad which will filter http and https content.
Update
I found that we can use both a transperrent proxy for port 80 and a wpad for 443 https content (UPDATE or you can use splice all in MITM), the wpad will be setup to use port 80 and 443. The transperrent proxy is going to catch every thing that the wpad misses, enable transperrent proxy in squid once you have the wpad setup.



First we are going to setup squidguard
Update
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).

In Package/Proxy filter SquidGuard: General settings/General settings
click blacklist
enter http://www.shallalist.de/Downloads/shallalist.tar.gz
download
wait to finish

Now we are going to create a new target category.
click Target categories (Do not skip this step).
This will be a white list.
add
name whitelist
description whitelist

Because google and bing are the only search engines (as of writing) that can force safes search we are going to block all other search engines except google and bing, white list google and bing
Domain list

NOTE NOT ALL ADDED YET FOR GOOGLE
Trying to fix google domains like play.google.com accounts.google.com mail.google.com and sites like www.google.com/contacts from getting blocked
Fixed

Code: [Select]
google.ac google.ad google.ae google.al google.am google.as google.at google.az google.ba google.be google.bf google.bg google.bi google.bj google.bs google.bt google.by google.ca google.cat google.cd google.cf google.cg google.ch google.ci google.cl google.cm google.cn google.co.ao google.co.bw google.co.ck google.co.cr google.co.hu google.co.id google.co.il google.co.in google.co.je google.co.jp google.co.ke google.co.kr google.co.ls google.com google.co.ma google.com.af google.com.ag google.com.ai google.com.ar google.com.au google.com.bd google.com.bh google.com.bn google.com.bo google.com.br google.com.bz google.com.co google.com.cu google.com.cy google.com.do google.com.ec google.com.eg google.com.et google.com.fj google.com.gh google.com.gi google.com.gr google.com.gt google.com.hk google.com.jm google.com.kh google.com.kw google.com.lb google.com.ly google.com.mm google.com.mt google.com.mx google.com.my google.com.na google.com.nf google.com.ng google.com.ni google.com.np google.com.om google.com.pa google.com.pe google.com.pg google.com.ph google.com.pk google.com.pr google.com.py google.com.qa google.com.sa google.com.sb google.com.sg google.com.sl google.com.sv google.com.tj google.com.tr google.com.tw google.com.ua google.com.uy google.com.vc google.com.vn google.co.mz google.co.nz google.co.th google.co.tz google.co.ug google.co.uk google.co.uz google.co.ve google.co.vi google.co.za google.co.zm google.co.zw google.cv google.cz google.de google-directory.co.uk google.dj google.dk google.dm google.dz google.ee google.es google.fi google.fm google.fr google.ga google.ge google.gg google.gl google.gm google.gp google.gr google.gy google.hn google.hr google.ht google.hu google.ie google.im google.iq google.is google.it google.je google.jo google.kg google.ki google.kz google.la google.li google.lk google.lt google.lu google.lv google.md google.me google.mg google.mk google.ml google.mn google.ms google.mu google.mv google.mw google.ne google.nl google.no google.nr google.nu google.off.ai googlepirate.com google.pl google.pn google.ps google.pt google.ro google.rs google.ru google.rw google.sc google.se google.sh google.si google.sk google.sm google.sn google.so google.sr google.st google.td google.tg google.tk google.tl google.tm google.tn google.to google.tt google.uz google.vg google.vu google.ws bing.com

save

click Common ACL
click the plus button
target categories whitelist access whitelist
[blk_BL_searchengines] access deny
Default access [all] allow

To block ads (including on android and ios)
[blk_BL_adv] access deny

To block proxy sites
[blk_BL_anonvpn] access deny
Read though all the other categories and deny the ones you want

next click Do not allow IP-Addresses in URL (If this causes issues deselect it)
use safe search engines no longer works however you can click it as well.
Save
click General settings
click Apply
click Save

If you want you can do a quick test by setting up your pc to use the proxy and see how thing are working.

Part 3
Now we are going to set up a wpad read more here about wpad https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid
ssh in to pfsense
8
cd /
create the wpad.da file
vi /usr/local/www/wpad.da
wq

Create two new symbolic link files
Code: [Select]
ln -s /usr/local/www/wpad.da /usr/local/www/wpad.dat
ln -s /usr/local/www/wpad.da /usr/local/www/proxy.pac


Then go Diagnostics /Edit File
click browse
user
local
www
click wpad.da
add

Code: [Select]
function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.1.1:3128";
}

save


If you connect to a VPN you need to go direct for the VPN instead of the proxy, Remember you need to add the correct network class for the VPN  either A, B or C

Code: [Select]
function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

        if (isInNet(dnsResolve(host), "1.0.0.0",  "255.0.0.0" ))
        { return "DIRECT"; }
 
    return "PROXY 192.168.1.1:3128";
}

save
Go to Configure DNS Resolver add new host overrides
Host: wpad
Domain: mylocaldomain.local
IP Address: 192.168.1.1
Description: WPAD Autoconfigure Host
save
Next go to Services: DHCP server under Additional BOOTP/DHCP Options
add
Code: [Select]
number: 252 type: string value: "http://192.168.1.1/wpad.dat"
number: 252 type: string value: "http://192.168.1.1/wpad.da"
number: 252 type: string value: "http://192.168.1.1/proxy.pac"
save

set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save

Set your system to automatically detect settings (for windows it is in internet options connections lan settings).

You also have to set up the proxy setting for each program that cant connect (firefox, graphics drive software, vlc etc)

If you have programs that cannot connect and have no proxy setting you need to setup a firewall aliases
 Firewall/Aliases/IP
and add the destination server ip (use wire shark to help find the blocked Ips or in your firewall block rule enable Log packets that are handled by this rule, use http://ip-lookup.net/index.php to check what it is and add to the Aliases. If it is part of a domain add the domain)
now create a new firewall lan rule
IPv4 TCP * * * passAliases 80- 443 * pass rule.

Save

A note on smart phones (android, IOS, etc)
With android (not sure on other smart phones OS) you can not set it so that all the apps on the device use the proxy (not without rooting and other hacks), web browsers (google) will work fine using the proxy (if set in wireless connection options) but not apps or things like google play, so unless there is an option to use proxy for all apps on the device the most practical option here is just to allow smart phones to use port 80 and 443.
 
UPDATE 24 JUNE 2016
I have found that if you have connection issues using auto config for android or other smart phones try manually setting the proxy, now opening port 80 and 443 is not needed.

Now we should have pfsense all set up for web filtering. I hope this has been helpful and thanks to everyone on the forum who has help me in creating this guide.

Just a note for any specific issues with squid, squidguard or dns please create a new topic in the correct areas of the forum and link it here if needed
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: stilez on June 05, 2016, 08:30:21 pm
One step you missed:

Quote
make a symbolic link between the file
Then go Diagnostics /Edit File

You didn't say which file (and which target) to make the symbolic link, or the command you use for it. Some people might need to know.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 06, 2016, 12:31:52 am
Thanks, will update it soon.

Done
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: stilez on June 07, 2016, 04:34:58 am
Is "click wpad.da" and other "wpad.da" a typo?
If it's correct, it might be worth commenting after it that you do mean "da" not "dat", because having two files called wpad.da and wpad.dat might not be noticed, looks like a typo, is confusing, etc :)
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 07, 2016, 05:17:05 am
Hi, there are 3 wpad files
wpad.dat
wpad.da
proxy.pac
https://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol

I made the wpad.da the main file you edit and made a symbolic link for wpad.dat and proxy.pac (so all you need to do is just edit the wpad.da file).
If this is still confusing me know and I will update the guide.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: 91X on June 13, 2016, 08:48:29 pm
thanks for the guide  ;)
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Dogfish on June 16, 2016, 05:02:43 pm
Thank you for the guide!

Some guides have a step to assign MIME file types eg: ".wpad"    =>   "application/x-ns-proxy-autoconfig",

Is this not necessary if using the webconfig http file server?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 16, 2016, 05:41:16 pm
Quote
Some guides have a step to assign MIME file types eg: ".wpad"    =>   "application/x-ns-proxy-autoconfig",

Is this not necessary if using the webconfig http file server?

I do not believe it is needed.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 18, 2016, 03:55:59 pm
Can you please expand on the "use our route as the DNS server." part? Your list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.

edit: Also, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?

Also, how/where does the certs for https inspection get created/used?

Thanks
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 18, 2016, 06:57:23 pm
Quote
Can you please expand on the "use our route as the DNS server." part?
Sure, since we enabled the DNS resolver to do Host overrides to force safe search on a few search engines I also set up a rule that will force the local network to use the pfsense router as the DNS server. The pfsense router will cache DNS addresses and use them instead of calling a DNS server, you can also change the cache size.
Read more here https://doc.pfsense.org/index.php/Unbound_DNS_Resolver


Quote
Your list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.
Depending on your setup this could change, this is what I use
Tic Enable DNS resolver
Listen Port default which is 53
Network Interfaces all
Outgoing Network Interfaces all
System Domain Local Zone Type transperrent
Tic Enable DNSSEC Support
Untic Enable Forwarding Mode
Tic Register DHCP leases in the DNS Resolver
Tic Register DHCP static mappings in the DNS Resolver

For advance settings
Tic
Tic
Tic
Tic
Tic
Set cache size (I set 100MB)
10
10
4096
513
200
86400
0
15 min
20000
Disable
1
Untic
Untic

Quote
Also, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?
It should be just under the anti block out rule, so the second rule

Quote
Also, how/where does the certs for https inspection get created/used?

WPAD does not use certs, as there is no man in the middle attack

Hope this helps

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 18, 2016, 09:50:54 pm


Thanks aGeek

Actually I was referring to the "Firewall/NAT/Port forward" settings but I got it figured out anyhow.


Quote

WPAD does not use certs, as there is no man in the middle attack

Hope this helps

After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?


Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 18, 2016, 11:27:41 pm
Quote
After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?

No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!

Quote
Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.

I hope that I have cleared a few things up, have you got your setup working?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 19, 2016, 08:48:47 am
Quote
After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?

No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!

Quote
Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.

I hope that I have cleared a few things up, have you got your setup working?


no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Asterix on June 19, 2016, 03:38:20 pm
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 19, 2016, 06:17:42 pm
Quote
no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.
Is that the port 80 and 443 block rule? If so than that is correct, now set your device to auto configure proxy. In windows go to global internet settings and there is an option for that, and for each program you have set its proxy settings. For programs with no proxy settings create a pass rule.

Quote
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.

Hi, first try above post, second I have not tried clamd scanning because I have found it to have issues, best to ask this question in the proxy forum.

Quote
Using proxy setting in browser since wpad isn't giving the results I am expecting
You have to tell your browser to use system settings and in global internet settings set proxy to auto configure.

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 20, 2016, 07:49:39 am
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.

Same issue here now.

I enabled the 80-443 block rule, unchecked the 'Transparent" option in Squid and I can only get access if I manually enter the wpad.dat location into my local computer(s) settings. Auto discovery does not work. I'm on all Apple computers/devices here btw. Same issue with ClamAV, it scans http but not https as poster above.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Asterix on June 20, 2016, 09:40:56 am
Auto config works for me. What I meant in the original post was that the WPAD direction info I was writing was not proper hence was using straight proxy settings for the time being. Will be experimenting with the WPAD file at a later time.

I seriously doubt HTTPS scanning (not filtering) with clamd is working. I have followed the configuration directions to the T and yet the only thing which does not work is https clamd scans for viruses. I believe its a Squid issue.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 20, 2016, 06:13:12 pm
Post a link to the https fake virus test file that you are testing and I will see if it works for me.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 20, 2016, 07:53:52 pm
http://www.eicar.org/85-0-Download.html (http://www.eicar.org/85-0-Download.html)


There is one group for http, and one group for https.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Asterix on June 20, 2016, 10:23:53 pm
There are 4 https links

https://secure.eicar.org/eicar.com
https://secure.eicar.org/eicar.com.txt
https://secure.eicar.org/eicar_com.zip
https://secure.eicar.org/eicarcom2.zip
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Asterix on June 20, 2016, 10:34:58 pm
Would you know how to get the below google safesearch info in pfSense BIND DNS?


server: include: /var/unbound/forecegoogle.conf
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 20, 2016, 11:42:18 pm
Ok done some research squidclamav only supports http not https because it is encrypted.

Quote
would you know how to get the below google safesearch info in pfSense BIND DNS?

Not sure you will need to ask that in the proxy forum.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 21, 2016, 07:39:48 am
Ok done some research squidclamav only supports http not https because it is encrypted.


I just did a test. Squidclamav will scan https traffic when using Squids MITM option..
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Asterix on June 21, 2016, 08:38:49 am
Ok done some research squidclamav only supports http not https because it is encrypted.


I just did a test. Squidclamav will scan https traffic when using Squids MITM option..

And that would need certificates have to be installed on the clients..
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 21, 2016, 08:40:08 am
Ok done some research squidclamav only supports http not https because it is encrypted.


I just did a test. Squidclamav will scan https traffic when using Squids MITM option..

And that would need certificates have to be installed on the clients..

Correct.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 21, 2016, 06:17:24 pm
To help stop virus or spywhere we can enable squidguard block list blk_BL_spyware,
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: johnpoz on June 30, 2016, 09:39:28 am
What is the point of all the safesearch nonsense and redirecting users to only use your dns..  You do understand when a proxy is being used, the proxy does the query not the client..

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on June 30, 2016, 10:15:37 am
While setting up wpad we'r supposed ot enter the following into /usr/local/www/wpad.da:

Quote
function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.1.1:3128";
}

I've actually changed my pfsense server ip to 192.168.0.1 - do I need to edit both ip addresses listed above?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 30, 2016, 10:18:14 am
What is the point of all the safesearch nonsense and redirecting users to only use your dns..  You do understand when a proxy is being used, the proxy does the query not the client..

I think the point of the safe search is to stop, say your young kids, from Googling porn images. I'm not sure about the dns redirecting.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 30, 2016, 09:56:59 pm
What is the point of all the safesearch nonsense and redirecting users to only use your dns..  You do understand when a proxy is being used, the proxy does the query not the client..

I think the point of the safe search is to stop, say your young kids, from Googling porn images. I'm not sure about the dns redirecting.

1. What is the point of forcing search engines from using safe search?
Answer: To aid in the filtering of adult content, this is most importantly for google images as squidguard does not block them, if you do not want to filter web content then this guide is not designed for you.

2. What is the point of redirecting users to use THEIR pfsense router as the DNS server.
Answer: There are many advantages not all relating to web filtering however tries and stops the user from bypassing the dns redirect rule.

While setting up wpad we'r supposed ot enter the following into /usr/local/www/wpad.da:

Quote
function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.1.1:3128";
}

I've actually changed my pfsense server ip to 192.168.0.1 - do I need to edit both ip addresses listed above?
Yes, set it to 192.168.0.0
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on July 01, 2016, 08:11:24 am
I'm constantly getting these entries in the Squid Real Time log:

Code: [Select]
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:59488 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:55735 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:49806 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:46365 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:38156 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:25012 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:24866 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:14826 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:10196 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:6263 - -

192.168.1.5 is my local machine. Is this normal?


Update: Resolved this in another thread..
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on July 04, 2016, 09:07:38 am
Quote
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443

Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just  bypassing the autoproxy/wpad stuff?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on July 04, 2016, 09:38:07 am
The Squid Guard settings aren't working for me.  I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and    Default access [all] options there.

On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 04, 2016, 05:59:17 pm
Quote
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443

Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just  bypassing the autoproxy/wpad stuff?

If port 80 and 443 are lefted open then the user can simply untic the auto configure proxy (on their PC, Mac, phone etc) and set the setting to go direct. This will not call for the wpad and the user will go direct and not use the proxy.

The Squid Guard settings aren't working for me.  I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and    Default access [all] options there.

On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?

1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_package

2. Transperrent proxy does not use a wpad, if you want to use transperrent proxy for http then you need SSL Man In the Middle Filtering for https which must have a certificate installed on evey device.

So you can either set auto configure proxy on all devices or install a certificate on all devices.

You cannot use a transperrent proxy and a wpad at the same time. If you did get a transperrent proxy with SSL mitm working then there is no need for manual proxy mode.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on July 05, 2016, 09:24:48 am
1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_package

That was the problem - there were no other categories than whitelist and default access.  I managed to get the others by enabling "Blackilist" under the general settings of Squid Guard.  Forgive me if I just missed it, but I couldn't see that step in the guide.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2016, 05:42:36 pm
I added
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 07, 2016, 09:31:03 am
I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on July 15, 2016, 02:14:22 pm
Just to point out that this approach does not play well with many Roku channels - at least on my Roku 3 anyway.

Forcing youtube to be restricted causes all videos to fail on the Roku app.

The step " Do not allow IP-Addresses in URL" breaks Netflix.

There is also something that breaks the ITV Hub channel - although I haven't figured out what yet.  My experiments suggest that it's something to do with Squid, but probably no squid guard.

The Roku remote is also somewhat less reliable in connecting after a reboot of the Roku, although I'm surprised at that, because I thought the Roku used it's own wifi to connect to the remote independently of the main network.

I'm not saying all this because I'm asking for fixes (although any suggestions that might provide fixes would be useful), but as information for anyone setting this up who has a Roku on their network.

I'm still, largely, using this approach, although, obviously, it's somewhat weaker than it would be if I didn't have to make compromises to get my Roku working.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 15, 2016, 08:18:59 pm
Bypass the proxy for Roku and Netflix.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: wifiuk on July 24, 2016, 01:32:45 pm
So i have got wpad all working fine using some other guides, everything goes via the autodisovery and all is good.

That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.


If i set a firewall rule to block 80 - 443 on the LAN it works, and all the devices work except the android devices, which is what i expect to happen.

What i want to do is redirect all 80 - 443 requests to the squid port 3128.

But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.

Can someone advise me the best way to force all android traffic via squid without manually setting up each device..
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 24, 2016, 06:19:49 pm
Quote
That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.

Manual setup is the only way I got it to work for android (5.1) and for older versions manual setup is the only option.

Try enabling transperrent proxy with the wpad, this will help with port 80 getting blocked.

Quote
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.
I have read that too (never tried) might be your best option.

Or just get your users to setup the proxy?

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on July 24, 2016, 11:59:43 pm
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.

You are not obliged to redirect to LAN IP. Thik about localhost  ;)  then I don't think there is any loop.

What makes this "dual proxy" option difficult is that, in order to set-up transparent proxy, you will have to allow requests ton internet on port 80, while, when configuring explicit proxy, you may (should) want to deny such access so that you ensure everything goes through your proxy.
And same for HTTPS except that transparent proxy, unless you configure ssl-bump, will not intercept HTTPS flow.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on July 30, 2016, 08:25:22 am
Bypass the proxy for Roku and Netflix.

This fixes the issue with itv hub, although bypassing for netflix.co.uk - works on the roku, but then breaks netflix on Android - presumably they're using different destination ips.

Will have to re-enable ip addresses in the url in squid.

Also, this doesn't fix the problem with youtube - you just have to allow unsafe mode...
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: everwake on August 03, 2016, 02:29:27 pm
this guide was very nice, but it does not work for me.

http proxy part works fine (transparent mode)

on android using automatic discovery it uses the wpad file, and looking in the log I can see entries like:

10.111.11.111/android-a41b317d63f30562.local   sb.scorecardresearch.com:443   Request(default/blk_BL_tracker/-) - CONNECT REDIRECT

which implies to me that https is working, but regular expression matching for a site only works if it is http, not over https, which means that filtering using regexp for youtube and google does not work.
no filtering on sites using https seems to be happening.

safe search for youtube only works for desktop-pc, not on youtube apps. the desktop pc's say that safe search was forced by the network admin (which is what I want), but our android pads don't seem to care (rendering safe search totally useless)

I found this looking for a way to filter youtube and google, but since both sites use https it does not work, has anyone got it working or should I go to using certificates?

Regards

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: dotdash on August 03, 2016, 04:52:33 pm
Just a note, you can simplify the directions by touching the file instead of creating with vi.
e.g.
Ssh into the router
type 8
cd /
cd var/unbound
vi forecegoogle.conf
leave blank for now
save (wq)


Could be done with-
Ssh into the router
type 8
touch /var/unbound/forecegoogle.conf

Or you could just paste the touch command into diagnostics, command prompt and not shell in.
Just a suggestion, there are many ways to do these things.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: oddworld19 on August 04, 2016, 08:06:38 am
I have followed the guide, WPAD is working and the proxy is working in NON-transparent mode. I want to block specific subreddits, like Reddit.com/r/nsfw without blocking the remainder of Reddit.com. I do not want to block the entire domain, and I use pfblockerng and DNSBL with unbound to handle DNS and Top level domain blocking. I don't want to block all of Reddit... Just the NSFW material (and I have a list of all websites in squidguard).

At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on August 04, 2016, 08:19:28 am
At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.

No this will not work without MITM because CONNECT method on which Squidguard relies in order to block HTTPS knows only the left part of your URL, meaning here http://reddit.com
This part is sent in clear text but then everything else is within HTTPS tunnel thus this can't be read therefore blocked.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: oddworld19 on August 04, 2016, 08:42:16 am
Since that is literally the only site I'm trying to block on squid (and everything else is blocked using DNSBL / pfblockerng) how would you suggest I intercept HTTPS?

Do you think I could alter the WPAD rules to PROXY reddit.com and default to pass/DIRECT all other traffic? Then on squid, I enable MITM/SSL. I shouldn't use transparent mode, right, because WPAD would notify users of the proxy?

Would I use port 3128 or 3129 (I'm a little confused about the separate SSL port in the options).

Since this isn't a banking site or google site, any idea if I could configure squid such that no certs are needed on the host computers?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on August 04, 2016, 09:52:35 am
I can't really answer  :-[

To me DNSBL doesn't aim at replacing HTTP filtering. These are on 2 different layers. If you implement DNSBL, then you will access less sites, that's it (and this is already not so bad) but HTTP proxy permits to filer at fqdn level (like DNSBL does) but also in URL (for HTTP or HTTPS is MITM) and, on top of that, to check for virus (again HTTP and also HTTPS if MITM)

Then decision to deploy proxy in transparent mode or not is entirely on your side. I'm definitely not prone to deploy transparent proxy but can understand that is some cases it may help.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: oddworld19 on August 04, 2016, 10:18:12 am
Thanks. Yes - I use DNSBL just to block at the DNS level. pfblockerng has a new update where it also includes top level domains. It does a pretty good job of blocking content at the "domain-level" via DNS. I then set-up rules in pfsense to require the network to use pfsense's DNS unbound server. I agree these are two fundamentally different technologies. However, the DNS blacklist does not cover quite everything I intend to block. I want reddit.com to work, but want to block certain sub-reddits. Obviously, DNS is the wrong tool for this job, which brought me to squid.

I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.

Does SSL filtering work in non-transparent mode?

I assume so. Is the port still 3128?



Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on August 04, 2016, 10:22:36 am
I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.
Does SSL filtering work in non-transparent mode?
I assume so. Is the port still 3128?

yes and yes.
And BTW you don"t need to edit proxy.pac in order to go direct for all sites but reddit.
You can still use proxy for all and only block reddit content.
Other HTTP sites will benefit from cache and all sites will potentially benefit from anti-virus if one day you enable it.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: oddworld19 on August 04, 2016, 10:28:29 am
Thanks. Right. My only thought there was:

I understand SSL decryption and inspection gets.... Nasty.

For example, if browsers require HSTS or whatever other protocols check certificates, then it might get annoying to keep installing certs on all on my computers on the LAN. For example... A small FTP server has no need to access Reddit, but I don't know if SSL bumping will cause issues with package updates in the future (without a cert).

Maybe it will... Maybe it won't. I don't know... Do you think it would potentially be an annoyance if I don't have certs installed?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: maverik1 on August 17, 2016, 12:03:31 am
Lots of good info! Thanks!!

Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on August 17, 2016, 12:31:43 am
Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.


What "notification" are you speaking about?
I'm confused  :-[
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: maverik1 on August 17, 2016, 07:36:35 pm
I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: maverik1 on August 17, 2016, 07:39:59 pm


Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.


How can we get this working with mobile devices Android and iOS that use the youtube mobile app or m.youtube.com?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: chris4916 on August 18, 2016, 02:49:33 am
I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: maverik1 on August 18, 2016, 12:01:34 pm
Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)

Push notifications are alerts that are received by cellular phones;mostly smart phones, for different types of apps such as social media, email, games and whatnot. I have no issues receiving these alerts when using data only. However, when I was on my wifi network they were not coming through. I spend a few weeks working with the firewall rules and proxy settings trying everything I could think of to troubleshoot the issue. Turns out the problem was with the phone and not pfsense.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: kpa on August 25, 2016, 01:39:12 pm
I'd be very surprised if all those "push" system weren't all emulated by polling a server periodically, that kind of set up needs absolutely nothing else but an outgoing connection from the device to the server. The other option would be a service running on the device and the notifications would be then really pushed on the device by an incoming connection from the server to the device. I don't think you'd want implement it like that however  ;)
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: laurenzzo on August 27, 2016, 07:43:36 am
[...]

First we are going to setup squidguard
Update
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).


In Package/Proxy filter SquidGuard: General settings/General settings
click blacklist
enter http://www.shallalist.de/Downloads/shallalist.tar.gz
download
wait to finish

[...]

Hi aGeekHere,

Thanks a lot for your guide !
I tried to follow your steps to get my blacklist downloaded but without success.

Could you please have a look on my post for more details ?

https://forum.pfsense.org/index.php?topic=117314.msg649961#msg649961 (https://forum.pfsense.org/index.php?topic=117314.msg649961#msg649961)

I really don't understand why the download doesn't work...

Thanks in advance for any help !

EDIT : SOLVED !
this was an issue with Firefox...
Could you please update your nice guide with a small note ? (e.g. "don't use Firefox to download the blacklist, IE do the job correctly")

Thanks !

Kind regards,
Laurenzzo
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: jadog on September 02, 2016, 08:21:54 am
Thanks for taking the time to post this excellent guide! Just a few additions I would recommend adding.

YouTube mobile still allows unrestricted access. Including the below corrected it for me.

Code: [Select]
Click add under Host overrides
Host = m
Domain = youtube.com
IP =  216.239.38.120
Description = youtube mobile
Save

Also when using vi after you ssh into your router, using wq does not allow saving. You need to use ":wq" (without the quotes). All working perfectly for me!
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: eixel05 on September 06, 2016, 01:06:35 am
Hi,

First of all thank you so much for this guide, just what I've been looking for!

Before this guide I was using Transparent+MITM approach but because some HTTPS sites doesn't load as they should (I'm not sure if the cause was indeed MITM but after turning it off the sites loads fine)

I followed this guide thoroughly, HTTP filtering works fine but I don't know if HTTPS filtering is working? Before, I am using the tail command (/var/squid/logs/access.log) and while MITM is on, https site access is being logged but now with this approach, not a single HTTPS is being logged.

Is there another way to check if indeed this approach is working for HTTPS?

Out-of-topic question, since my main purpose for this Proxy server is local cache, Are HTTPS objects being cache as well? or only HTTP?


Update (09/07/2016): NVM, I figured it out :D all is working great, thank you so much for this guide!

I'M REALLY SORRY FOR BEING NOOB, I just started my way to "Networking", please be patient with me.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: Maxim_Al on September 15, 2016, 10:58:54 pm
Can anybody help me?
I test  pfsense for gateway/proxy for a company. But I have one strange trouble:
The pfsense is installed and work but if I use it for access (through proxy) to https://code.getmdl.io/1.2.1/material.grey-orange.min.css  I have in proxy log:
Date    IP    Status    Address    User    Destination
16.09.2016 14:37:28    192.168.1.222    TAG_NONE/503    code.getmdl.io:443    -    -
What is it an how I must to do that resolve it?


UPDATE 28/10/2016 - This was provider :) not pfsense!
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: klmiciano on October 27, 2016, 08:20:08 pm
Which rule should be above between these two?

FIRST -- 
"Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save"

SECOND --
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save"


Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: klmiciano on October 27, 2016, 08:36:38 pm
I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.



How to do this?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on October 29, 2016, 03:21:55 am
Which rule should be above between these two?

FIRST -- 
"Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save"

SECOND --
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save"
first one on top

I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.

How to do this?

after you have the wpad setup and working enable transperrent proxy in squid
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: phunni on November 24, 2016, 11:08:19 am
If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?

I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there...
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: jbourn1907 on November 25, 2016, 02:45:01 am
Sorry I'm new in pfsense.

I have an internal DNS server windows server 2008. How can I blocked internal computers to direct access to http. I want to filter it through pfsense firewall. Can you give me some screen shots or sample of rules and where can I put this?

I want all computers to go first in pfsense proxy before connecting to internet. I already configure transparent proxy.

Thanks for the help.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on November 25, 2016, 07:04:57 pm
"How can I blocked internal computers to direct access to http"
read the part in the guide about blocking port 80 and 443 and instead of using pfsense as your DNS server use your internal one, (for more help on this issue ask in the Cache/Proxy forum).

"If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?....I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there..."
If the transparent proxy has issue with the vpn you may need to bypass it.

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: genesislubrigas on December 16, 2016, 06:19:24 am
thanks for this tutorial.  i am not using dns resolver but bind.  can you also show how to configure bind.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on December 20, 2016, 09:39:07 pm
Have not used bind, but after a quick google try http://blog.muhammadattique.com/configuring-bind-dns-server-on-pfsense-firewall/

If it does not help ask in the main forum
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: nib01 on December 23, 2016, 02:46:29 pm
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save

Set your system to automatically detect settings (for windows it is in internet options connections lan settings).

You also have to set up the proxy setting for each program that cant connect (firefox, graphics drive software, vlc etc)

If you have programs that cannot connect and have no proxy setting you need to setup a firewall aliases
 Firewall/Aliases/IP
and add the destination server ip (use wire shark to help find the blocked Ips or in your firewall block rule enable Log packets that are handled by this rule, use http://ip-lookup.net/index.php to check what it is and add to the Aliases. If it is part of a domain add the domain)
now create a new firewall lan rule
IPv4 TCP * * * passAliases 80- 443 * pass rule.

Save"

Not sure where to setup/configure above firewall lan rule.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: rsnsmh on December 30, 2016, 12:40:42 am
Thank you for the guide - i got everything working as expected. However, I need to enable content filter on a 2nd interface (Guest). What sort of config addition is needed to enable content filtering on the 2nd additional interface.   
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: bole5 on February 20, 2017, 11:40:26 am
In the Part 3 you refer to *.local in proxy.pac file as well as unbound configuration.

Code: [Select]
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

When user sets up pfSense the domain defaults to "localdomain" in System/General Setup and there is explicit sentence about not using .local as a domain name:
Code: [Select]
Do not use 'local' as a domain name. It will cause local hosts running mDNS (avahi, bonjour, etc.) to be unable to resolve local hosts not running mDNS.

Do we need to modify the domain to "local" in System/General Setup or should we replace "*.local" with "*.localdomain"?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on February 20, 2017, 06:41:56 pm
Use a domain like this
pfsense.thisismydomain.local

Do not use
pfsense.local.local

The PAC does not need changing
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: molykule on April 05, 2017, 12:50:35 pm
Hi,

I am using the method mentioned here, through unbound. Is there a way i can let few static IP's skip the youtube safe settings in the unbound. I tried asking about segregating those ip's but then i have to use bind which i dont know about. If there is a way to skip the DMZ (i have lan, opt1, opt2 and dmz), that would help also, as i can put those static IP's on DMZ
I think it was mentioned somewhere to use port 5353 and use DNS forwader in conjunction with Unbound, but i cant find any tutorial for it. Also, i cant find any way to split the DNS i dont know if that would help. All I want is some computers can visit youtube and dont get blocked for videos. Even pfsense tutorials get blocked by it.
Any ideas as to where to look for or what to look for would help,
thanks for the great tutorial.

Molykule
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: huuur on April 10, 2017, 06:50:45 am
Beautiful guide!
I decided to move from ipcop to pfsense just to filter some https, I followed this guide (with the updates). Everything seems to work fine before some users start complaining mainly android apps not working (play store, snapchat, whatsapp..etc) I managed to locate the blocked (using firewall log) then create a bypass rule with the IP or Port for the whole network, however other clients start having similar issues as if those apps working on different IPs for certain devices (all android)
I read this topic 4 times, reinstall from zero three times without any improvement towards this issue which begins after applying the proxy with wpad.
I appreciate any advice even if an easier approach to filter some unwanted https sites.
Thank you.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on April 10, 2017, 09:59:08 pm
On the android phone (wireless setting) try setting the proxy settings manually, instead of auto or none.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: huuur on April 11, 2017, 07:10:51 am
On the android phone (wireless setting) try setting the proxy settings manually, instead of auto or none.

Finally, with manual proxy settings I have a stable communication with android devices.

You are the MAN!
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: huuur on May 23, 2017, 06:26:32 am


Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.


How can we get this working with mobile devices Android and iOS that use the youtube mobile app or m.youtube.com?

I can't express how useful this guide been for me.

For the record I found the below is working for youtube mobile app safe search:

Code: [Select]
Host = youtube
Domain = googleapis.com
IP =  216.239.38.120
Description = youtube app1

Code: [Select]
Host = youtubei
Domain = googleapis.com
IP =  216.239.38.120
Description = youtube app2

and maybe..
Code: [Select]
Host = www
Domain = youtube-nocookie.com
IP =  216.239.38.120
Description = youtube nocookie

in addition to what's mentioned in Reply#60 for mobile browsers.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on May 24, 2017, 04:41:47 am
Thanks huuur, that should help others.

Though it would be good if youtube was better at filtering videos.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 08, 2017, 08:25:43 pm
Update

You can try setting up MITM by setting the SSL/MITM Mode to splice all, that way you do not need to create a certificate for each device on the network. (you still need to create a main certificate though).
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on June 09, 2017, 05:47:50 am
Update

You can try setting up MITM by setting the SSL/MITM Mode to splice all, that way you do not need to create a certificate for each device on the network. (you still need to create a main certificate though).

That works fine for me also setting Client proxy settings manually:
Proxy address/PORT= SQUID_IP 3128

And that only! Indeed if i specify different proxy settings for http/https in (client win10=> Internet Settings=>Lan settings=>Advanced=>
- http=SQUID_IP 3128
- https=SQUID_IP 3129

It does not work, as in the contrary I'd excpect....  ::)
Geek can you explain why?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 10, 2017, 01:52:00 am
Does automatically detect settings work?

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on June 14, 2017, 07:22:22 am
Does automatically detect settings work?
Ehmm sorry, could you explain me better what you mean? As I stated I do not use WPAD, I configure proxy manually on client see picture...

Thanks.
PS. Sorry to answer late
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 19, 2017, 04:10:13 am
if you have the transparent proxy working then you do not need to define the proxy server.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on June 19, 2017, 09:37:25 am
If I use it in transparent mode, https does not work!
Better sometimes it works, sometimes it does not! (http works always!)

If anyone managed to get http+https in splice all + transparent mode work please let me know... ;)
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 19, 2017, 06:57:33 pm
I just enabled it mitm, added a cert, set it to splice all and its working.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on June 20, 2017, 09:51:01 pm
The guide is update but confusing.  Can you please clean up the guide and have it step by step.  Its somewhat hard to follow if you are not familiar with wpad and related firewall rules. 

Thanks for the guide.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 21, 2017, 02:33:14 am
hi techbee, yeah after discovering a few new things the guide is a little messy now.

The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.

when i get time I will try an clean it up a bit.


Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on June 22, 2017, 10:27:17 am
The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.
Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive          Advantages/features                                                                    Disadvantage
Splice (all)         No needs of certificate installation on Clients + webfiltering              No traffic analysis i.e no AntiVirus
Bump (MITM)    Traffic analysis i.e noYES AntiVirus     + webfiltering                               Needs to install certs on clients


Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

 
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: AR15USR on June 22, 2017, 12:08:45 pm
The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.
Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive          Advantages/features                                                                    Disadvantage
Splice (all)         No needs of certificate installation on Clients + webfiltering              No traffic analysis i.e no AntiVirus
Bump (MITM)    Traffic analysis i.e no AntiVirus     + webfiltering                               Needs to install certs on clients


Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

Shouldn't the second one not have the word "no"..
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on June 23, 2017, 02:58:43 am
Shouldn't the second one not have the word "no"..
Sorry cut & paste mistake, I fixed it.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: mahnonsaprei on June 29, 2017, 04:59:44 pm
Hi at all,

I'm Marcello from italy. Nice to meet you, i'm a newbie of pfsense.

I have read the tutorial, but i have one problem.
When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

I'm looking for in google but i not found the answer.

Please help me.
Thank you veeeeery much!!!
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on June 29, 2017, 07:10:59 pm
Quote
When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

Known issue with squid, I do not think there is a fix for it.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: mahnonsaprei on June 30, 2017, 09:08:53 am
Thank you so much. There aren't alternatives to redirect https via proxy also blank page or other?

B.r.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 03, 2017, 04:45:50 am
hello ageekhere,

can you update your guide now.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 03, 2017, 08:44:22 am
what part are you stuck on?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 03, 2017, 09:03:58 am
1. I dont know what exact firewall rules to add
2. I am confused if I need to install unofficial wpad package of marcelloc.
3. I am confused if there is a need to have separate webserver of wpad.
4. I am confused if I can enable both transparent and ssl mitm filtering on squid.
5. I cant nslookup wpad on clients command prompt to resolve to my pfsense box ip.

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 03, 2017, 10:15:15 pm
1.
The firewall rules that you need are
Code: [Select]
IPv4+6 TCP * * * 80 (HTTP) * none port 80 block
IPv4+6 TCP * * * 443 (HTTPS) * none port 443 block 
This blocks access to port 80 and 443, it stops users from bypassing the proxy.

Also if you want to force pfsense to be the DNS server use.

Code: [Select]
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

Have a look through a few of these videos https://www.youtube.com/playlist?list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk
There are a few really good configs he does there.


2. Wait for the package to become stable and available in the package manager, in the mean time you can use the pfsense webserver, put the wpad files in /usr/local/www/ read Part 3. (all that needs to be done is add the wpad files (wpad.da, proxy.pac and wpad.dat) to /usr/local/www/).

3. Just use pfsense webserver /usr/local/www/ (I have not used an external webserver)

4. This is what I did, I set up a WPAD and transparent proxy with ssl mitm (splice all so you do not have to install cert on every device) and set your device to auto configure. I found that just using WPAD a lot of programs that cannot use the proxy were getting blocked so the transparent proxy catches that and something the transparent proxy ssl has issues (like windows updates) however the WPAD is used for that. Best of both worlds.

5 I have had issues as well, if you put http://192.168.1.1/wpad.dat in your web browser and it downloads than it is working.

This guide goes a few steps fever by forcing google and bing to be in safe mode (even youtube).

So what I would do is get WPAD working then enable the transparent proxy with ssl mitm (create a cert for the router).

Hope this helps
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 03, 2017, 11:11:51 pm
thanks for the reply aGeekHere.

What is the tcp port that you used on your webconfigurator ?  If I block port 80, then, I will also be blocked opening the pfsense gui.

Also, regarding to my question number 4, do you mean that you did not enable the http transparent proxy on port 3128 but only the HTTPS/SSL MITM on port 3129 and setup wpad. did I understood you correctly ?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 03, 2017, 11:58:57 pm
Quote
What is the tcp port that you used on your webconfigurator ?


set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

Quote
If I block port 80, then, I will also be blocked opening the pfsense gui.
No it should not (have the rule above the anti lockout rule);

Quote
Also, regarding to my question number 4
Enable squid proxy 3128
Get WPAD working follow guide

Enable transparent mode to forward all requests for destination port 80 to the proxy server.

Enable SSL filtering
Splice all
create Certificate (for pfsense).

Summery
Set Advanced: Admin Access Protocol to http
Enable squid proxy
Follow guide on setting up WPAD
Enable transparent mode
Enable SSL filtering Splice all
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 04, 2017, 12:09:24 am
Yep, done that all.

I can download the wpad files from the browser. I can access the http sites but still cannot access the https sites.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 04, 2017, 12:22:17 am
what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?



Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 04, 2017, 12:41:17 am
what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?
'

Hi aGeekHere,

Yes, cache is cleared,
os=win7 pro sp1,
I disabled the update coz I dont need it,
autoconfigure is set,
yes, google and yahoo https sites. facebook and youtube does not load and shows error.

I use splice-all.  Don't I really need to install certs to client device ?

If https site is successfully blocked, what error page do I expect to see ?  Can you show me the screenshot for the error page that will show !

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 04, 2017, 12:52:43 am
firefox ie or chrome?

Quote
I use splice-all.  Don't I really need to install certs to client device ?
not when splice all

Quote
If https site is successfully blocked, what error page do I expect to see ?
The default cannot access page of the web browser.

If you enabled squid, transparent mode and SSL filtering Splice all with a cert on pfsense and you still cannot access https sites, then something is broken.
Try reinstall.
If you still have the same issue post screen shots of your setup.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 04, 2017, 01:17:42 am
firefox ie or chrome?

Quote
I use splice-all.  Don't I really need to install certs to client device ?
not when splice all

Quote
If https site is successfully blocked, what error page do I expect to see ?
The default cannot access page of the web browser.

If you enabled squid, transparent mode and SSL filtering Splice all with a cert on pfsense and you still cannot access https sites, then something is broken.
Try reinstall.
If you still have the same issue post screen shots of your setup.

I am using chrome. most of the users are chrome.
On the other hand, the firewall rules you shown was to block port 80 and 443 right and redirect dns resolution to pfsense dns server and no more else.  That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right ?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 04, 2017, 01:32:37 am
Quote
That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right
That is what the transparent proxy is doing, remove that rule, that could be what is causing the issue.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 04, 2017, 01:44:02 am
Quote
That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right
That is what the transparent proxy is doing, remove that rule, that could be what is causing the issue.

Alright

So what I have now on my firewall rules are:
1. antilock rule on port 80 which says (allow all on tcp port 80)
2. block LAN_NET on tcp port 80 and 443
3. redirect rule for dns resolution

would that be ok ?

On the other hand, what is the default cannot access page of the web browser, is it the same like of that of http transparent proxy error page ?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 04, 2017, 11:37:21 pm
aGeekHere,

My wpad now works.  I can download it from chrome browser.
I can ping and nslookup wpad now.
All squidguard group ACL and its target rules are applied to block facebook and youtube for testing purposes for now.
I HAVE NOT INSTALLED THE CA CERT ON CLIENT DEVICES.

I used http and https filtering with splice all option so that I dont need to install the cert to client devices.

I also cleared browser cache and history.

The result is I can browse http but cannot browse https.  Https sites like yahoo and google mail works though.

I have screenshot of the error for browsing https site.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 12:11:00 am
Is squidguard blocking it by mistake? is it set to deny all?

Try allowing all in squidguard rules to test.

Post screen shots of squidguard.

Try clearing states in pfsense.

Reboot.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 01:13:55 am
Squidguard is not blocking by mistake.

I tried clearing firewall states and Rebooted, still the same situation.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 01:27:21 am
I also have this wireshark done where I can see that it made a CONNECT method to facebook, then it has established connection, send client hello but later have RST and there is no SERVER HELLO thus no application data received.

What could I miss here.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 02:03:18 am
lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 02:08:27 am
lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.


If I use only the http proxy without the ssl mitm and set browser to auto, then the http proxy filter works.

If you are saying to setup http transparent proxy and ssl mitm filtering and install cert to client device and set browser proxy, then I have not done this yet. 
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 02:35:09 am
Quote
and install cert to client device and set browser proxy, then I have not done this yet.

Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
Set browser to direct.
This will use transparent proxy for both http and https.

If this works then there is an issue with your WPAD.
If this does not work then something is broken.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 02:41:06 am
Quote
and install cert to client device and set browser proxy, then I have not done this yet.

Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
Set browser to direct.
This will use transparent proxy for both http and https.

If this works then there is an issue with your WPAD.
If this does not work then something is broken.

What do you mean for "Set browser to direct", is this the automatic detect settings ?

On the other hand, my wpad only have the following inside:

 function FindProxyForURL(url,host)
 {
 return "PROXY 192.168.1.1:3128";
 }

I am not sure what you example below means so I use the example from pfsense wiki.

function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.1.1:3128";
}
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 02:44:18 am
yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 03:00:11 am
yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.

By turn off, you mean uncheck the automatic detect settings and others options as well ?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 03:27:08 am
yes
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 05:08:45 am
May I ask though, what is the exact version of pfsense that you are using just for comparison.  If this will not work, then I will need to reinstall my pfsense box, maybe its broken.

I am using:

2.3.4-RELEASE (amd64)
built on Wed May 03 15:13:29 CDT 2017
FreeBSD 10.3-RELEASE-p19


On the other hand, below is the screen shot of my testing NOT USING WPAD but setup the browser proxy.


Also, I tried direct approach that you said, and the result the same error as before.

What exact pfsense version are you using ?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 08:21:58 am
2.3.4-RELEASE (amd64)
Something is broken.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: techbee on July 05, 2017, 08:54:27 am
2.3.4-RELEASE (amd64)
Something is broken.

what is in your wpad file?

did you install your pfsense or upgraded to 2.3.4 release?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 05, 2017, 10:16:32 am
wpad is in the guide.

I have been upgrading since i joined.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: 4Qman on July 11, 2017, 08:38:01 am
Hi,

I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: 4Qman on July 11, 2017, 02:39:26 pm
I have followed this guide.

When I run proxy test it shows that I am under proxy. Google works fine, and so do most websites. But such websites as Amazon won't load, gives error.

Any advice would be greatly appreciated.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 11, 2017, 06:22:37 pm
Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: 4Qman on July 12, 2017, 12:07:21 pm
Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

Hi thanks for your reply.

I followed the guide, page 1.  I copied and pasted logs from PackageSquidGuardLogs


Quote
10.07.2017 17:06:47   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35   [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 17:05:35   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 17:05:35   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 17:05:35   [sg_create_config] Add Default
10.07.2017 17:05:35   [sg_create_config] Add rewrites: safesearch;
10.07.2017 17:05:35   [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 17:05:35   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 17:05:35   [sg_create_config] Add blacklist entries
10.07.2017 17:05:35   [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 17:05:24   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 17:05:24   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 17:05:24   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 17:05:24   [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 17:05:24   [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 17:05:24   [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 17:05:24   [sg_reconfigure_user_db] Add user entries
10.07.2017 17:05:24   [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
10.07.2017 16:36:15   [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 16:36:15   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 16:36:15   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 16:36:15   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 16:36:15   [sg_create_config] Add Default
10.07.2017 16:36:15   [sg_create_config] Add rewrites: safesearch;
10.07.2017 16:36:15   [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 16:36:15   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 16:36:15   [sg_create_config] Add blacklist entries
10.07.2017 16:36:15   [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 16:36:05   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 16:36:05   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 16:36:05   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 16:36:05   [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 16:36:05   [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 16:36:05   [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 16:36:05   [sg_reconfigure_user_db] Add user entries
10.07.2017 16:36:05   [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
09.07.2017 18:01:06   [squid_reconfigure] Add new redirector options to Squid config.
09.07.2017 18:01:06   [squid_reconfigure] Remove old redirector options from Squid config.
09.07.2017 18:01:06   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
09.07.2017 18:01:06   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
09.07.2017 18:01:06   [sg_create_config] Add Default
09.07.2017 18:01:06   [sg_create_config] Add rewrites: safesearch;
09.07.2017 18:01:06   [sg_create_config] Add destinations: whitelist; blacklist;
09.07.2017 18:01:06   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
09.07.2017 18:01:06   [sg_create_config] Add blacklist entries
09.07.2017 18:01:06   [squidguard_rebuild_db] Start rebuild DB.
09.07.2017 18:00:56   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
09.07.2017 18:00:56   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
09.07.2017 18:00:56   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.


Proxy Config
# This file is automatically generated by pfSense
# Do not edit manually !

http_port 192.168.1.1:3128
http_port 127.0.0.1:3128
icp_port 0
digest_generation off
dns_v4_first off
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr drobinson@brightsure.co.uk
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/local/libexec/squid/pinger

logfile_rotate 2
debug_options rotate=2
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  192.168.1.0/24 127.0.0.0/8
forwarded_for on
httpd_suppress_version_string on
uri_whitespace strip


cache_mem 1000 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 1000 MB
cache_dir ufs /var/squid/cache 3000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|?) 0  0%  0
refresh_pattern .    0  20%  4320


#Remote proxies


# Setup some default acls
# ACLs all, manager, localhost, and to_localhost are predefined.
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
acl sslports port 443 563 

acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings


# Custom options before auth


# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc


Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 13, 2017, 07:46:14 pm
if you set squidguard to allow all do still get the issue?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: 4Qman on July 14, 2017, 03:30:28 pm
if you set squidguard to allow all do still get the issue?

Hi, thank you for the reply.

I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

At the moment I have Wpad.dat working, I also have MSSL MITM working on splice all, I have android working now (manually entered wpad.dat address).

Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on July 17, 2017, 06:07:47 pm
Quote
Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?
Not sure, maybe if you had a very large number of users you could use another webserver for the wpad.

Quote
I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings
Even if you left that unticked the transparent proxy should have still connected you, verify that the transparent proxy is working (e.g block port 80 and 443 and use a program that gets updates form the net to check).
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: rebman77 on September 09, 2017, 09:32:21 am
Hi,

I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

Was looking at this guide, https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense (https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense) and comparing to your screenshot.  What are the implications of unchecking the "invert match" box and not disabling NAT reflection?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: bwlinux on September 30, 2017, 01:05:50 pm
Thanks aGeekHere!   This guide encouraged me to revisit HTTPS filtering for several of my sites.

The WPAD options seems great for PCs but didn't work well on Android devices.  Things get really difficult when you have 2 or more network interfaces you want to proxy/filter using WPAD.

In the end, I just set up transparent proxy for both http and https. Creating a CA on the firewall and then assigning that in Squid for the CA along with "Splice All" for SSL/MITM Mode works great!

I think it's been mentioned before but your guide would really benefit from having 2 completely separate sets of instructions for transparent only and WPAD only.


Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on September 30, 2017, 06:16:17 pm
I now recommend that you use both, WPAC as the default and transparent with splice all as backup. So software that has no proxy settings still get redirected to the proxy in stead of getting blocked by the firewall rule 80 443. For android you can manually set the proxy, sometimes splice all can show SSL errors when web browsing. I will clean up the guide when I have time.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: minhgi on November 29, 2017, 02:29:26 pm
After using the guide and having everything setup correctly, I noticed pfblockerng easy_list stop blocking ads.  Have anyone have this issue and know how to fix it?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on December 05, 2017, 11:20:42 am
sometimes splice all can show SSL errors when web browsing.
I'd say "often" a  lot of SSL errors!  ;)
In my opinion Splice All is not usable in the way it is now (without WPAD).

Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: originalsin on December 20, 2017, 08:34:29 pm
guess it is way too late for asking abbration.

But what does "Tic" stands for?

I can not find out the Tic in Pfsense and can not figure it out.

Plz somebody help me!!!
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on December 21, 2017, 05:12:04 pm
Tick, select.
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: yaska on December 21, 2017, 11:51:37 pm
I have multiple vlans with different IP addresses 

VLAN_50 = 192.168.50.0
VLAN_60 = 192.168.60.0

How do I add all the IPs to the below section

function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.60.0",  "255.255.255.0"))
        return "DIRECT";
 
    return "PROXY 192.168.60.1:3128";
}
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: shan52 on January 05, 2018, 01:43:22 pm
Why is the difference between Squiguard and OpenDNS?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: vielfede on January 08, 2018, 11:09:50 am
Just one thing I can not get working:

Windows 7 updates, W7 detects them, but it's not able to download. I read something about here, but I do not know how to manage it.
https://wiki.squid-cache.org/SquidFaq/WindowsUpdate#Squid_with_SSL-Bump_and_Windows_Updates

Anyone with the same problem?
Solutions? Workarounds?
Thanks in advance
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: shan52 on January 08, 2018, 02:13:53 pm
We have lots of laptop and pc, is there an easier way to install the cert for all the devices?
Title: Re: Guide to filtering web content (http and https) with pfsense 2.3
Post by: aGeekHere on January 08, 2018, 04:28:01 pm
Quote
I have multiple vlans with different IP addresses 
I do not use vlans so I do not know, sorry

Quote
Why is the difference between Squiguard and OpenDNS?
Squiguard block content on rules you set on your router (totally private).
OpenDNS block content on rules you set on their servers (they see all traffic, also can be slower)

Quote
Just one thing I can not get working
Windows updates (as well as Mac and linux) should all be working, if it is not that you may have missed a step. Check that the WPAD is working and your pc is set to auto configure.
If it is still not working read though the guide again, it should work.

Quote
We have lots of laptop and pc, is there an easier way to install the cert for all the devices?
You only have to create a cert of squid when using splice all not for all devices

Hope this helps