pfSense Support Subscription

Author Topic: Guide to filtering web content (http and https) with pfsense 2.3  (Read 76279 times)

0 Members and 2 Guests are viewing this topic.

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +11/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #30 on: July 01, 2016, 08:11:24 am »
I'm constantly getting these entries in the Squid Real Time log:

Code: [Select]
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:59488 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:55735 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:49806 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:46365 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:38156 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:25012 - -
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:24866 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:14826 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:10196 - -
01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:6263 - -

192.168.1.5 is my local machine. Is this normal?


Update: Resolved this in another thread..
« Last Edit: July 04, 2016, 04:07:53 pm by AR15USR »
_________________________

Release: pfSense 2.3.4

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #31 on: July 04, 2016, 09:07:38 am »
Quote
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443

Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just  bypassing the autoproxy/wpad stuff?

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #32 on: July 04, 2016, 09:38:07 am »
The Squid Guard settings aren't working for me.  I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and    Default access [all] options there.

On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #33 on: July 04, 2016, 05:59:17 pm »
Quote
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443

Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just  bypassing the autoproxy/wpad stuff?

If port 80 and 443 are lefted open then the user can simply untic the auto configure proxy (on their PC, Mac, phone etc) and set the setting to go direct. This will not call for the wpad and the user will go direct and not use the proxy.

The Squid Guard settings aren't working for me.  I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and    Default access [all] options there.

On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?

1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_package

2. Transperrent proxy does not use a wpad, if you want to use transperrent proxy for http then you need SSL Man In the Middle Filtering for https which must have a certificate installed on evey device.

So you can either set auto configure proxy on all devices or install a certificate on all devices.

You cannot use a transperrent proxy and a wpad at the same time. If you did get a transperrent proxy with SSL mitm working then there is no need for manual proxy mode.
Never Fear, A Geek is Here!

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #34 on: July 05, 2016, 09:24:48 am »
1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_package

That was the problem - there were no other categories than whitelist and default access.  I managed to get the others by enabling "Blackilist" under the general settings of Squid Guard.  Forgive me if I just missed it, but I couldn't see that step in the guide.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #35 on: July 05, 2016, 05:42:36 pm »
I added
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).
Never Fear, A Geek is Here!

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #36 on: July 07, 2016, 09:31:03 am »
I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.
« Last Edit: July 07, 2016, 08:53:17 pm by aGeekHere »
Never Fear, A Geek is Here!

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #37 on: July 15, 2016, 02:14:22 pm »
Just to point out that this approach does not play well with many Roku channels - at least on my Roku 3 anyway.

Forcing youtube to be restricted causes all videos to fail on the Roku app.

The step " Do not allow IP-Addresses in URL" breaks Netflix.

There is also something that breaks the ITV Hub channel - although I haven't figured out what yet.  My experiments suggest that it's something to do with Squid, but probably no squid guard.

The Roku remote is also somewhat less reliable in connecting after a reboot of the Roku, although I'm surprised at that, because I thought the Roku used it's own wifi to connect to the remote independently of the main network.

I'm not saying all this because I'm asking for fixes (although any suggestions that might provide fixes would be useful), but as information for anyone setting this up who has a Roku on their network.

I'm still, largely, using this approach, although, obviously, it's somewhat weaker than it would be if I didn't have to make compromises to get my Roku working.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #38 on: July 15, 2016, 08:18:59 pm »
Bypass the proxy for Roku and Netflix.
Never Fear, A Geek is Here!

Offline wifiuk

  • Jr. Member
  • **
  • Posts: 80
  • Karma: +1/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #39 on: July 24, 2016, 01:32:45 pm »
So i have got wpad all working fine using some other guides, everything goes via the autodisovery and all is good.

That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.


If i set a firewall rule to block 80 - 443 on the LAN it works, and all the devices work except the android devices, which is what i expect to happen.

What i want to do is redirect all 80 - 443 requests to the squid port 3128.

But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.

Can someone advise me the best way to force all android traffic via squid without manually setting up each device..

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #40 on: July 24, 2016, 06:19:49 pm »
Quote
That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.

Manual setup is the only way I got it to work for android (5.1) and for older versions manual setup is the only option.

Try enabling transperrent proxy with the wpad, this will help with port 80 getting blocked.

Quote
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.
I have read that too (never tried) might be your best option.

Or just get your users to setup the proxy?

Never Fear, A Geek is Here!

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1830
  • Karma: +91/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #41 on: July 24, 2016, 11:59:43 pm »
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.

You are not obliged to redirect to LAN IP. Thik about localhost  ;)  then I don't think there is any loop.

What makes this "dual proxy" option difficult is that, in order to set-up transparent proxy, you will have to allow requests ton internet on port 80, while, when configuring explicit proxy, you may (should) want to deny such access so that you ensure everything goes through your proxy.
And same for HTTPS except that transparent proxy, unless you configure ssl-bump, will not intercept HTTPS flow.
Jah Olela Wembo: Les mots se muent en maux quand ils indisposent, agressent ou blessent.

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #42 on: July 30, 2016, 08:25:22 am »
Bypass the proxy for Roku and Netflix.

This fixes the issue with itv hub, although bypassing for netflix.co.uk - works on the roku, but then breaks netflix on Android - presumably they're using different destination ips.

Will have to re-enable ip addresses in the url in squid.

Also, this doesn't fix the problem with youtube - you just have to allow unsafe mode...

Offline everwake

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #43 on: August 03, 2016, 02:29:27 pm »
this guide was very nice, but it does not work for me.

http proxy part works fine (transparent mode)

on android using automatic discovery it uses the wpad file, and looking in the log I can see entries like:

10.111.11.111/android-a41b317d63f30562.local   sb.scorecardresearch.com:443   Request(default/blk_BL_tracker/-) - CONNECT REDIRECT

which implies to me that https is working, but regular expression matching for a site only works if it is http, not over https, which means that filtering using regexp for youtube and google does not work.
no filtering on sites using https seems to be happening.

safe search for youtube only works for desktop-pc, not on youtube apps. the desktop pc's say that safe search was forced by the network admin (which is what I want), but our android pads don't seem to care (rendering safe search totally useless)

I found this looking for a way to filter youtube and google, but since both sites use https it does not work, has anyone got it working or should I go to using certificates?

Regards


Offline dotdash

  • Hero Member
  • *****
  • Posts: 1929
  • Karma: +99/-3
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #44 on: August 03, 2016, 04:52:33 pm »
Just a note, you can simplify the directions by touching the file instead of creating with vi.
e.g.
Ssh into the router
type 8
cd /
cd var/unbound
vi forecegoogle.conf
leave blank for now
save (wq)


Could be done with-
Ssh into the router
type 8
touch /var/unbound/forecegoogle.conf

Or you could just paste the touch command into diagnostics, command prompt and not shell in.
Just a suggestion, there are many ways to do these things.