Netgate SG-1000 microFirewall

Author Topic: Guide to filtering web content (http and https) with pfsense 2.3  (Read 70819 times)

0 Members and 1 Guest are viewing this topic.

Offline oddworld19

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +1/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #45 on: August 04, 2016, 08:06:38 am »
I have followed the guide, WPAD is working and the proxy is working in NON-transparent mode. I want to block specific subreddits, like Reddit.com/r/nsfw without blocking the remainder of Reddit.com. I do not want to block the entire domain, and I use pfblockerng and DNSBL with unbound to handle DNS and Top level domain blocking. I don't want to block all of Reddit... Just the NSFW material (and I have a list of all websites in squidguard).

At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.
« Last Edit: August 04, 2016, 08:09:40 am by oddworld19 »
Supermicro SYS-5018A-FTN4 (Atom c2758)
pfSense 2.3.2

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1806
  • Karma: +89/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #46 on: August 04, 2016, 08:19:28 am »
At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.

No this will not work without MITM because CONNECT method on which Squidguard relies in order to block HTTPS knows only the left part of your URL, meaning here http://reddit.com
This part is sent in clear text but then everything else is within HTTPS tunnel thus this can't be read therefore blocked.

Offline oddworld19

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +1/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #47 on: August 04, 2016, 08:42:16 am »
Since that is literally the only site I'm trying to block on squid (and everything else is blocked using DNSBL / pfblockerng) how would you suggest I intercept HTTPS?

Do you think I could alter the WPAD rules to PROXY reddit.com and default to pass/DIRECT all other traffic? Then on squid, I enable MITM/SSL. I shouldn't use transparent mode, right, because WPAD would notify users of the proxy?

Would I use port 3128 or 3129 (I'm a little confused about the separate SSL port in the options).

Since this isn't a banking site or google site, any idea if I could configure squid such that no certs are needed on the host computers?
Supermicro SYS-5018A-FTN4 (Atom c2758)
pfSense 2.3.2

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1806
  • Karma: +89/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #48 on: August 04, 2016, 09:52:35 am »
I can't really answer  :-[

To me DNSBL doesn't aim at replacing HTTP filtering. These are on 2 different layers. If you implement DNSBL, then you will access less sites, that's it (and this is already not so bad) but HTTP proxy permits to filer at fqdn level (like DNSBL does) but also in URL (for HTTP or HTTPS is MITM) and, on top of that, to check for virus (again HTTP and also HTTPS if MITM)

Then decision to deploy proxy in transparent mode or not is entirely on your side. I'm definitely not prone to deploy transparent proxy but can understand that is some cases it may help.

Offline oddworld19

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +1/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #49 on: August 04, 2016, 10:18:12 am »
Thanks. Yes - I use DNSBL just to block at the DNS level. pfblockerng has a new update where it also includes top level domains. It does a pretty good job of blocking content at the "domain-level" via DNS. I then set-up rules in pfsense to require the network to use pfsense's DNS unbound server. I agree these are two fundamentally different technologies. However, the DNS blacklist does not cover quite everything I intend to block. I want reddit.com to work, but want to block certain sub-reddits. Obviously, DNS is the wrong tool for this job, which brought me to squid.

I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.

Does SSL filtering work in non-transparent mode?

I assume so. Is the port still 3128?



Supermicro SYS-5018A-FTN4 (Atom c2758)
pfSense 2.3.2

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1806
  • Karma: +89/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #50 on: August 04, 2016, 10:22:36 am »
I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.
Does SSL filtering work in non-transparent mode?
I assume so. Is the port still 3128?

yes and yes.
And BTW you don"t need to edit proxy.pac in order to go direct for all sites but reddit.
You can still use proxy for all and only block reddit content.
Other HTTP sites will benefit from cache and all sites will potentially benefit from anti-virus if one day you enable it.

Offline oddworld19

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +1/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #51 on: August 04, 2016, 10:28:29 am »
Thanks. Right. My only thought there was:

I understand SSL decryption and inspection gets.... Nasty.

For example, if browsers require HSTS or whatever other protocols check certificates, then it might get annoying to keep installing certs on all on my computers on the LAN. For example... A small FTP server has no need to access Reddit, but I don't know if SSL bumping will cause issues with package updates in the future (without a cert).

Maybe it will... Maybe it won't. I don't know... Do you think it would potentially be an annoyance if I don't have certs installed?
Supermicro SYS-5018A-FTN4 (Atom c2758)
pfSense 2.3.2

Offline maverik1

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #52 on: August 17, 2016, 12:03:31 am »
Lots of good info! Thanks!!

Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1806
  • Karma: +89/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #53 on: August 17, 2016, 12:31:43 am »
Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box?  I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard.  Cannot for the life of me understand what is blocking notifications. Very annoying.


What "notification" are you speaking about?
I'm confused  :-[

Offline maverik1

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #54 on: August 17, 2016, 07:36:35 pm »
I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.


Offline maverik1

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #55 on: August 17, 2016, 07:39:59 pm »


Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.


How can we get this working with mobile devices Android and iOS that use the youtube mobile app or m.youtube.com?

Offline chris4916

  • Hero Member
  • *****
  • Posts: 1806
  • Karma: +89/-11
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #56 on: August 18, 2016, 02:49:33 am »
I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)

Offline maverik1

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #57 on: August 18, 2016, 12:01:34 pm »
Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members  ;)

Push notifications are alerts that are received by cellular phones;mostly smart phones, for different types of apps such as social media, email, games and whatnot. I have no issues receiving these alerts when using data only. However, when I was on my wifi network they were not coming through. I spend a few weeks working with the firewall rules and proxy settings trying everything I could think of to troubleshoot the issue. Turns out the problem was with the phone and not pfsense.

Online kpa

  • Hero Member
  • *****
  • Posts: 1186
  • Karma: +131/-6
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #58 on: August 25, 2016, 01:39:12 pm »
I'd be very surprised if all those "push" system weren't all emulated by polling a server periodically, that kind of set up needs absolutely nothing else but an outgoing connection from the device to the server. The other option would be a service running on the device and the notifications would be then really pushed on the device by an incoming connection from the server to the device. I don't think you'd want implement it like that however  ;)

Offline laurenzzo

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #59 on: August 27, 2016, 07:43:36 am »
[...]

First we are going to setup squidguard
Update
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).


In Package/Proxy filter SquidGuard: General settings/General settings
click blacklist
enter http://www.shallalist.de/Downloads/shallalist.tar.gz
download
wait to finish

[...]

Hi aGeekHere,

Thanks a lot for your guide !
I tried to follow your steps to get my blacklist downloaded but without success.

Could you please have a look on my post for more details ?

https://forum.pfsense.org/index.php?topic=117314.msg649961#msg649961

I really don't understand why the download doesn't work...

Thanks in advance for any help !

EDIT : SOLVED !
this was an issue with Firefox...
Could you please update your nice guide with a small note ? (e.g. "don't use Firefox to download the blacklist, IE do the job correctly")

Thanks !

Kind regards,
Laurenzzo
« Last Edit: August 30, 2016, 03:11:09 am by laurenzzo »