Netgate SG-1000 microFirewall

Author Topic: Guide to filtering web content (http and https) with pfsense 2.3  (Read 91376 times)

0 Members and 4 Guests are viewing this topic.

Offline vielfede

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #90 on: June 22, 2017, 10:27:17 am »
The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.
Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive          Advantages/features                                                                    Disadvantage
Splice (all)         No needs of certificate installation on Clients + webfiltering              No traffic analysis i.e no AntiVirus
Bump (MITM)    Traffic analysis i.e noYES AntiVirus     + webfiltering                               Needs to install certs on clients


Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

 
« Last Edit: June 23, 2017, 02:56:59 am by vielfede »

Offline AR15USR

  • Full Member
  • ***
  • Posts: 268
  • Karma: +11/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #91 on: June 22, 2017, 12:08:45 pm »
The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.
Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive          Advantages/features                                                                    Disadvantage
Splice (all)         No needs of certificate installation on Clients + webfiltering              No traffic analysis i.e no AntiVirus
Bump (MITM)    Traffic analysis i.e no AntiVirus     + webfiltering                               Needs to install certs on clients


Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

Shouldn't the second one not have the word "no"..
_________________________

Release: pfSense 2.3.4

Offline vielfede

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +2/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #92 on: June 23, 2017, 02:58:43 am »
Shouldn't the second one not have the word "no"..
Sorry cut & paste mistake, I fixed it.

Offline mahnonsaprei

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #93 on: June 29, 2017, 04:59:44 pm »
Hi at all,

I'm Marcello from italy. Nice to meet you, i'm a newbie of pfsense.

I have read the tutorial, but i have one problem.
When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

I'm looking for in google but i not found the answer.

Please help me.
Thank you veeeeery much!!!

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 535
  • Karma: +46/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #94 on: June 29, 2017, 07:10:59 pm »
Quote
When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

Known issue with squid, I do not think there is a fix for it.
Never Fear, A Geek is Here!

Offline mahnonsaprei

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #95 on: June 30, 2017, 09:08:53 am »
Thank you so much. There aren't alternatives to redirect https via proxy also blank page or other?

B.r.

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #96 on: July 03, 2017, 04:45:50 am »
hello ageekhere,

can you update your guide now.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 535
  • Karma: +46/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #97 on: July 03, 2017, 08:44:22 am »
what part are you stuck on?
Never Fear, A Geek is Here!

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #98 on: July 03, 2017, 09:03:58 am »
1. I dont know what exact firewall rules to add
2. I am confused if I need to install unofficial wpad package of marcelloc.
3. I am confused if there is a need to have separate webserver of wpad.
4. I am confused if I can enable both transparent and ssl mitm filtering on squid.
5. I cant nslookup wpad on clients command prompt to resolve to my pfsense box ip.


Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 535
  • Karma: +46/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #99 on: July 03, 2017, 10:15:15 pm »
1.
The firewall rules that you need are
Code: [Select]
IPv4+6 TCP * * * 80 (HTTP) * none port 80 block
IPv4+6 TCP * * * 443 (HTTPS) * none port 443 block 
This blocks access to port 80 and 443, it stops users from bypassing the proxy.

Also if you want to force pfsense to be the DNS server use.

Code: [Select]
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

Have a look through a few of these videos https://www.youtube.com/playlist?list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk
There are a few really good configs he does there.


2. Wait for the package to become stable and available in the package manager, in the mean time you can use the pfsense webserver, put the wpad files in /usr/local/www/ read Part 3. (all that needs to be done is add the wpad files (wpad.da, proxy.pac and wpad.dat) to /usr/local/www/).

3. Just use pfsense webserver /usr/local/www/ (I have not used an external webserver)

4. This is what I did, I set up a WPAD and transparent proxy with ssl mitm (splice all so you do not have to install cert on every device) and set your device to auto configure. I found that just using WPAD a lot of programs that cannot use the proxy were getting blocked so the transparent proxy catches that and something the transparent proxy ssl has issues (like windows updates) however the WPAD is used for that. Best of both worlds.

5 I have had issues as well, if you put http://192.168.1.1/wpad.dat in your web browser and it downloads than it is working.

This guide goes a few steps fever by forcing google and bing to be in safe mode (even youtube).

So what I would do is get WPAD working then enable the transparent proxy with ssl mitm (create a cert for the router).

Hope this helps
Never Fear, A Geek is Here!

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #100 on: July 03, 2017, 11:11:51 pm »
thanks for the reply aGeekHere.

What is the tcp port that you used on your webconfigurator ?  If I block port 80, then, I will also be blocked opening the pfsense gui.

Also, regarding to my question number 4, do you mean that you did not enable the http transparent proxy on port 3128 but only the HTTPS/SSL MITM on port 3129 and setup wpad. did I understood you correctly ?
« Last Edit: July 03, 2017, 11:19:12 pm by techbee »

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 535
  • Karma: +46/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #101 on: July 03, 2017, 11:58:57 pm »
Quote
What is the tcp port that you used on your webconfigurator ?


set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

Quote
If I block port 80, then, I will also be blocked opening the pfsense gui.
No it should not (have the rule above the anti lockout rule);

Quote
Also, regarding to my question number 4
Enable squid proxy 3128
Get WPAD working follow guide

Enable transparent mode to forward all requests for destination port 80 to the proxy server.

Enable SSL filtering
Splice all
create Certificate (for pfsense).

Summery
Set Advanced: Admin Access Protocol to http
Enable squid proxy
Follow guide on setting up WPAD
Enable transparent mode
Enable SSL filtering Splice all
Never Fear, A Geek is Here!

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #102 on: July 04, 2017, 12:09:24 am »
Yep, done that all.

I can download the wpad files from the browser. I can access the http sites but still cannot access the https sites.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 535
  • Karma: +46/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #103 on: July 04, 2017, 12:22:17 am »
what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?



Never Fear, A Geek is Here!

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #104 on: July 04, 2017, 12:41:17 am »
what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?
'

Hi aGeekHere,

Yes, cache is cleared,
os=win7 pro sp1,
I disabled the update coz I dont need it,
autoconfigure is set,
yes, google and yahoo https sites. facebook and youtube does not load and shows error.

I use splice-all.  Don't I really need to install certs to client device ?

If https site is successfully blocked, what error page do I expect to see ?  Can you show me the screenshot for the error page that will show !