Netgate SG-1000 microFirewall

Author Topic: Guide to filtering web content (http and https) with pfsense 2.3  (Read 70619 times)

0 Members and 2 Guests are viewing this topic.

Offline jadog

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #60 on: September 02, 2016, 08:21:54 am »
Thanks for taking the time to post this excellent guide! Just a few additions I would recommend adding.

YouTube mobile still allows unrestricted access. Including the below corrected it for me.

Code: [Select]
Click add under Host overrides
Host = m
Domain = youtube.com
IP =  216.239.38.120
Description = youtube mobile
Save

Also when using vi after you ssh into your router, using wq does not allow saving. You need to use ":wq" (without the quotes). All working perfectly for me!

Offline eixel05

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #61 on: September 06, 2016, 01:06:35 am »
Hi,

First of all thank you so much for this guide, just what I've been looking for!

Before this guide I was using Transparent+MITM approach but because some HTTPS sites doesn't load as they should (I'm not sure if the cause was indeed MITM but after turning it off the sites loads fine)

I followed this guide thoroughly, HTTP filtering works fine but I don't know if HTTPS filtering is working? Before, I am using the tail command (/var/squid/logs/access.log) and while MITM is on, https site access is being logged but now with this approach, not a single HTTPS is being logged.

Is there another way to check if indeed this approach is working for HTTPS?

Out-of-topic question, since my main purpose for this Proxy server is local cache, Are HTTPS objects being cache as well? or only HTTP?


Update (09/07/2016): NVM, I figured it out :D all is working great, thank you so much for this guide!

I'M REALLY SORRY FOR BEING NOOB, I just started my way to "Networking", please be patient with me.
« Last Edit: September 06, 2016, 09:03:27 pm by eixel05 »

Offline Maxim_Al

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #62 on: September 15, 2016, 10:58:54 pm »
Can anybody help me?
I test  pfsense for gateway/proxy for a company. But I have one strange trouble:
The pfsense is installed and work but if I use it for access (through proxy) to https://code.getmdl.io/1.2.1/material.grey-orange.min.css  I have in proxy log:
Date    IP    Status    Address    User    Destination
16.09.2016 14:37:28    192.168.1.222    TAG_NONE/503    code.getmdl.io:443    -    -
What is it an how I must to do that resolve it?


UPDATE 28/10/2016 - This was provider :) not pfsense!
« Last Edit: October 27, 2016, 08:25:43 pm by Maxim_Al »

Offline klmiciano

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #63 on: October 27, 2016, 08:20:08 pm »
Which rule should be above between these two?

FIRST -- 
"Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save"

SECOND --
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save"



Offline klmiciano

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #64 on: October 27, 2016, 08:36:38 pm »
I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.



How to do this?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #65 on: October 29, 2016, 03:21:55 am »
Which rule should be above between these two?

FIRST -- 
"Now we are going to create a rule that will force the network to use our route as the DNS server.
In Firewall/NAT/Port forward
add a new rule

Interface = LAN
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save"

SECOND --
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save"
first one on top

I found this post and I am researching to see how well it will work.

https://forum.pfsense.org/index.php?topic=106016.0

The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.

Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.

So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.

How to do this?

after you have the wpad setup and working enable transperrent proxy in squid
Never Fear, A Geek is Here!

Offline phunni

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #66 on: November 24, 2016, 11:08:19 am »
If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?

I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there...

Offline jbourn1907

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #67 on: November 25, 2016, 02:45:01 am »
Sorry I'm new in pfsense.

I have an internal DNS server windows server 2008. How can I blocked internal computers to direct access to http. I want to filter it through pfsense firewall. Can you give me some screen shots or sample of rules and where can I put this?

I want all computers to go first in pfsense proxy before connecting to internet. I already configure transparent proxy.

Thanks for the help.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #68 on: November 25, 2016, 07:04:57 pm »
"How can I blocked internal computers to direct access to http"
read the part in the guide about blocking port 80 and 443 and instead of using pfsense as your DNS server use your internal one, (for more help on this issue ask in the Cache/Proxy forum).

"If I configure a VPN, I assume this will all still work? i.e. the filtering is done post decryption?....I'm not using the wpad portion of this, so I've ignore d what it says about VPNs in there..."
If the transparent proxy has issue with the vpn you may need to bypass it.

Never Fear, A Geek is Here!

Offline genesislubrigas

  • Jr. Member
  • **
  • Posts: 72
  • Karma: +2/-3
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #69 on: December 16, 2016, 06:19:24 am »
thanks for this tutorial.  i am not using dns resolver but bind.  can you also show how to configure bind.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #70 on: December 20, 2016, 09:39:07 pm »
Have not used bind, but after a quick google try http://blog.muhammadattique.com/configuring-bind-dns-server-on-pfsense-firewall/

If it does not help ask in the main forum
Never Fear, A Geek is Here!

Offline nib01

  • Newbie
  • *
  • Posts: 22
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #71 on: December 23, 2016, 02:46:29 pm »
"To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save

Set your system to automatically detect settings (for windows it is in internet options connections lan settings).

You also have to set up the proxy setting for each program that cant connect (firefox, graphics drive software, vlc etc)

If you have programs that cannot connect and have no proxy setting you need to setup a firewall aliases
 Firewall/Aliases/IP
and add the destination server ip (use wire shark to help find the blocked Ips or in your firewall block rule enable Log packets that are handled by this rule, use http://ip-lookup.net/index.php to check what it is and add to the Aliases. If it is part of a domain add the domain)
now create a new firewall lan rule
IPv4 TCP * * * passAliases 80- 443 * pass rule.

Save"

Not sure where to setup/configure above firewall lan rule.

Offline rsnsmh

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #72 on: December 30, 2016, 12:40:42 am »
Thank you for the guide - i got everything working as expected. However, I need to enable content filter on a 2nd interface (Guest). What sort of config addition is needed to enable content filtering on the 2nd additional interface.   

Offline bole5

  • Newbie
  • *
  • Posts: 11
  • Karma: +7/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #73 on: February 20, 2017, 11:40:26 am »
In the Part 3 you refer to *.local in proxy.pac file as well as unbound configuration.

Code: [Select]
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

When user sets up pfSense the domain defaults to "localdomain" in System/General Setup and there is explicit sentence about not using .local as a domain name:
Code: [Select]
Do not use 'local' as a domain name. It will cause local hosts running mDNS (avahi, bonjour, etc.) to be unable to resolve local hosts not running mDNS.

Do we need to modify the domain to "local" in System/General Setup or should we replace "*.local" with "*.localdomain"?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #74 on: February 20, 2017, 06:41:56 pm »
Use a domain like this
pfsense.thisismydomain.local

Do not use
pfsense.local.local

The PAC does not need changing
Never Fear, A Geek is Here!