pfSense Support Subscription

Author Topic: Guide to filtering web content (http and https) with pfsense 2.3  (Read 68083 times)

0 Members and 3 Guests are viewing this topic.

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #120 on: July 05, 2017, 05:08:45 am »
May I ask though, what is the exact version of pfsense that you are using just for comparison.  If this will not work, then I will need to reinstall my pfsense box, maybe its broken.

I am using:

2.3.4-RELEASE (amd64)
built on Wed May 03 15:13:29 CDT 2017
FreeBSD 10.3-RELEASE-p19


On the other hand, below is the screen shot of my testing NOT USING WPAD but setup the browser proxy.


Also, I tried direct approach that you said, and the result the same error as before.

What exact pfsense version are you using ?
« Last Edit: July 05, 2017, 05:33:25 am by techbee »

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #121 on: July 05, 2017, 08:21:58 am »
2.3.4-RELEASE (amd64)
Something is broken.
Never Fear, A Geek is Here!

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #122 on: July 05, 2017, 08:54:27 am »
2.3.4-RELEASE (amd64)
Something is broken.

what is in your wpad file?

did you install your pfsense or upgraded to 2.3.4 release?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #123 on: July 05, 2017, 10:16:32 am »
wpad is in the guide.

I have been upgrading since i joined.
Never Fear, A Geek is Here!

Offline 4Qman

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #124 on: July 11, 2017, 08:38:01 am »
Hi,

I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

Offline 4Qman

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #125 on: July 11, 2017, 02:39:26 pm »
I have followed this guide.

When I run proxy test it shows that I am under proxy. Google works fine, and so do most websites. But such websites as Amazon won't load, gives error.

Any advice would be greatly appreciated.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #126 on: July 11, 2017, 06:22:37 pm »
Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.
Never Fear, A Geek is Here!

Offline 4Qman

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #127 on: July 12, 2017, 12:07:21 pm »
Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

Hi thanks for your reply.

I followed the guide, page 1.  I copied and pasted logs from PackageSquidGuardLogs


Quote
10.07.2017 17:06:47   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35   [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 17:05:35   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 17:05:35   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 17:05:35   [sg_create_config] Add Default
10.07.2017 17:05:35   [sg_create_config] Add rewrites: safesearch;
10.07.2017 17:05:35   [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 17:05:35   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 17:05:35   [sg_create_config] Add blacklist entries
10.07.2017 17:05:35   [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 17:05:24   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 17:05:24   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 17:05:24   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 17:05:24   [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 17:05:24   [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 17:05:24   [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 17:05:24   [sg_reconfigure_user_db] Add user entries
10.07.2017 17:05:24   [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
10.07.2017 16:36:15   [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 16:36:15   [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 16:36:15   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 16:36:15   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 16:36:15   [sg_create_config] Add Default
10.07.2017 16:36:15   [sg_create_config] Add rewrites: safesearch;
10.07.2017 16:36:15   [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 16:36:15   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 16:36:15   [sg_create_config] Add blacklist entries
10.07.2017 16:36:15   [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 16:36:05   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 16:36:05   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 16:36:05   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 16:36:05   [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 16:36:05   [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 16:36:05   [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 16:36:05   [sg_reconfigure_user_db] Add user entries
10.07.2017 16:36:05   [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
09.07.2017 18:01:06   [squid_reconfigure] Add new redirector options to Squid config.
09.07.2017 18:01:06   [squid_reconfigure] Remove old redirector options from Squid config.
09.07.2017 18:01:06   [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
09.07.2017 18:01:06   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
09.07.2017 18:01:06   [sg_create_config] Add Default
09.07.2017 18:01:06   [sg_create_config] Add rewrites: safesearch;
09.07.2017 18:01:06   [sg_create_config] Add destinations: whitelist; blacklist;
09.07.2017 18:01:06   [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
09.07.2017 18:01:06   [sg_create_config] Add blacklist entries
09.07.2017 18:01:06   [squidguard_rebuild_db] Start rebuild DB.
09.07.2017 18:00:56   [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
09.07.2017 18:00:56   [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
09.07.2017 18:00:56   [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.


Proxy Config
# This file is automatically generated by pfSense
# Do not edit manually !

http_port 192.168.1.1:3128
http_port 127.0.0.1:3128
icp_port 0
digest_generation off
dns_v4_first off
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr drobinson@brightsure.co.uk
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/local/libexec/squid/pinger

logfile_rotate 2
debug_options rotate=2
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  192.168.1.0/24 127.0.0.0/8
forwarded_for on
httpd_suppress_version_string on
uri_whitespace strip


cache_mem 1000 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 1000 MB
cache_dir ufs /var/squid/cache 3000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|?) 0  0%  0
refresh_pattern .    0  20%  4320


#Remote proxies


# Setup some default acls
# ACLs all, manager, localhost, and to_localhost are predefined.
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
acl sslports port 443 563 

acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings


# Custom options before auth


# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc



Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #128 on: July 13, 2017, 07:46:14 pm »
if you set squidguard to allow all do still get the issue?
Never Fear, A Geek is Here!

Offline 4Qman

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #129 on: July 14, 2017, 03:30:28 pm »
if you set squidguard to allow all do still get the issue?

Hi, thank you for the reply.

I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings

At the moment I have Wpad.dat working, I also have MSSL MITM working on splice all, I have android working now (manually entered wpad.dat address).

Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #130 on: July 17, 2017, 06:07:47 pm »
Quote
Is this method the best functioning method, or am i better to setup a websever to to host the wpdad files?  What are the main benifits of setting up a VM hosting these files?
Not sure, maybe if you had a very large number of users you could use another webserver for the wpad.

Quote
I have it sorted now, i didn't go to internet properties/connections/LAN settings - tick automatically detect settings
Even if you left that unticked the transparent proxy should have still connected you, verify that the transparent proxy is working (e.g block port 80 and 443 and use a program that gets updates form the net to check).
Never Fear, A Geek is Here!

Offline rebman77

  • Jr. Member
  • **
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #131 on: September 09, 2017, 09:32:21 am »
Hi,

I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.

Was looking at this guide, https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense and comparing to your screenshot.  What are the implications of unchecking the "invert match" box and not disabling NAT reflection?

Offline bwlinux

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #132 on: September 30, 2017, 01:05:50 pm »
Thanks aGeekHere!   This guide encouraged me to revisit HTTPS filtering for several of my sites.

The WPAD options seems great for PCs but didn't work well on Android devices.  Things get really difficult when you have 2 or more network interfaces you want to proxy/filter using WPAD.

In the end, I just set up transparent proxy for both http and https. Creating a CA on the firewall and then assigning that in Squid for the CA along with "Splice All" for SSL/MITM Mode works great!

I think it's been mentioned before but your guide would really benefit from having 2 completely separate sets of instructions for transparent only and WPAD only.



Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 523
  • Karma: +41/-1
    • View Profile
Re: Guide to filtering web content (http and https) with pfsense 2.3
« Reply #133 on: September 30, 2017, 06:16:17 pm »
I now recommend that you use both, WPAC as the default and transparent with splice all as backup. So software that has no proxy settings still get redirected to the proxy in stead of getting blocked by the firewall rule 80 443. For android you can manually set the proxy, sometimes splice all can show SSL errors when web browsing. I will clean up the guide when I have time.
Never Fear, A Geek is Here!