The pfSense Store

Author Topic: HTTP slow and HTTPS sometimes end up with error page...  (Read 88 times)

Chrismallia and 1 Guest are viewing this topic.

Online sqrobin

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Hi all,


I'm using pfsense 2.4.2-RELEASE-p1 (amd64)

System    VMware Virtual Machine
Netgate Device ID: 0b04cb9c68032f0927c2   
BIOS    Vendor: Phoenix Technologies LTD
Version: 6.00
Release Date: Tue Sep 30 2014
Version    2.4.2-RELEASE-p1 (amd64)
built on Tue Dec 12 13:45:26 CST 2017
FreeBSD 11.1-RELEASE-p6

The system is on the latest version.
Version information updated at Mon Jan 22 8:40:32 WIB 2018 
CPU Type    Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
56 CPUs: 1 package(s) x 56 core(s)
AES-NI CPU Crypto: Yes (inactive)

RAM 32Gb

having HTTPS MIM with splice all mode...
squidguard activated

below is my squid.conf

# This file is automatically generated by pfSense
# Do not edit manually !

http_port x.x.x.x:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

icp_port 0
digest_generation off
dns_v4_first on
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname xxxxxxx
cache_mgr xxxxxxxx
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable on
pinger_program /usr/local/libexec/squid/pinger
sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/squid/lib/ssl_db -M 4MB -b 2048
sslcrtd_children 5
sslproxy_capath /usr/local/share/certs/
sslproxy_options NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS
sslproxy_cert_error allow all
sslproxy_cert_adapt setValidAfter all

logfile_rotate 7
debug_options rotate=7
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  x.x.x.x/29
forwarded_for delete
via off
httpd_suppress_version_string on
uri_whitespace strip

# All Files
######################

refresh_pattern -i (\.|-)(exe|bin|[n|t]ar|acv|[r|j]ar|t?gz|[g|b]z[ip]?2?|7?z[ip]?|zip|wm[v|a]|patch|diff|mar|vpu|inc|r[a|p]m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)[\?.*]?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

#Apple Files
refresh_pattern -i (\.|-)(ap[k|p]|dmg|ip[a|sw]|pkg)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

#Video Audio, Flash
refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
refresh_pattern -i (\.|-)(mp(3|4)|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

#images
refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 override-lastmod reload-into-ims ignore-no-cache ignore-no-store ignore-private ignore-auth refresh-ims

#Office Online
refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 refresh-ims

#Website
refresh_pattern -i (\.|-)(xml|js|jsp|txt|css)(\?.*)?$ 360 40% 1440 refresh-ims
refresh_pattern -i .index.(html|htm)$ 0 40% 1440

cache_mem 15000 MB
maximum_object_size_in_memory 1024000 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 1000 KB
maximum_object_size 100 MB
cache_dir ufs /var/squid/cache 50000 16 256
offline_mode off
cache_swap_low 80
cache_swap_high 90
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
refresh_pattern .    0  20%  4320


#Remote proxies


# Setup some default acls
# ACLs all, manager, localhost, and to_localhost are predefined.
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
acl sslports port 443 563 

acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

# SslBump Peek and Splice
# http://wiki.squid-cache.org/Features/SslPeekAndSplice
# http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
# Match against the current step during ssl_bump evaluation [fast]
# Never matches and should not be used outside the ssl_bump context.
#
# At each SslBump step, Squid evaluates ssl_bump directives to find
# the next bumping action (e.g., peek or splice). Valid SslBump step
# values and the corresponding ssl_bump evaluation moments are:
#   SslBump1: After getting TCP-level and HTTP CONNECT info.
#   SslBump2: After getting TLS Client Hello info.
#   SslBump3: After getting TLS Server Hello info.
# These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that
# they can be used there for custom configuration.
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
acl allowed_subnets src x.x.x.x/8
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings


# Package Integration
url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
url_rewrite_bypass off
url_rewrite_children 16 startup=8 idle=4 concurrency=0

# Custom options before auth


ssl_bump peek step1
ssl_bump splice all
# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow allowed_subnets
http_access allow localnet
# Default block all to be sure
http_access deny allsrc




==============================


I'm having intermittent slowness on HTTP and HTTPS....
sometime HTTPS couldn't load so I need to refresh it again...

here is result from
squidclient -h 127.0.0.1 -p 3128 mgr:info


HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Mon, 22 Jan 2018 02:29:36 GMT
Content-Type: text/plain;charset=utf-8
Expires: Mon, 22 Jan 2018 02:29:36 GMT
Last-Modified: Mon, 22 Jan 2018 02:29:36 GMT
X-Cache: MISS from xxxxxxx
X-Cache-Lookup: MISS from xxxxxxxx:3128
Connection: close

Squid Object Cache: Version 3.5.27
Build Info:
Service Name: squid
Start Time:   Fri, 19 Jan 2018 01:41:33 GMT
Current Time:   Mon, 22 Jan 2018 02:29:36 GMT
Connection information for squid:
   Number of clients accessing cache:   8658
   Number of HTTP requests received:   6244979
   Number of ICP messages received:   0
   Number of ICP messages sent:   0
   Number of queued ICP replies:   0
   Number of HTCP messages received:   0
   Number of HTCP messages sent:   0
   Request failure ratio:    0.00
   Average HTTP requests per minute since start:   1429.7
   Average ICP messages per minute since start:   0.0
   Select loop called: 295699390 times, 0.886 ms avg
Cache information for squid:
   Hits as % of all requests:   5min: 0.1%, 60min: 0.1%
   Hits as % of bytes sent:   5min: 31.2%, 60min: 29.6%
   Memory hits as % of hit requests:   5min: 94.1%, 60min: 83.2%
   Disk hits as % of hit requests:   5min: 0.0%, 60min: 3.8%
   Storage Swap size:   40931384 KB
   Storage Swap capacity:   79.9% used, 20.1% free
   Storage Mem size:   12797904 KB
   Storage Mem capacity:   83.3% used, 16.7% free
   Mean Object Size:   11648.09 KB
   Requests given to unlinkd:   1821
Median Service Times (seconds)  5 min    60 min:
   HTTP Requests (All):   0.76407  0.64968
   Cache Misses:          0.25890  0.22004
   Cache Hits:           274.90301 28.47649
   Near Hits:             0.00000 221.51346
   Not-Modified Replies:  0.00000  0.08729
   DNS Lookups:           0.07284  0.06083
   ICP Queries:           0.00000  0.00000
Resource usage for squid:
   UP Time:   262082.878 seconds
   CPU Time:   57314.937 seconds
   CPU Usage:   21.87%
   CPU Usage, 5 minute avg:   100.00%
   CPU Usage, 60 minute avg:   99.75%
   Maximum Resident Size: 96936368 KB
   Page faults with physical i/o: 46157
Memory accounted for:
   Total accounted:       417220 KB
   memPoolAlloc calls: 767721337
   memPoolFree calls:  816550705
File descriptor usage for squid:
   Maximum number of file descriptors:   942417
   Largest file desc currently in use:   6763
   Number of file desc currently in use: 5758
   Files queued for open:                   0
   Available number of file descriptors: 936659
   Reserved number of file descriptors:   100
   Store Disk files open:                   8
Internal Data Structures:
     6225 StoreEntries
     4183 StoreEntries with MemObjects
     1519 Hot Object Cache Items
     3514 on-disk objects


I have 56 core but I found only 1 CPU utilized by squid with 100% CPU persistently

seems that squid only single threaded...

but I read from https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
saying that pfsense 2.2 or later already use multiple cores...

what I missed on my configuration....


last pid: 41316;  load averages:  1.12,  1.20,  1.22                                                                                            up 8+12:07:37  09:56:43
114 processes: 3 running, 111 sleeping
CPU:  1.7% user,  0.1% nice,  0.4% system,  0.1% interrupt, 97.6% idle
Mem: 10G Active, 7186M Inact, 11G Laundry, 2491M Wired, 1571M Buf, 513M Free
Swap: 4096M Total, 251M Used, 3845M Free, 6% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
13212 squid         1 103    0 23729M 22535M CPU46  46 982:21 100.59% squid
11539 root          1  52    0   261M 22536K accept 23   0:00   1.30% php-fpm
87784 root          1  52   20 13084K  2156K wait   35   1:55   0.26% sh
 8091 root          1  20    0 12700K  1888K bpf    28   7:47   0.20% filterlog
 1622 squid         1  20    0 34124K 14152K sbwait 22   0:17   0.18% squidGuard
 9237 squid         1  20    0 34124K 14148K sbwait  1   0:16   0.15% squidGuard
11166 squid         1  20    0 34124K 14156K sbwait 17   0:10   0.15% squidGuard
15785 squid         1  20    0 34124K 14152K sbwait 54   0:09   0.15% squidGuard
44726 root          1  20    0 10484K  1984K select 51   3:37   0.10% syslogd
21613 root          1  20    0 20060K  3704K CPU33  33   0:00   0.08% top
16266 squid         1  20    0 34124K 14152K sbwait  4   0:07   0.07% squidGuard
87330 root          1  20    0 37712K  7044K kqread  2   0:32   0.07% nginx
22108 squid         1  20    0 34124K 14156K sbwait 12   0:06   0.05% squidGuard
22534 squid         1  20    0 34124K 14152K sbwait 53   0:05   0.04% squidGuard
61005 squid         1  20    0 33780K  3520K select 49   0:50   0.03% pinger
26037 squid         1  20    0 33780K  3512K select 30   0:47   0.02% pinger
36541 squid         1  20    0 33780K  3592K select 52   0:08   0.02% pinger
32991 squid         1  20    0 33780K  3512K select 36   0:48   0.02% pinger
 8509 squid         1  20    0 33780K  3520K select 41   0:43   0.02% pinger
25623 squid         1  20    0 33780K  3512K select 33   0:50   0.02% pinger
66183 squid         1  20    0 33780K  2940K select 28   0:47   0.02% pinger
29798 squid         1  20    0 34124K 14148K sbwait 55   0:04   0.02% squidGuard
18928 squid         1  20    0 33780K  2940K select 31   0:45   0.02% pinger
51648 squid         1  20    0 33780K  3512K select 19   0:47   0.02% pinger
30062 squid         1  20    0 33780K  3852K select 20   0:03   0.02% pinger
62063 squid         1  20    0 33780K  3512K select  2   0:49   0.02% pinger
65590 squid         1  20    0 33780K  3512K select 22   0:49   0.02% pinger
42315 squid         1  20    0 34124K 14148K sbwait 17   0:04   0.02% squidGuard
80972 squid         1  20    0 33780K  3568K select  9   0:26   0.02% pinger
20730 squid         1  20    0 33780K  3520K select  3   0:47   0.02% pinger
75460 squid         1  20    0 33780K  2944K select  6   0:47   0.02% pinger
66930 root          5  52    0 13032K  2060K uwait   7   1:54   0.02% dpinger
89505 squid         1  20    0 33780K  2936K select  8   0:47   0.01% pinger
63016 squid         1  20    0 33780K  2944K select 21   0:48   0.01% pinger
28848 squid         1  20    0 33780K  2940K select  9   0:46   0.01% pinger
66070 root          5  52    0 13032K  2012K uwait  20   1:54   0.01% dpinger
66431 root          5  52    0 10984K  2016K uwait  20   1:55   0.01% dpinger
  336 root          1  20    0  9560K   488K select 55   0:30   0.01% devd
88531 root          1  20    0 78844K  7128K select 38   0:00   0.01% sshd
25055 root          1  20    0 24612K 12432K select 10   0:34   0.00% ntpd
 5913 root          1  20    0 43140K  5428K kqread 29   0:10   0.00% lighttpd_ls

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2305
  • Karma: +174/-9
    • View Profile
Re: HTTP slow and HTTPS sometimes end up with error page...
« Reply #1 on: Today at 01:10:34 am »
Hi,

Why 56 cores ??

I miss something : is this squid related or not ? I mean, when you disable squid, the problem is solved - no more problems ??

I advise you to post and read here : pfSense Forum pfSense English Support Packages Cache/Proxy


Note : not related but strange :
Code: [Select]
AES-NI CPU Crypto: Yes (inactive)

Online sqrobin

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: HTTP slow and HTTPS sometimes end up with error page...
« Reply #2 on: Today at 01:35:42 am »
yes this is squid related as PFSense rely with squid to perform cache and filtering...


yes if squid turn off it will fix the issue...

now I turn off the HTTPS MITM... and CPU usage lower a bit... but still high.....

for the AES-NI CPU, i think its because my hardware support it but by configuration is not selected using AES-NI as I'm not yet in the phase using VPN...

 


Online sqrobin

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: HTTP slow and HTTPS sometimes end up with error page...
« Reply #3 on: Today at 01:38:31 am »
Hi Gertjan

as Why 56 Cores... 
as simple that I have the resource and I thought that I can limit it or change it later from VM....


Online Chrismallia

  • Full Member
  • ***
  • Posts: 274
  • Karma: +20/-4
    • View Profile
Re: HTTP slow and HTTPS sometimes end up with error page...
« Reply #4 on: Today at 05:36:38 am »
yes this is squid related as PFSense rely with squid to perform cache and filtering...


yes if squid turn off it will fix the issue...

now I turn off the HTTPS MITM... and CPU usage lower a bit... but still high.....

for the AES-NI CPU, i think its because my hardware support it but by configuration is not selected using AES-NI as I'm not yet in the phase using VPN...

 

MITM = Huge can of worms +  many apps/devices are having hard certs so you can not use your own, if you want to filter I would try PfblockNG, As for caching with todays bigger pipes and dynamic content there is not much use for it

Online sqrobin

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: HTTP slow and HTTPS sometimes end up with error page...
« Reply #5 on: Today at 05:54:49 am »
HI Chris,

Thanks for replying...

Could you please let me know why I should go with PfBlockNG rather than SquidGuard...

as I ready, PFBlockNG is used if I host mail server and this will prevent IP Block Country that is known as spammer to reach our server....

if I compare to SquidGuard, its different of purpose..... even though you can put the filter on the outbound from your internal LAN....

So, anyone can give me a clue as why I have almost 100% CPU utilisation persistently on 1 CPU rather then spread into multiple CPU?

CPU usage information on the Dashboard is useless  as its represent to all CPU I have... since I have many.. then if 1 CPU is high the CPU Dashboard info doesn't tell me anything..