pfSense Gold Subscription

Author Topic: squid3 only reports TCP_MISS, never TCP_HIT  (Read 9749 times)

0 Members and 1 Guest are viewing this topic.

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
squid3 only reports TCP_MISS, never TCP_HIT
« on: March 02, 2015, 06:40:28 am »
Hi!

I recently installed squid3 (pfSense 2.2).

I've been following some guides, tweaks etc, but when I look at the logs, and Realtime (Servces->Proxy Server->Realtime), all requests are TCP_MISS...Not _one_ single TCP_HIT.

This is how it is configurated at the moment:
General
Proxy Interfaces: Bridge (Lan+Wifi)
Allow users on interface: On

Transparent Proxy interface(s): Bridge (Lan+Wifi)
Transparent Proxy: On

Local cache
Low-water-mark in %: 90
High-water-mark in %: 95
Enable offline mode: Off
Hard disk cache size: 5000 (mb)
Hard disk cache system: UFS
Minimum object size: 0
Maximum object size: 512000 (kb)

Memory cache size: 0
Maximum object size in RAM: 32 (kb)
Memory replacement policy: Head GDSF
Cache Dynamic Content: On

All the other stuff available I havent touched at all, such as: Antivirus, ACL, Auth, Traffic MGMT etc.

Help would be much appreciated.


This is a sample of /var/squid/logs/access.log

Code: [Select]
425297689.414  10051 192.168.0.10 TCP_MISS/200 1435 GET http://svl4.bredbandskollen.se/ulinfo/3043210925.txt - ORIGINAL_DST/192.36.135.235 text/plain
1425297689.986    104 192.168.0.10 TCP_MISS/200 457 POST http://www.bredbandskollen.se/api.php - ORIGINAL_DST/5.178.76.98 text/html
1425297691.343   1265 192.168.0.10 TCP_MISS/200 1405 GET http://www.bredbandskollen.se/api.php? - ORIGINAL_DST/5.178.76.98 text/xml
1425297691.353   1276 192.168.0.10 TCP_MISS/200 506 GET http://www.google-analytics.com/__utm.gif? - ORIGINAL_DST/77.53.0.154 image/gif
1425297691.384     12 192.168.0.10 TCP_MISS/200 506 GET http://www.google-analytics.com/__utm.gif? - ORIGINAL_DST/77.53.0.154 image/gif
1425297691.417     18 192.168.0.10 TCP_MISS/200 575 GET http://www.bredbandskollen.se/api.php? - ORIGINAL_DST/5.178.76.98 text/xml
1425297692.391  12969 192.168.0.10 TCP_MISS_ABORTED/000 0 POST http://svl4.bredbandskollen.se/cgi/upload.cgi? - ORIGINAL_DST/192.36.135.235 -
1425297692.708  13309 192.168.0.10 TCP_MISS_ABORTED/000 0 POST http://svl4.bredbandskollen.se/cgi/upload.cgi? - ORIGINAL_DST/192.36.135.235 -
1425297697.306  15831 192.168.0.10 TCP_MISS/200 339 GET http://svl4.bredbandskollen.se/huinfo/3043210925.txt? - ORIGINAL_DST/192.36.135.235 text/plain
1425297698.037    109 192.168.0.10 TCP_MISS/200 43829 GET http://www.aftonbladet.se/ - ORIGINAL_DST/144.63.250.5 text/html
1425297698.239     76 192.168.0.10 TCP_MISS/200 11944 GET http://www.svd.se/Bilder/Artikelbilder/4371071.svd/binary/a/ab-volvo.jpg - ORIGINAL_DST/144.63.252.10 image/jpeg
1425297698.239     77 192.168.0.10 TCP_MISS/200 4580 GET http://www.svd.se/Bilder/Artikelbilder/4374735.svd/binary/a/sasen.jpg - ORIGINAL_DST/144.63.252.10 image/jpeg
1425297698.282     97 192.168.0.10 TCP_MISS/200 2740 GET http://gfx.aftonbladet-cdn.se/image/20395294/90/fourByThree/bcf1ec93c88b6/ccb8b687-72cc-46b3-9d57-97240362fa44 - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.282     97 192.168.0.10 TCP_MISS/200 2634 GET http://gfx.aftonbladet-cdn.se/image/20398521/90/fourByThree/b41f6d2c30187/917d6bd9-2e7c-42a6-912e-56b8604d28ca - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.282     97 192.168.0.10 TCP_MISS/200 3350 GET http://gfx.aftonbladet-cdn.se/image/20397871/90/fourByThree/ce3cbafa23f78/saskastrup.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.282     97 192.168.0.10 TCP_MISS/200 1873 GET http://gfx.aftonbladet-cdn.se/image/20340566/90/fourByThree/5e23f894626f1/Norwegian-B787-Dreamliner.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.282     97 192.168.0.10 TCP_MISS/200 3022 GET http://gfx.aftonbladet-cdn.se/image/20398488/90/fourByThree/6e9d0823f1227/d745cce8-e54b-4afe-b8dc-0bed11ce33b3 - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.283     97 192.168.0.10 TCP_MISS/200 5371 GET http://gfx.aftonbladet-cdn.se/image/20397871/133/fourByThree/09bf5af2f8f78/saskastrup.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.283     97 192.168.0.10 TCP_MISS/200 7224 GET http://gfx.aftonbladet-cdn.se/image/20399180/315/imageColumnDouble/c85630966175b/68792-a03382fd1617eb0e0044fec87692be7e401fae8b.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.283     97 192.168.0.10 TCP_MISS/200 12365 GET http://gfx2.aftonbladet-cdn.se/image/20400044/485/normal/3adce41c36813/duscha.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.283     97 192.168.0.10 TCP_MISS/200 14586 GET http://gfx2.aftonbladet-cdn.se/image/20400499/485/imageColumnDouble/e29b8aa4fe0b2/hahnepuff.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.283     98 192.168.0.10 TCP_MISS/200 19474 GET http://gfx2.aftonbladet-cdn.se/image/20400420/475/normal/c28476356e2a4/blavitt.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.287    102 192.168.0.10 TCP_MISS/200 39910 GET http://gfx.aftonbladet-cdn.se/image/19911491/655/normal/b5dd20d77b6a9/asp.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.309     90 192.168.0.10 TCP_MISS/200 338 GET http://wwwapp.aftonbladet.se/eae-logger/Logger? - ORIGINAL_DST/192.71.240.35 image/gif
1425297698.375     27 192.168.0.10 TCP_MISS/200 3559 GET http://gfx.aftonbladet-cdn.se/image/20398152/90/fourByThree/11838d90acef2/SK%C3%84RMAVBILD-2015-03-01-KL.-15.44.15.jpg - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.408    137 192.168.0.10 TCP_MISS/200 8777 GET http://gfx.omni-cdn.se/images/62da823f-2224-4571-81f1-e650ba96543f? - ORIGINAL_DST/193.240.120.254 image/jpeg
1425297698.789    139 192.168.0.10 TCP_REFRESH_UNMODIFIED/200 16812 GET http://cdn.mxpnl.com/libs/mixpanel-2.2.min.js - ORIGINAL_DST/23.46.124.19 application/x-javascript
1425297699.024    121 192.168.0.10 TCP_MISS/200 523 GET http://api.mixpanel.com/track/? - ORIGINAL_DST/159.122.4.74 application/json
1425297699.045   8969 192.168.0.10 TCP_MISS_ABORTED/000 0 GET http://www.bredbandskollen.se/api.php? - ORIGINAL_DST/5.178.76.98 -
1425297699.627  20108 192.168.0.10 TCP_MISS/200 350 GET http://priscilla.lrfmedia.se/socket.io/1/xhr-polling/d-7j4zW-Qt0OZ4Ctq-6w? - ORIGINAL_DST/185.20.14.15 text/plain
1425297700.114  20741 192.168.0.10 TCP_MISS/502 3470 POST http://svl4.bredbandskollen.se/cgi/upload.cgi? - ORIGINAL_DST/192.36.135.235 text/html
1425297700.117  20659 192.168.0.10 TCP_MISS/502 3470 POST http://svl4.bredbandskollen.se/cgi/upload.cgi? - ORIGINAL_DST/192.36.135.235 text/html
1425297700.445    999 192.168.0.10 TCP_MISS/200 1515 GET http://gfx.aftonbladet-cdn.se/assets/gfx/streamers/extra/extra-655.gif? - ORIGINAL_DST/193.240.120.254 image/gif

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #1 on: March 02, 2015, 08:25:35 am »
Squid as a caching platform is of limited use these days due to the dynamic nature of the web.  My tests show that my hit rate is in the area of 3-7%.  Not very good, but I use Squid more as the base for filtering with SquidGuard.

SSH in and run this command:

Code: [Select]
squidclient -h pfSense_LAN_IP_Address -p 3128 mgr:info
and look for Cache information for Squid

Her eis a sample of my output:

Cache information for squid:
        Hits as % of all requests:      5min: 0.7%, 60min: 1.1%
        Hits as % of bytes sent:        5min: 67.8%, 60min: 51.4%
        Memory hits as % of hit requests:       5min: 0.0%, 60min: 12.5%
        Disk hits as % of hit requests: 5min: 62.5%, 60min: 22.0%
        Storage Swap size:      459662 KB
        Storage Swap capacity:  89.8% used, 10.2% free
        Storage Mem size:       87208 KB
        Storage Mem capacity:   34.3% used, 65.7% free
        Mean Object Size:       42.29 KB
        Requests given to unlinkd:      255644


1% hit rate over the past hour.  1%.

I could play with various Squid settings and refresh_pattern definitions to try and improve it, but I have a fast link and tons of capacity.

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #2 on: March 02, 2015, 10:08:37 am »
Code: [Select]
squidclient -h pfSense_LAN_IP_Address -p 3128 mgr:info

Hi!

This command gives me a bunch of html-code, and when I run
Code: [Select]
squidclient -h 192.168.0.1 -p 3128 mgr:info >> test.php and check the page, it is the "squid-access denied-page".

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #3 on: March 02, 2015, 10:25:14 am »
Just SSH in and run the command as given or do it via Diagnostics - Command Prompt.

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #4 on: March 02, 2015, 11:08:43 am »
Just SSH in and run the command as given or do it via Diagnostics - Command Prompt.


I've tried that too, it is as if squid is blocking the LAN ip.

Edit: I believe I used the wrong IP earlier, however, now when I am using the new IP it results:
Code: [Select]
[2.2-RELEASE][admin@gate.ninya.org]/usr/local/www: squidclient -h 192.168.0.1 -p 3128 mgr:info >> test.php
client: ERROR: Cannot connect to 192.168.0.1:3128: Operation timed out

I've doubled checked, and the port is correct.

Edit 2:

Even this times out:
Code: [Select]
squidclient mgr:info
client: ERROR: Cannot connect to [::1]:3128: Operation timed out
« Last Edit: March 02, 2015, 11:23:02 am by rocketdog »

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #5 on: March 02, 2015, 12:55:48 pm »
You're sure that Squid is running??

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #6 on: March 03, 2015, 07:25:17 am »
You're sure that Squid is running??

AFAIK, yes. And according to ps -aux.

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #7 on: March 03, 2015, 08:06:11 am »
I've never seen that before.  You have Squid listening on port 3128?  Any thing of note in your System log?

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #8 on: March 03, 2015, 08:50:22 am »
Yes, it's on 3128.. forgot to say that it is listening on a bridged interface. Could that be the reason it's "time outing"? I haven't opened any ports or so.
The only rules I have is to let the Bridge to WIFI and LAN net.


Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #9 on: March 03, 2015, 12:10:56 pm »
I can't imagine that bridged mode would affect the squidclient running on the local server.

Do you have anything in Services - Proxy server - Local Cache - External Cache-Managers?  If not, try putting 127.0.0.1; LAN_IP_Address there, save and see if it makes a difference.  For example, I use:

127.0.0.1; 10.10.4.1

where 10.10.4.1 is my pfSense LAN IP address.

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #10 on: March 11, 2015, 10:40:11 am »
I can't imagine that bridged mode would affect the squidclient running on the local server.

Do you have anything in Services - Proxy server - Local Cache - External Cache-Managers?  If not, try putting 127.0.0.1; LAN_IP_Address there, save and see if it makes a difference.  For example, I use:

127.0.0.1; 10.10.4.1

where 10.10.4.1 is my pfSense LAN IP address.

External Cache-Manager was empty, added my IPs now tho..

I've noticed that when I google something, and hit a link, it takes +30 seconds to leave Google for the new link.. What could be the reason for this?

Edit: And how do I get rid of the local hits on "Real Time"? I have added the WAN and LAN IP at "Do not cache", but it still floods 'squid_monitor_data.php'
« Last Edit: March 11, 2015, 10:51:44 am by rocketdog »

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #11 on: March 11, 2015, 12:29:07 pm »
Usually long delays in the cache process are due to DNS issues.  Shell in and run:

Code: [Select]
squidclient -h Your_pfSense_LAN-IP -p 3128 mgr:info
For for Median Service Times section.  Look for long delays relative to the other counters.

I don't have an answer to your real-time question.

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #12 on: March 11, 2015, 12:47:36 pm »
Usually long delays in the cache process are due to DNS issues.  Shell in and run:

Code: [Select]
squidclient -h Your_pfSense_LAN-IP -p 3128 mgr:info
For for Median Service Times section.  Look for long delays relative to the other counters.

I don't have an answer to your real-time question.

This is the result:

Code: [Select]
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.05331  0.02069
        Cache Misses:          0.15048  0.10857
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.03622  0.03829
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.02231  0.02231

I dont know if this is good or bad? Do you want to look for more info?
Some sites (like Google, and Wikipedia in particular, as I said before) can really be a b*tch sometimes..

Offline rocketdog

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +1/-0
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #13 on: April 14, 2015, 06:43:39 pm »
I am bumping the post above:

Is the result of that info good or bad? I am not really that familiar with squid3 yet, and not sure how to tweak it.

Offline KOM

  • Hero Member
  • *****
  • Posts: 5265
  • Karma: +657/-18
    • View Profile
Re: squid3 only reports TCP_MISS, never TCP_HIT
« Reply #14 on: April 14, 2015, 07:20:51 pm »
Nothing really out of the ordinary.  It really depends on what you're loading.  For instance, you can go to a site like CacheFly and download one of their 1 or 10 MB test files, and do it again and watch it come out of the cache.  But for websites these days there is a lot of dynamic content that is generated on the fly, with file names that change based on hashes, or web mechanics that force reloading or prevent caching, etc.  I must admit that I've never seen a zero before for hits if there is web activity for a reasonable amount of time.

If you really want to know, check your access.log(s) in /var/log/squid and search for hits there.  The manager info display only shows the last hour of data.