Netgate SG-1000 microFirewall

Author Topic: Improve Custom refresh pattern  (Read 9868 times)

0 Members and 1 Guest are viewing this topic.

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #15 on: February 16, 2017, 08:55:05 am »
I use squid for caching, it saves bandwidth for Microsoft Updates and Apple updates + SquidGuard as Ad+Malware+porn filter

Offline KOM

  • Hero Member
  • *****
  • Posts: 5506
  • Karma: +681/-23
    • View Profile
Re: Improve Custom refresh pattern
« Reply #16 on: February 16, 2017, 09:18:40 am »
Quote
it saves bandwidth for Microsoft Updates

I have yet to see a working config with squid and WU.  Most people give up and use WSUS.

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #17 on: February 16, 2017, 09:20:04 am »
I cannt use WSUS, PCs are not under my control.

Offline KOM

  • Hero Member
  • *****
  • Posts: 5506
  • Karma: +681/-23
    • View Profile
Re: Improve Custom refresh pattern
« Reply #18 on: February 16, 2017, 09:53:52 am »
If you have a working squid config that caches WU for Win7-10 clients, I'd love to see it.

Offline danbak365

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #19 on: February 16, 2017, 02:33:42 pm »
These refresh pattern made a big difference on my WI-fi network.
Since I started using it my clients have been very happy with the speed.
Even though, now it seems like they need some update, I still appreciate this well done job.
Thanks

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #20 on: February 16, 2017, 03:52:39 pm »
You can use squidlite to check if the refresh pattens are improving the hit rate. I have found that the comodo antivirus updates were caching ( cannot retest at the moment) and so did window updates. The best thing is to test if the hit ratio is improving.
Never Fear, A Geek is Here!

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #21 on: February 20, 2017, 10:49:59 am »
« Last Edit: February 20, 2017, 03:02:34 pm by saluto »

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #22 on: February 21, 2017, 09:41:19 am »
@KOM

Here are changes made to squid.conf (or through GUI on pfsense in corresponding custom fields)
 that allows caching of Windows Update for me:

Code: [Select]

acl Windows_Update dstdomain windowsupdate.microsoft.com
acl Windows_Update dstdomain .update.microsoft.com
acl Windows_Update dstdomain download.windowsupdate.com
acl Windows_Update dstdomain www.download.windowsupdate.com
acl Windows_Update dstdomain au.download.windowsupdate.com
acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
range_offset_limit 200 MB Windows_Update
quick_abort_min 1000 KB
quick_abort_max 5000 KB
quick_abort_pct 80
request_body_max_size 0 KB





refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe)                     259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf)                  259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims

cache_mem 15000 MB
# Should be less than 50% of avail. RAM


maximum_object_size_in_memory 5000000 KB
#should be less than cache_mem

memory_replacement_policy heap LFUDA
cache_replacement_policy heap LFUDA

minimum_object_size 1000 KB
#Do not store very small object on disk

maximum_object_size 5000 MB
#should be less or equal to maximum_object_size_in_memory



Here is output for today access.log of squid (number of HITS):
cat  /var/squid/logs/access.log | grep HIT | grep windowsupdate -c
2860
cat /var/squid/logs/access.log | grep HIT | grep microsoft.com -c
19128

Here is output for today access.log of squid (number of MISSes):
 cat /var/squid/logs/access.log | grep MISS | grep windowsupdate -c
61834
cat /var/squid/logs/access.log | grep MISS | grep microsoft.com -c
55567


I had to rebuild my disk cache 2 days ago, so I assume not everything is cached yet. Or some of my settings needs improvement...

It might not be ideal setup.
If you have some comments, improvements, please let me know.


Regards,


Valeriy

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #23 on: February 23, 2017, 02:07:59 pm »
How did your dynamic cache get now?

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #24 on: February 24, 2017, 03:28:56 am »
How did your dynamic cache get now?
It is good.
This is result for yesterday:

Code: [Select]
[2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep MISS | grep microsoft.com -c
96270
[2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep HIT | grep microsoft.com -c
21177
(18% HITS vs. 82% MISS)
[2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep HIT | grep windowsupdate -c
8352
[2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep MISS | grep windowsupdate -c
34794
(19% HITS vs. 81% MISS)


Offline KOM

  • Hero Member
  • *****
  • Posts: 5506
  • Karma: +681/-23
    • View Profile
Re: Improve Custom refresh pattern
« Reply #25 on: February 24, 2017, 07:59:28 am »
 :o

81% miss rate seems awfully high for something that's supposedly working good.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #26 on: February 24, 2017, 04:40:16 pm »
Should look at the hit/miss ratio of file sizes. If you miss 99% of 1kB files it does not matter however it can through your ratio out.
Never Fear, A Geek is Here!

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #27 on: February 26, 2017, 09:34:32 pm »
« Last Edit: February 27, 2017, 10:54:36 am by saluto »

Offline SaschaITM

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +2/-2
    • View Profile
Re: Improve Custom refresh pattern
« Reply #28 on: February 27, 2017, 06:14:58 pm »
KOM: "19% hit rate seems awfully high for something that's supposedly not working at all." (;

Even 19 % hit rate is a benefit if you have to work with a slow/congested line. I think hit rate can be optimized by setting up a machine that will download updates before the standard time frame windows is using to check for/download updates, so that updates are already cached when other machines check for them (I'm going to implement this on one of the systems I'm managing as soon as I find the time). Also, like aGeekHere said, you'd have to look at the byte ratio to be able to see how effective this form of caching actually is. Not sure how this could be done, maybe with a tool like Calamaris and a pre-filtered Squid log file?

Bottom line: caching Windows updates does indeed work, and is actually useful for (at least) some people.

Offline KOM

  • Hero Member
  • *****
  • Posts: 5506
  • Karma: +681/-23
    • View Profile
Re: Improve Custom refresh pattern
« Reply #29 on: February 28, 2017, 08:05:31 am »
Quote
Bottom line: caching Windows updates does indeed work

I still have yet to see any evidence to back this claim.  A 19% hit rate for WU is crap.  And no, there aren't a zillion little 1K files skewing the ratio.

Here is a proper test:

take two Windows 7 clients that you installed fresh and put them both behind the proxy, leaving only 1 powered on.
Install all updates until there are none left
Power on the second unit and then do all the same updates
Once complete, on pfSense run:

Code: [Select]
squidclient -h LAN_IP -p 3128 mgr:info
and post the output.  That will give a lot more detail about the recent session.