pfSense Support Subscription

Author Topic: Improve Custom refresh pattern  (Read 9825 times)

0 Members and 1 Guest are viewing this topic.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #30 on: March 02, 2017, 06:16:50 pm »
updated, thanks to Valeriy

if there are any mistakes in the refresh pattern let me know
Never Fear, A Geek is Here!

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #31 on: March 07, 2017, 06:35:35 am »
KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.

Few days ago I have made some improvements to utilize store_rewrite module.
They were tested, as suggested above by KOM, successfully.

Indeed, some of WU were not cached properly since they were received from different CDN hosts.


Config files are below (squid.conf is only part that is related to this issue)

Apple and Symantec updates are cached well, too.

Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.

The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.

Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.


P.S. Questions:
- not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?

Comments:
 - I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.

Current byte/hit rate is 20-45%


/usr/local/etc/squid/squid.conf

Code: [Select]
.......
# No range offset limit for windowsupdate sites
acl Windows_Update dstdomain windowsupdate.microsoft.com
acl Windows_Update dstdomain update.microsoft.com
acl Windows_Update dstdomain download.windowsupdate.com
acl Windows_Update dstdomain windowsupdate.com
acl Windows_Update dstdomain au.download.windowsupdate.com
acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com
acl Windows_Update dstdomain au.b1.download.windowsupdate.com
range_offset_limit -1  Windows_Update

# Storeid_rewrite configuration

acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt"

store_id_access allow updatesites
store_id_access deny all
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf
store_id_children 200 startup=60 idle=1 concurrency=0


# Refresh patterns
# 525600 min is one year

# Gaming CDN

refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

#windows update

refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth


# Symantec Updates
# Only reload-into-ims works properly, other flags break update process

refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7)          525600 100% 525600 reload-into-ims



/usr/local/etc/squid/updatesites.txt
Code: [Select]
\.adobe.com
\.java.com
\.sun.com
\.oracle.com
\.apple.com
\.microsoft.com
\.windowsupdate.com
\.ubuntu.com
\.steampowered.com
\.llnwd.net
\.symantecliveupdate.com


/usr/local/etc/squid/storeid_rewrite.conf
Code: [Select]
^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx)     http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.cs\.steampowered\.com\/(.*)      http://steamupdates.squid.internal/$1
^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
^http:\/\/.+?\.llnwd\.net\/(.*)                 http://llnwd.net.squid.internal/$1


Regards,


Valeriy




« Last Edit: April 11, 2017, 02:43:54 am by Valeriy »

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #32 on: March 07, 2017, 06:38:10 am »
Just a note: manually modifying squid.conf is not good idea, since it can and will be rewritten by PfSense later. With next post I ll try to summarize all the changes and how to incorporate them using GUI of pfSense.

Offline SaschaITM

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +2/-2
    • View Profile
Re: Improve Custom refresh pattern
« Reply #33 on: March 07, 2017, 11:42:17 pm »
This is interesting, I wish I had the time to test this thoroughly. Will do that as soon as I find the time.

FWIW, I had massive problems with the "range_offset_limit -1" directive and Windows 10 updates, with "psf" added to the "refresh_pattern". Update download was extremely slow and used up all available bandwidth after a while. It looked like Win10 was aborting and restarting the update download every few seconds, while Squid was still downloading from previous requests. Commented out the "range_offset_limit -1", and updates were downloading "normally" again. The "-1" value isn't documented in the current Squid docs, but it's documented for Squid 2.7. I wonder if "range_offset_limit -1" is still valid/needed with recent Squid versions.

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #34 on: March 09, 2017, 11:06:35 am »
It worked very well.

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #35 on: March 09, 2017, 05:26:24 pm »
No caches of video, can you help?

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #36 on: March 13, 2017, 02:53:37 am »
Forget about video caching...
Most video services push content via HTTPS, which we cannt cache.

If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

Offline pfsensation

  • Full Member
  • ***
  • Posts: 214
  • Karma: +3/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #37 on: March 17, 2017, 01:36:32 pm »
Forget about video caching...
Most video services push content via HTTPS, which we cannt cache.

If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

Unless...You do MITM inspection.

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #38 on: March 18, 2017, 06:53:39 am »
Forget about video caching...
Most video services push content via HTTPS, which we cannt cache.

If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

Unless...You do MITM inspection.

I do not know the MITM inspection.

Offline pfsensation

  • Full Member
  • ***
  • Posts: 214
  • Karma: +3/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #39 on: March 18, 2017, 02:06:49 pm »
Forget about video caching...
Most video services push content via HTTPS, which we cannt cache.

If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

Unless...You do MITM inspection.

I do not know the MITM inspection.

I'm currently having issues with it after the latest Squid update, unsure as to why. However, you can do MITM inspection by enabling SSL filtering and deploying out a CA certificate to your clients.

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #40 on: April 05, 2017, 06:14:05 pm »
Any update not refresh_pattern?

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #41 on: April 05, 2017, 10:59:26 pm »
Caching video is bad idea, thats all I can say :)

Offline kikawala

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
    • MedSynergies, Inc.
Re: Improve Custom refresh pattern
« Reply #42 on: April 09, 2017, 04:39:41 pm »
I have this for Apple iOS updates and iOS apps.

Code: [Select]
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #43 on: April 09, 2017, 08:29:43 pm »
Thanks updated
Never Fear, A Geek is Here!

Offline waldopulanco

  • Jr. Member
  • **
  • Posts: 38
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #44 on: April 16, 2017, 07:09:01 am »
how about youtube?