pfSense Support Subscription

Author Topic: Improve Custom refresh pattern  (Read 9832 times)

0 Members and 1 Guest are viewing this topic.

Offline pfsensation

  • Full Member
  • ***
  • Posts: 214
  • Karma: +3/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #45 on: April 17, 2017, 01:17:56 pm »
I have this for Apple iOS updates and iOS apps.

Code: [Select]
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

Doesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.

Offline PFsense Learner

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #46 on: April 19, 2017, 11:24:32 pm »
KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.

Few days ago I have made some improvements to utilize store_rewrite module.
They were tested, as suggested above by KOM, successfully.

Indeed, some of WU were not cached properly since they were received from different CDN hosts.


Config files are below (squid.conf is only part that is related to this issue)

Apple and Symantec updates are cached well, too.

Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.

The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.

Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.


P.S. Questions:
- not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?

Comments:
 - I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.

Current byte/hit rate is 20-45%


/usr/local/etc/squid/squid.conf

Code: [Select]
.......
# No range offset limit for windowsupdate sites
acl Windows_Update dstdomain windowsupdate.microsoft.com
acl Windows_Update dstdomain update.microsoft.com
acl Windows_Update dstdomain download.windowsupdate.com
acl Windows_Update dstdomain windowsupdate.com
acl Windows_Update dstdomain au.download.windowsupdate.com
acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com
acl Windows_Update dstdomain au.b1.download.windowsupdate.com
range_offset_limit -1  Windows_Update

# Storeid_rewrite configuration

acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt"

store_id_access allow updatesites
store_id_access deny all
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf
store_id_children 200 startup=60 idle=1 concurrency=0


# Refresh patterns
# 525600 min is one year

# Gaming CDN

refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

#windows update

refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth

refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth


# Symantec Updates
# Only reload-into-ims works properly, other flags break update process

refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7)          525600 100% 525600 reload-into-ims



/usr/local/etc/squid/updatesites.txt
Code: [Select]
\.adobe.com
\.java.com
\.sun.com
\.oracle.com
\.apple.com
\.microsoft.com
\.windowsupdate.com
\.ubuntu.com
\.steampowered.com
\.llnwd.net
\.symantecliveupdate.com


/usr/local/etc/squid/storeid_rewrite.conf
Code: [Select]
^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx)     http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.cs\.steampowered\.com\/(.*)      http://steamupdates.squid.internal/$1
^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
^http:\/\/.+?\.llnwd\.net\/(.*)                 http://llnwd.net.squid.internal/$1


Regards,


Valeriy
May I know.
1. Is it just copy CODE in Table 1(/usr/local/etc/squid/squid.conf) and paste to "Custom refresh_patterns" of squid on Pfsense ? Or any other changes need on Table 1.
2. How useful Table 2(/usr/local/etc/squid/updatesites.txt ) and Table 3(/usr/local/etc/squid/storeid_rewrite.conf)? Need to insert these tables on Table 1? If needed, guide me pls(I am Beginner)
My aim is to get steam update cache(dota2)

Offline kikawala

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
    • MedSynergies, Inc.
Re: Improve Custom refresh pattern
« Reply #47 on: April 22, 2017, 10:11:31 pm »
I have this for Apple iOS updates and iOS apps.

Code: [Select]
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate

Doesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.

Apps and iOS updates are over HTTP

Offline trinitech

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #48 on: May 26, 2017, 09:41:42 am »
Hi,

Sorry for the stupid question..
But where does the following code goes:
Code: [Select]
#new refresh patterns 3
acl Windows_Update dstdomain windowsupdate.microsoft.com
acl Windows_Update dstdomain .update.microsoft.com
acl Windows_Update dstdomain download.windowsupdate.com
acl Windows_Update dstdomain www.download.windowsupdate.com
acl Windows_Update dstdomain au.download.windowsupdate.com
acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
Does it go to the ACLs tab?
Is it white list or does it go in local cache -> Custom refresh_patterns

Thank you

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 13406
  • Karma: +591/-7
    • View Profile
Re: Improve Custom refresh pattern
« Reply #49 on: May 26, 2017, 04:17:57 pm »

Is it white list or does it go in local cache -> Custom refresh_patterns


Custom refresh_patterns

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #50 on: June 01, 2017, 07:29:57 pm »
Any refresh for videos in http?

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #51 on: June 02, 2017, 06:52:24 pm »
(
Any refresh for videos in http?

No, although refresh patterns are not my strong point
Never Fear, A Geek is Here!

Offline erviesuarez

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #52 on: June 03, 2017, 04:00:51 am »
do i just need to copy this codes and paste it into my custom pattern_refresh box? soory im a newbie here.

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #53 on: June 10, 2017, 01:54:05 am »
Yes, did it improve your hit rate?
Never Fear, A Geek is Here!

Offline remzej

  • Newbie
  • *
  • Posts: 16
  • Karma: +12/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #54 on: June 18, 2017, 10:47:26 pm »
Two invalid expressions.
Somebody can fix them?

/usr/local/etc/squid/squid.conf line 52: refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
refreshAddToList: Invalid regular expression '\.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv)': empty (sub)expression



  kid1| /usr/local/etc/squid/squid.conf line 136: refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
 refreshAddToList: Invalid regular expression '^http://(khm?)([^/]*?)\.google\.(de|com)': repetition-operator operand invalid


I just change it to ^http://(khm?)([^/]?)\.google\.(de|com) by removing the *

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #55 on: June 19, 2017, 04:08:14 am »
So replace

refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
with ?
refresh_pattern -i ^http://(khm?)([^/]?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload


replace
refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload

with ?

refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload

Can someone confirm the changes are correct.
Never Fear, A Geek is Here!

Offline brianc69

  • Full Member
  • ***
  • Posts: 150
  • Karma: +4/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #56 on: June 20, 2017, 05:12:12 pm »
Facebook is a heavy traffic site for me which I could probably benefit from caching. I'm curious what the ip is?

#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)           129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)            129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)  129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store

Offline aGeekHere

  • Sr. Member
  • ****
  • Posts: 525
  • Karma: +43/-1
    • View Profile
Re: Improve Custom refresh pattern
« Reply #57 on: June 21, 2017, 02:38:09 am »
Just a question has anyone found an Improvement in there hit rate using this custom refresh pattern?

I wounder if there is a simple refresh_pattern * that sates cache all instead of defining all these file types and domains.
Never Fear, A Geek is Here!

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 48
  • Karma: +7/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #58 on: June 21, 2017, 04:02:21 am »
A simple refresh pattern will break HTTP heavily.
You  can increase number of minutes in default pattern and see yourself.

Offline saluto

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +0/-0
    • View Profile
Re: Improve Custom refresh pattern
« Reply #59 on: July 22, 2017, 10:30:17 am »
Facebook is a heavy traffic site for me which I could probably benefit from caching. I'm curious what the ip is?

#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)           129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)            129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)  129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store

something in the link below, but I was not able to  implement in pfsense.
http://comastnet.blogspot.com.br/2016/06/squid-youtube-cache-2016-terbaru-suport.html
« Last Edit: July 22, 2017, 12:57:33 pm by saluto »