Hello fellow Netgate community members,
After lots of trial and error while researching this I have gotten StoreID to start working finally correctly. Well I think I have there is always bugs that pop up. I wanted to share the with you incase other hardcore Squid users are still out there.
"....ICAP + StoreID helper idea works in production with more then one site for quite some time but it has some overheads and I would rate this kind of a setup as an Expert only" (wiki.squid-cache.org).
This does require SSL certificates to work so you can not just make it work without owning the devices and hardware to install the certs.
You can't see the data in the cache you can only see hits as it is inside of the cache encrypted so no bad guys also...
Squid does come with a very nice useable StoreID program that is built into the package and it has all you need to do is create your database for it.
"This program acts as a store_id helper program, rewriting URLs passed by Squid into storage-ids that can be used to achieve better caching for websites that use different URLs for the same content.
It takes a text file with two tab separated columns. Column 1: Regular expression to match against the URL Column 2: Rewrite rule to generate a Store-ID Eg: ^http://[^.]+.dl.sourceforge.net/(.*) http://dl.sourceforge.net.squid.internal/$1
Rewrite rules are matched in the same order as they appear in the rules file. So for best performance, sort it in order of frequency of occurrence.
This program will automatically detect the existence of a concurrecy channel-ID and adjust appropriately. It may be used with any value 0 or above for the store_id_children concurrency= parameter" (systutorials).
I placed this under custom refresh_patterns
acl getmethod method GET
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow all #I have major concern with this rule this is a work in progress
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0
refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
#FACEBOOK
refresh_pattern ^http://*.facebook.com/* 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
#FACEBOOK IMAGES
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth
The path to Squids store id program is here, it is part of Squid package and will be on your system already just waiting for someone to dust off that code and activate it.
/usr/local/libexec/squid/storeid_file_rewrite
Keep in mind you are required to use a database. I created one with this location and just saved my needs in it for testing
/var/squid/storeid/storeid_rewrite.txt
I created my db and saved it in the same location as a txt file.
Mine is a very basic setup as I am new to StoreID however Squid lists a useable database here for many sites. I am sure the major ISP systems have monster cache systems.
https://wiki.squid-cache.org/Features/StoreID/DB
Keep in mind there is risks if you do not store items correctly it can display items on unwanted terminals. So don't mess this up on anyone. Know the risks associated with this.
# Facebook
^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/.*\/v\/.*\/(.*\.mp4) http://facebook.squid.internal/$3
^https?:\/\/fbcdn\-(static|profile)\-a\.akamaihd\.net\/static\-ak\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*) http://facebook.squid.internal/static/$2
^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/(h|s)(profile|photos).*\/(.*\.(png|gif|jpg))(\?.+)? http://facebook.squid.internal/$5
^https?:\/\/fbstatic\-a\.akamaihd\.net\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*) http://facebook.squid.internal/static/$1
^http:\/\/.*[steampowered|steamcontent]\.com\/([^?]*) http://steamupdates.squid.internal/$1
^https?\:\/\/download\.oracle\.com\/((otn\-pub|otn)\/[\d\w]+\/[\d\w]+\/[\w\d\-]+\/[\w\d\-]+\.(exe|dmg|rpm|msi|tar\.(gz|Z)))\? http://java.oracle.otn.ngtech.squid.internal/$1
^https?\:\/\/([\d\w\-]+)\.oracle\.com\/(([\d\w]+)\/[\d\w]+\/[\d\w]+\/([\d\w\-]+)\/([\d\w]+\/)?[\d\w\-\.\_]+\.(dmg|msi|exe|tar\.gz|tar\.Z))\? http://java.oracle.download.ngtech.squid.internal/$2
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
^http:\/\/[^\.]+\.c\.android\.clients\.google\.com\/(.*) http://androidupdates.google.squid.internal/$1
Screenshot 2024-03-18 at 23.30.25.png
Now watch the hits come on in and after a while you will have your cache working great with other CDN s just add them to the database. Soon they say they Squid will also add something called Metalink support into squid. Ref http://www.metalinker.org to simplify this process and help cut down on energy use for downloading the same thing over and over and over across networks. There is no point when the videos and images are the same why not just redeliver it locally and securely.
This really does accelerate traffic great.
ref:
https://wiki.squid-cache.org/Features/StoreID
https://www.systutorials.com/docs/linux/man/8-storeid_file_rewrite/
I hope you enjoyed this research of the tools already built into this package. If anyone is out there that knows how to make this more secure or wants to add anything please comment below. I am amazed it started working for me. Please if you see something wrong let me know. I have been excited to see this work from 2019 until today and this was the first time I have seen it in action and it did not display the same add over every photo this time.
I also wanted to note I have a custom Squid option installed here. So the system works with both splice only systems and the systems I want the cache/AV running on inspecting traffic.
acl manager proto cache_object
acl localhost src 192.168.1.1/32
http_access allow manager localhost
http_access deny manager
acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad
acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell
ssl_bump peek step1
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all
acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all
#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept
Enjoy please message below if something is completely off.