Netgate SG-1000 microFirewall

Author Topic: squid - diskd bug (ipcs and ipcrm not available)  (Read 2178 times)

0 Members and 1 Guest are viewing this topic.

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile
squid - diskd bug (ipcs and ipcrm not available)
« on: March 30, 2014, 10:30:38 am »
Tested with squid3-dev 3.3.10 pkg 2.2.1

I'm using squid2 with diskd (on FreeBSD machines) since year 2009 without problems.

However, I had to modify /usr/local/etc/rc.d/squid script to nicely clean diskd squid processes:

Modified code for squid stop
Code: [Select]
squid_stop() {
echo "Stopping ${name}."
${command} ${squid_flags} -k shutdown
run_rc_command poll
# http://man.chinaunix.net/newsoft/squid/Squid_FAQ/FAQ-22.html#ss22.8
sleep 5
ipcs | grep '^[mq]' | awk '{printf "ipcrm -%s %s\n", $1, $2}' | /bin/sh
killall -u squid 2>/dev/null
}

Original code for squid stop
Code: [Select]
squid_stop() {
echo "Stopping ${name}."
${command} ${squid_flags} -k shutdown
run_rc_command poll
}

Now, I'm migrating to squid into pfSense...

I wanted to do the same with /usr/pbi/squid-amd64/etc/rc.d/squid but ipcs and ipcrm aren't available in pfSense.

http://www.freebsd.org/cgi/man.cgi?query=ipcs
http://www.freebsd.org/cgi/man.cgi?query=ipcrm

How to see unclean diskd processes?
Code: [Select]
ps aux | grep diskd
proxy    740  0.0  0.0 11620  2180  ??  IN   Fri07PM   0:00.01 diskd 99774468 99774469 99774470
proxy  12825  0.0  0.0 11620  2316  ??  IN   Sat11AM   0:00.07 diskd 8102916 8102917 8102918
proxy  14947  0.0  0.0 11620  2244  ??  IN   Fri08PM   0:00.01 diskd 13276164 13276165 13276166
proxy  20939  0.0  0.0 11620  2180  ??  IN   Fri07PM   0:00.01 diskd 18546692 18546693 18546694
proxy  53058  0.0  0.0 11620  2264  ??  S    12:24PM   0:00.34 diskd 52115460 52115461 52115462

In the example, squid last stop/start has been at Sunday, 12:24PM. We can see many zombie diskd processes.
« Last Edit: March 30, 2014, 10:41:26 am by bellera »

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #2 on: March 30, 2014, 11:32:26 am »
Download livefs.iso from

ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/8.3/

mount it and copy /usr/bin/ipcs and /usr/bin/ipcrm to your system and set them as executables.

or http://www.filewatcher.com/m/ipcrm.12888-0.html

64 bit
Code: [Select]
fetch -o /usr/bin/ipcs ftp://dns.ruweb.net/FreeBSD/8.3-RELEASE-amd64-livefs/usr/bin/ipcs
chmod 555 /usr/bin/ipcs
fetch -o /usr/bin/ipcrm ftp://dns.ruweb.net/FreeBSD/8.3-RELEASE-amd64-livefs/usr/bin/ipcrm
chmod 555 /usr/bin/ipcrm
rehash

Now, here is my pfSense running squid with diskd problems
Code: [Select]
ipcs
Message Queues:
T           ID          KEY MODE        OWNER    GROUP   
q       131072     99774468 --rwa------ proxy    proxy   
q       131073     99774469 --rwa------ proxy    proxy   
q        65538     18546692 --rwa------ proxy    proxy   
q        65539     18546693 --rwa------ proxy    proxy   
q        65540     13276164 --rwa------ proxy    proxy   
q        65541     13276165 --rwa------ proxy    proxy   
q       131078      8102916 --rwa------ proxy    proxy   
q       131079      8102917 --rwa------ proxy    proxy   
q        65544     52115460 --rwa------ proxy    proxy   
q        65545     52115461 --rwa------ proxy    proxy   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP   
m       131072     99774470 --rw------- proxy    proxy   
m        65537     18546694 --rw------- proxy    proxy   
m        65538     13276166 --rw------- proxy    proxy   
m       131075      8102918 --rw------- proxy    proxy   
m        65540     52115462 --rw------- proxy    proxy   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP

« Last Edit: March 30, 2014, 11:57:59 am by bellera »

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #3 on: March 30, 2014, 01:08:47 pm »
Looking at /usr/local/pkg/squid.inc the code writes /usr/local/etc/rc.d/squid.sh

I patched /usr/local/pkg/squid.inc

Code: [Select]
cp -p squid.inc squid.inc-2014-04-30
vi squid.inc

Original code
Code: [Select]
{$squid_local_base}/sbin/squid -k shutdown -f {$squid_conffile_var}
# Just to be sure...
sleep 5
killall -9 squid 2>/dev/null
killall pinger 2>/dev/null

Patched code
Code: [Select]
{$squid_local_base}/sbin/squid -k shutdown -f {$squid_conffile_var}
# Just to be sure...
sleep 5
       
# http://man.chinaunix.net/newsoft/squid/Squid_FAQ/FAQ-22.html#ss22.8
ipcs | grep '^[mq]' | awk '{printf "ipcrm -%s %s\\n", $1, $2}' | /bin/sh

killall -9 squid 2>/dev/null
killall pinger 2>/dev/null

I modified also /usr/local/etc/rc.d/squid.sh in order to have the changes and stop/start squid. Everything ok!

proxy processes
Code: [Select]
ps aux | grep ^proxy
proxy  84290  0.0  0.4 48996 18532  ??  SN    8:20PM   0:03.12 (squid-1) -f /usr/pbi/squid-amd64/etc/squid/squid.conf (squid)
proxy  84619  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  84890  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  85084  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  85292  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  85630  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  85844  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  85945  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  86112  0.0  0.2 64640 11008  ??  IN    8:20PM   0:00.04 (squidGuard) -c /usr/pbi/squidguard-squid3-amd64/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  86450  0.0  0.0 11252  2124  ??  IN    8:20PM   0:00.01 (unlinkd) (unlinkd)
proxy  86470  0.0  0.0 11620  2244  ??  IN    8:20PM   0:00.01 diskd 86312964 86312965 86312966

Correct messages queues & shared memory
Code: [Select]
ipcs
Message Queues:
T           ID          KEY MODE        OWNER    GROUP   
q       196608     86312964 --rwa------ proxy    proxy   
q       196609     86312965 --rwa------ proxy    proxy   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP   
m       131081     86312966 --rw------- proxy    proxy   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP
« Last Edit: April 02, 2014, 03:36:46 am by bellera »

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #4 on: April 02, 2014, 03:39:06 am »
Corrected /usr/local/pkg/squid.inc patched code

It was

ipcs | grep '^[mq]' | awk '{printf "ipcrm -%s %s\n", $1, $2}' | /bin/sh

Must be

ipcs | grep '^[mq]' | awk '{printf "ipcrm -%s %s\\n", $1, $2}' | /bin/sh

Also corrected at previous message.

Offline marcelloc

  • Hero Member
  • *****
  • Posts: 13319
  • Karma: +572/-6
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #5 on: April 03, 2014, 12:43:47 pm »
Package updated to include install instructions and checks

Offline bellera

  • Hero Member
  • *****
  • Posts: 5569
  • Karma: +117/-51
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #6 on: April 03, 2014, 12:48:14 pm »
Fantastic! Thanks!

Offline Tikimotel

  • Full Member
  • ***
  • Posts: 201
  • Karma: +11/-0
    • View Profile
Re: squid - diskd bug (ipcs and ipcrm not available)
« Reply #7 on: January 17, 2015, 02:47:08 pm »
For Squid3 (3.4.10_2 0.2.5), on pfsense 2.2 you need ipcs and ipcrm from a 10.1 base.
The diskd text should reflect that, now it still refers to 8.3.

Code: [Select]
diskd uses a separate process to avoid blocking the main Squid process on disk-I/O.
To use ipcs and ipcrm on squid, Download livefs.iso from ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/8.3/ mount it and copy /usr/bin/ipcs and /usr/bin/ipcrm to your system and set them as executables.

Code: [Select]
diskd uses a separate process to avoid blocking the main Squid process on disk-I/O.
To use ipcs and ipcrm on squid, Download livefs.iso from ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/ mount it and copy /usr/bin/ipcs and /usr/bin/ipcrm to your system and set them as executables.