pfSense Gold Subscription

Author Topic: SNORT - Snort 2.9.2.3 pkg v. 2.2.1 process do not quit via update scripts or GUI  (Read 5459 times)

0 Members and 1 Guest are viewing this topic.

Offline breusshe

  • Jr. Member
  • **
  • Posts: 47
    • View Profile
Since installing the Snort 2.9.2.3 pkg v. 2.2.1, when the snort rules are updated at night, the command to kill existing snort processes fails.  Also, when in the GUI, clicking the red 'X' boxes does not kill the processes.  The GUI acts like it did, but a quick run of ps aux | grep snort shows otherwise.  While the processes fail to stop, this does not stop additional snort processes from starting.  So, after a few days of running snort and auto-updates going on, in my case, I have 2*{a few days} number of snort processes running (2 since I have two snort processes configured on my box).

The only fix I've found is to run 'kill -QUIT {pid}' on all snort processes and then start new snort processes once those have terminated.  I actually have to use the -QUIT signal or they don't stop.  No idea why, but 'kill {pid}' won't do it.  Yes, I'm logged in as admin when I do this.

I'm wondering how the GUI and the rules update scripts go about restarting Snort so I can look at what is wrong with those scripts or does anyone know what the issue is, if not the scripts?

Thanks.


UPDATE:
  • 'kill {pid}' works sometimes, other times it does not.  However, 'kill -QUIT {pid}' always works.  No idea why.
  • Rebooting pfSense causes four processes of Snort (I believe this is because of some, long running oddity where Snort will start at boot time, then stop, then start again).
  • Running 'ps aux | grep snort' does not always show the running Snort processes.  However, using 'top' always works.  No idea why.
« Last Edit: June 24, 2012, 01:36:03 pm by breusshe »

Offline rcfa

  • Sr. Member
  • ****
  • Posts: 565
    • View Profile
Noticed more or less the same here...
...worse, at least for a while, snort was running but the UI would claim otherwise. So hence IPv6 traffic was blocked and I was pulling my hair out, why despite a wide open FW rules, no traffic would flow...

Also, the GUI will at most show Snort active on three of four interfaces, even though ps shows it being active on all four.

Here you see the GUI showing it active on two interfaces, while ps shows it active on all four, with the process running twice on one of the interfaces...

Oh, and this is on 2.1-BETA0 (amd64) built on Wed Jun 20 12:19:46 EDT 2012 FreeBSD 8.3-RELEASE-p3
with the regular non-dev snort package.
« Last Edit: June 22, 2012, 06:36:50 pm by rcfa »

Offline dwood

  • Jr. Member
  • **
  • Posts: 84
    • View Profile
2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011
FreeBSD 8.1-RELEASE-p6

Seeing same thing here with Snort 2.9.2.3 pkg v. 2.2.1, installed and running (after killing dependent packages and clean install of pfsense/snort).  The GUI shows the two SNORT interfaces have stopped when requested, but they continue to run, blocking etc.

Executing ps aux | grep snort at the shell, shows two running process, with the GUI interfaces toggled off.
Executing kill -QUIT {pid} does stop the processes.

For now, I'm guessing any SNORT changes need to accompanied by a router reboot?  Everytime there issues like this crop up, there's a few good lessons passed along in management from the shell :-)  My official favourite new shell commands for relative newbies like myself... and thanks to all who provided them :-)

to list all packages installed:
pkg_info

to delete a package (in this case, perl-5.12.3):
pkg_delete -f perl-5.12.3

to find all snort references:
find /* | grep snort

to find and remove snort references:
find /* | grep -i snort | xargs rm -rv

find processes with snort description:
ps aux | grep snort

kill same processes:
kill -QUIT {pid}



Cheers,
Dennis.
« Last Edit: June 24, 2012, 11:10:33 am by dwood »

Offline krankykoder

  • Newbie
  • *
  • Posts: 3
    • View Profile
I also see the same behavior.

I have snort only on one interface.

Every time snort restarts (auto update rules, manually) all that happens is a new instance of snort running. I have to kill/killall to get them to stop or reboot.


Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14934
    • View Profile
When it's still running, run:

Code: [Select]
ps uxawww | grep snort
And show the output. It's possible that the way it's run via the PBI wrapper the process check/test may be failing to catch it.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline mschiek01

  • Full Member
  • ***
  • Posts: 153
    • View Profile
When it's still running, run:

Code: [Select]
ps uxawww | grep snort
And show the output. It's possible that the way it's run via the PBI wrapper the process check/test may be failing to catch it.



2.0.1-RELEASE (amd64)
Snort 2.9.2.3 pkg v. 2.2.1

root           4501  0.0  9.6 1117960 802636  ??  Ss    5:03PM   0:21.03 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root           6101  0.0  9.6 1117960 799584  ??  Ss    9:34PM   0:00.31 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root          24261  0.0  9.6 1117960 802800  ??  Ss    4:51PM   0:21.83 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root          45617  0.0  9.5 1111816 793256  ??  Ss    9:37PM   0:00.35 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root          55522  0.0  9.9 1126280 826816  ??  Ss    5:06PM  17:23.89 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root          55915  0.0  9.6 1115912 801000  ??  SNs   4:56PM   0:21.24 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
Administrator 55857  0.0  0.0  9120  1452   0  S+    9:39PM   0:00.00 grep snort

Offline miles267

  • Full Member
  • ***
  • Posts: 240
    • View Profile
what is the fix for this?  currently, I have to manually pgrep snort then pkill snort to kill all processes.  after I update rules manually and snort restarts with only one instance.

Offline breusshe

  • Jr. Member
  • **
  • Posts: 47
    • View Profile
Okay, guys, this is sorta fixed.  I found out what the problem is.  You need to edit the file:
Code: [Select]
/usr/local/etc/rc.d/snort.sh
Near the bottom of the file, there is this case statement:
Code: [Select]
case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_start
                ;;
esac

Change it to this:
Code: [Select]
case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_stop
                rc_start
                ;;
esac

Essentially, the rc_stop function call is missing from the restart segment.  Here is the only problem with this fix.  This file is auto-generated whenever there is a change to the snort service.  So, if you add/remove/edit any of your snort interfaces, you will need to add this back in.  I'm looking for the script that auto-generates this file, but have not located it yet.

At least this fix will get you to a place where your Snort isn't failing to kill old iterations of your service.

Oh, one other thing, this only fixes the auto-restart that is done during rules updates.
« Last Edit: July 01, 2012, 06:55:10 pm by breusshe »

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14934
    • View Profile
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline breusshe

  • Jr. Member
  • **
  • Posts: 47
    • View Profile
Huh.  I spoke too soon.  While it is true the rc_stop needed to be added to the snort.sh file, the problem is still not fixed.

There is one other problem in the /usr/local/pkg/snort/snort.inc file.  Line number 894 currently reads as:
Code: [Select]
if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" = "0" ]; then
It should be:
Code: [Select]
if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" != "0" ]; then
The reason is that this line is part of the rc_stop() function and is checking to see if there are any existing instances of Snort running.  "0" simply means the command completed.  However, if an active pid is found in the pid file, that pid number is returned.  Therefore, "=" should be "!=" in order for the pkill command to be run, thus stopping any running instances.  This also needs to be updated in the snort.sh file and can be done by either making some arbitrary change to your snort config, saving, changing back, then resaving or manually editing /usr/local/etc/rc.d/snort.sh so that in the rc_stop() function, each instance of the above if statement is updated to "!=".  Here's an example:
Code: [Select]
if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" = "0" ]; then
becomes:
Code: [Select]
if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" != "0" ]; then
The pid file name will be different for each instance, so don't worry about matching the name.

If you manually edit the snort.sh file DO NOT make this change to the rc_start() function.  It messes things up if you do.
« Last Edit: June 30, 2012, 06:39:37 pm by breusshe »

Offline miles267

  • Full Member
  • ***
  • Posts: 240
    • View Profile
Fixed that for ya:
https://github.com/bsdperimeter/pfsense-packages/commit/cd645a1b11544eda4f3db68ba49caaec8bbf973e

What does this mean?  Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Offline breusshe

  • Jr. Member
  • **
  • Posts: 47
    • View Profile
What does this mean?  Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository.  Snort.inc is a support file that is downloaded during the post-install of Snort.  So, anyone installing/reinstalling Snort will have the fixed snort.inc.  However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected.  As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.
« Last Edit: July 01, 2012, 09:12:06 pm by breusshe »

Offline breusshe

  • Jr. Member
  • **
  • Posts: 47
    • View Profile
Just realized all I needed to do was create a GitHub account and I could do the second part of the fix.  So, I did.  Just waiting for an admin to approve the change and pull it back into the master branch.  Once that is done, this problem should be completely fixed.

https://github.com/bsdperimeter/pfsense-packages/pull/275

Offline miles267

  • Full Member
  • ***
  • Posts: 240
    • View Profile
What does this mean?  Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository.  Snort.inc is a support file that is downloaded during the post-install of Snort.  So, anyone installing/reinstalling Snort will have the fixed snort.inc.  However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected.  As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.

Great!  Thanks for investigating this issue.  Unfortunately, I attempted the manual fix.  While it appeared to work at first, I woke up only to once again find 4-6 instances of snort (I only have 2 interfaces) after the nightly update of definitions so it didn't work as I had hoped.  By sounds of it, we shouldn't have to suffer very much longer.

Offline fragged

  • Full Member
  • ***
  • Posts: 219
    • View Profile
Re-install Snort (I did remove + install from gui) and you should be ok. There's no need for the package version number to be bumped for these fixes to be applied.