The pfSense Store

Author Topic: Suricata keeps crashing since 2.4.2 upgrade  (Read 269 times)

0 Members and 1 Guest are viewing this topic.

Offline chiefgyk

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Suricata keeps crashing since 2.4.2 upgrade
« on: December 01, 2017, 09:43:19 pm »
Suricata will not stay running whenever I restart the service it just crashes again and I have tried reinstalling the package and such.

Offline chiefgyk

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #1 on: December 02, 2017, 06:28:52 am »
Ever since I updated to 2.4.2 I( noticed Suricata won't stay on for more than a few seconds without crashing. I had it setup and running perfectly since 2.3.4, updated to 2.3.5, 2.4.0, and 2.4.1 with no problems. I even tried backing up and restoring a config file. Only thing that's changed is I loaded the SSD and NIC cards into a new boxand had to reorder the NIC assignments em0 used to be WAN now it's em4 em1 used to be LAN now it''s em0 DMZ used to be em2 now it's em1 but everything else is the same. I tried reinstalling Suricata as well to no availin fixing my issue

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #2 on: December 02, 2017, 02:39:52 pm »
Ever since I updated to 2.4.2 I( noticed Suricata won't stay on for more than a few seconds without crashing. I had it setup and running perfectly since 2.3.4, updated to 2.3.5, 2.4.0, and 2.4.1 with no problems. I even tried backing up and restoring a config file. Only thing that's changed is I loaded the SSD and NIC cards into a new boxand had to reorder the NIC assignments em0 used to be WAN now it's em4 em1 used to be LAN now it''s em0 DMZ used to be em2 now it's em1 but everything else is the same. I tried reinstalling Suricata as well to no availin fixing my issue

New interface names should not make it crash so long as the underlying NIC driver is essentially the same.  However, changing interface names could mess up your rule assignments as what you might have on LAN is now maybe DMZ or WAN, for example.

What kind of messages are you getting in the suricata.log file?  You can view that log using the LOGS VIEW tab.  Also, is there anything logged in the pfSense system log?

Are you using Legacy Mode for blocking or Inline IPS Mode?

Bill


Offline chiefgyk

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #3 on: December 04, 2017, 10:16:54 pm »
I am using Legacy mode.

Here is what Suricata.log is saying
http://dumptext.com/LxBvOu58

Nothing really shows up in the system log that is out of the ordinary. Just says it is starting Suricata and nothing else

Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   SuricataStartup   55863   Suricata START for WAN(45069_em4)...
Dec 4 23:08:34   SuricataStartup   60882   Suricata START for LAN(42126_em0)...
Dec 4 23:08:35   SuricataStartup   63724   Suricata START for VoIP(10756_em1)...
Dec 4 23:09:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:09:00   SuricataStartup   13977   Suricata START for WAN(45069_em4)...
Dec 4 23:09:01   SuricataStartup   15477   Suricata START for LAN(42126_em0)...
Dec 4 23:09:02   SuricataStartup   18308   Suricata START for VoIP(10756_em1)...
Dec 4 23:10:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:10:00   SuricataStartup   44977   Suricata START for WAN(45069_em4)...
Dec 4 23:10:01   SuricataStartup   50292   Suricata START for LAN(42126_em0)...
Dec 4 23:10:02   SuricataStartup   50890   Suricata START for VoIP(10756_em1)...
Dec 4 23:11:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:11:00   SuricataStartup   78995   Suricata START for WAN(45069_em4)...
Dec 4 23:11:01   SuricataStartup   80028   Suricata START for LAN(42126_em0)...
Dec 4 23:11:02   SuricataStartup   80695   Suricata START for VoIP(10756_em1)...
Dec 4 23:12:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:12:00   SuricataStartup   6914   Suricata START for WAN(45069_em4)...
Dec 4 23:12:01   SuricataStartup   8617   Suricata START for LAN(42126_em0)...
Dec 4 23:12:02   SuricataStartup   10044   Suricata START for VoIP(10756_em1)...
Dec 4 23:12:58   SuricataStartup   73963   Suricata START for WAN(45069_em4)...
Dec 4 23:12:59   SuricataStartup   80831   Suricata START for LAN(42126_em0)...
Dec 4 23:13:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:13:00   SuricataStartup   82613   Ignoring additional START command since Suricata is already starting...
Dec 4 23:13:00   SuricataStartup   84841   Suricata START for VoIP(10756_em1)...
Dec 4 23:14:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:14:00   SuricataStartup   33805   Suricata START for WAN(45069_em4)...
Dec 4 23:14:01   SuricataStartup   34953   Suricata START for LAN(42126_em0)...
Dec 4 23:14:02   SuricataStartup   35531   Suricata START for VoIP(10756_em1)...

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #4 on: December 05, 2017, 08:18:04 am »
I am using Legacy mode.

Here is what Suricata.log is saying
http://dumptext.com/LxBvOu58

Nothing really shows up in the system log that is out of the ordinary. Just says it is starting Suricata and nothing else

Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33   bandwidthd      Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Opening em0
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   bandwidthd      Packet Encoding: Ethernet
Dec 4 23:08:33   SuricataStartup   55863   Suricata START for WAN(45069_em4)...
Dec 4 23:08:34   SuricataStartup   60882   Suricata START for LAN(42126_em0)...
Dec 4 23:08:35   SuricataStartup   63724   Suricata START for VoIP(10756_em1)...
Dec 4 23:09:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:09:00   SuricataStartup   13977   Suricata START for WAN(45069_em4)...
Dec 4 23:09:01   SuricataStartup   15477   Suricata START for LAN(42126_em0)...
Dec 4 23:09:02   SuricataStartup   18308   Suricata START for VoIP(10756_em1)...
Dec 4 23:10:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:10:00   SuricataStartup   44977   Suricata START for WAN(45069_em4)...
Dec 4 23:10:01   SuricataStartup   50292   Suricata START for LAN(42126_em0)...
Dec 4 23:10:02   SuricataStartup   50890   Suricata START for VoIP(10756_em1)...
Dec 4 23:11:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:11:00   SuricataStartup   78995   Suricata START for WAN(45069_em4)...
Dec 4 23:11:01   SuricataStartup   80028   Suricata START for LAN(42126_em0)...
Dec 4 23:11:02   SuricataStartup   80695   Suricata START for VoIP(10756_em1)...
Dec 4 23:12:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:12:00   SuricataStartup   6914   Suricata START for WAN(45069_em4)...
Dec 4 23:12:01   SuricataStartup   8617   Suricata START for LAN(42126_em0)...
Dec 4 23:12:02   SuricataStartup   10044   Suricata START for VoIP(10756_em1)...
Dec 4 23:12:58   SuricataStartup   73963   Suricata START for WAN(45069_em4)...
Dec 4 23:12:59   SuricataStartup   80831   Suricata START for LAN(42126_em0)...
Dec 4 23:13:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:13:00   SuricataStartup   82613   Ignoring additional START command since Suricata is already starting...
Dec 4 23:13:00   SuricataStartup   84841   Suricata START for VoIP(10756_em1)...
Dec 4 23:14:00   php-cgi      servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:14:00   SuricataStartup   33805   Suricata START for WAN(45069_em4)...
Dec 4 23:14:01   SuricataStartup   34953   Suricata START for LAN(42126_em0)...
Dec 4 23:14:02   SuricataStartup   35531   Suricata START for VoIP(10756_em1)...

Do not run the Service Watchdog package against Suricata or Snort.  It can cause crashing.  The Watchdog package does not properly account for the multiple Suricata instances (one running process per configured interface).  It also does not understand that Suricata stops and restarts itself as part of rule updates and such.  Remove Suricata from the Service Watchdog list and I bet it will work for you.  From your logs you can see that the Watchdog package is sending a START command to Suricata even while Suricata is already starting up.  Multiple start commands on the same interface spells trouble.

Edit:  I failed to notice another error in your suricata.log until later, so posting this edit with additional info.  This error is why your startup is failing:

Code: [Select]

1/12/2017 -- 23:08:59 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
1/12/2017 -- 23:08:59 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
1/12/2017 -- 23:08:59 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?


This suggested fix is in the error message.  Expand the size of your Stream Memcap.  I believe the default is either 32 MB or 64 MB.  Users with higher CPU core counts (or hyperthreading) frequently need 128 MB or even 256 MB of stream.memcap.

Bill
« Last Edit: December 05, 2017, 10:47:11 am by bmeeks »

Offline chiefgyk

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #5 on: December 05, 2017, 06:54:20 pm »
So I followed your advice, increased the memcap to 256MB ( 268,435,456 ) and it also had an update to 4.0.1_1 so now it is working as it should be. Thank you!

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #6 on: December 05, 2017, 08:09:33 pm »
So I followed your advice, increased the memcap to 256MB ( 268,435,456 ) and it also had an update to 4.0.1_1 so now it is working as it should be. Thank you!

You are welcome.  I saw in your suricata.log file that your CPU has 8 cores.  That's why a significant increase in Stream Memory is needed.  I think there were also some changes to that part of the Suricata binary from upstream back when the 4.0 series was released.

I still recommend strongly that you do not use Service Watchdog with Suricata (or Snort, for you Snort users reading this thread).

Bill

Offline micropone

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +2/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #7 on: December 09, 2017, 02:33:11 pm »
same problem here. sorry I just posted under beta forum for 2.4 i followed all advise from Bill.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #8 on: December 12, 2017, 03:15:21 pm »
same problem here. sorry I just posted under beta forum for 2.4 i followed all advise from Bill.

If boosting your stream memcap value did not help, post the output of the suricata.log file.  You can view under LOGS VIEW within the package GUI.  Any error will be in that file.

Bill

Offline micropone

  • Jr. Member
  • **
  • Posts: 51
  • Karma: +2/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #9 on: December 13, 2017, 06:05:51 pm »
               Crash report begins.  Anonymous machine information:

amd64
11.1-RELEASE-p6
FreeBSD 11.1-RELEASE-p6 #421 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 09:20:59 CST 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859


Filename: /var/crash/minfree
2048
            

this happens after I reinstall the whole package

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Suricata keeps crashing since 2.4.2 upgrade
« Reply #10 on: Yesterday at 10:43:04 am »
               Crash report begins.  Anonymous machine information:

amd64
11.1-RELEASE-p6
FreeBSD 11.1-RELEASE-p6 #421 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 09:20:59 CST 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning:  filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859


Filename: /var/crash/minfree
2048
            

this happens after I reinstall the whole package

What type of hardware is this?  Those errors indicate problems within the file system.  Another possibility, if you have recently upgraded your hardware and imported an old config, is the interface names have changed (the em1 part of the error path).  So for example if your NIC driver is now say igb1 instead of em1, then you will get this error.  To fix it you will need to either delete the interface and recreate it from scratch, or manually go into your config.xml file and change all the instances of the strings "em0" and "em1" to match whatever the new name is for your physical interfaces.