Netgate Store

Author Topic: Upgrade Suricata 4.0.3  (Read 919 times)

0 Members and 1 Guest are viewing this topic.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3429
  • Karma: +898/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #15 on: January 16, 2018, 07:43:26 am »
Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.


*green shows idle, *yellow shows user

  • Upgrade to recent version / snort with barnyard2 was enabled
  • shutdown of snort barnyard2 & reenabled it
  • shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

Quote
CPU: 47.5% user,  0.0% nice, 16.8% system,  0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
32784 root             1 102    0   561M   489M CPU1    1  11:12  88.10% barnyard2
36442 root             1 102    0   555M   483M CPU2    2  11:11  85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance... due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(


Ok, figured it out - i am using barnyard2 with mysql and snorby - so the fact that some old references where in the table i resettet snorby and voila - barnyard2 is running under normal beheavior.

Barnyard2 needs a lot of "love" from its developer in the area of SQL statements.  It did that with me on Snort so often that I just abandoned using it completely.  I had the Barnyard2 and Snorby combination, but those high CPU utilizations while running flawed SQL statements finally exasperated me enough to abandon the Barnyard2 idea.

Bill

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3429
  • Karma: +898/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #16 on: January 16, 2018, 07:45:31 am »
Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

It is out.  What version of pfSense are you running?  It should be showing up for all 2.4.x versions.  I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill
« Last Edit: January 16, 2018, 09:07:04 am by bmeeks »

Offline The Sky Heart

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #17 on: January 16, 2018, 11:17:15 am »
Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 -- 18:57:24 - <Info> -- Invalid IP(\) parameter provided in Pass List, skipping...

i'm not sure where (\) came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards

I suspect this error message might be the result of fixing one of the bugs.  There was a logic flaw in how HOME_NET was populated in some circumstances.  That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration.  The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill


Hi Bill,
thanks a lot for the reply, and also sorry for the late update,

I'm not sure if I got your explanation about the Invalid IP(\), I do have some interfaces with no IP's but those interfaces are not enabled, also the Alias list I have is all Network Alias so even single IP's are added with /32, and if I check the list which is imported by suricata I can see that there is a record with \ which is different from (/), even if I remove it from that list it will be rewritten when I restart Suricata.

one more question please, does the below load look's ok? cause I have Suricata jumps to above 150% and it's only running on WAN interface with no extra settings apart from around 10 ET Rules list enabled, but I do admit that there is alot of traffic passing through this box.

the box is a Dell R430 Server with CPU E5-2603 v4 @ 1.70GHz and 32GB of Memory.

Code: [Select]
last pid: 25566;  load averages:  2.88,  2.85,  2.96                                                                             
41 processes:  2 running, 39 sleeping
CPU:  1.0% user, 14.9% nice,  5.3% system, 17.8% interrupt, 61.0% idle
Mem: 322M Active, 504M Inact, 2270M Wired, 414M Buf, 28G Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
69632 root         11  40   20   768M   687M nanslp  2  95.7H 100.06% suricata
36702 root          1  24    0 12696K  2820K bpf     0  26.6H   7.75% filterlog

 I also noticed that I have a lot of the below lines in suricata.log file:

Code: [Select]
16/1/2018 -- 18:18:56 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123136, ts.tv_usec:1399) flow_spare_q status(): 33% flows at the queue
16/1/2018 -- 18:19:05 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123145, ts.tv_usec:1896) flow_spare_q status(): 39% flows at the queue
16/1/2018 -- 18:19:09 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123149, ts.tv_usec:2995) flow_spare_q status(): 38% flows at the queue
16/1/2018 -- 18:19:14 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123153, ts.tv_usec:997415) flow_spare_q status(): 32% flows at the queue
16/1/2018 -- 18:19:15 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123155, ts.tv_usec:2263) flow_spare_q status(): 42% flows at the queue
16/1/2018 -- 18:19:19 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123159, ts.tv_usec:1779) flow_spare_q status(): 43% flows at the queue

is this something I have to worry about or need to fix?

Thanks again for the help.
« Last Edit: January 16, 2018, 11:21:51 am by The Sky Heart »

Offline RonpfS

  • Hero Member
  • *****
  • Posts: 750
  • Karma: +99/-2
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #18 on: January 16, 2018, 11:51:22 am »
It is out.  What version of pfSense are you running?  It should be showing up for all 2.4.x versions.  I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill
Under 2.3.5-RELEASE-p1 (amd64)  I have "Update available to 4.0.3"
2.3.5-RELEASE-p2 (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_3/Dev, suricata 4.0.4_1

Offline drewsaur

  • Jr. Member
  • **
  • Posts: 34
  • Karma: +1/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #19 on: January 16, 2018, 06:02:00 pm »
Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

It is out.  What version of pfSense are you running?  It should be showing up for all 2.4.x versions.  I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill

As of this evening, the update showed up. FWIW, I am using 2.4.x.

Drew
« Last Edit: January 17, 2018, 08:05:27 pm by drewsaur »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3429
  • Karma: +898/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #20 on: January 17, 2018, 12:13:58 pm »
Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 -- 18:57:24 - <Info> -- Invalid IP(\) parameter provided in Pass List, skipping...

i'm not sure where (\) came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards

I suspect this error message might be the result of fixing one of the bugs.  There was a logic flaw in how HOME_NET was populated in some circumstances.  That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration.  The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill


Hi Bill,
thanks a lot for the reply, and also sorry for the late update,

I'm not sure if I got your explanation about the Invalid IP(\), I do have some interfaces with no IP's but those interfaces are not enabled, also the Alias list I have is all Network Alias so even single IP's are added with /32, and if I check the list which is imported by suricata I can see that there is a record with \ which is different from (/), even if I remove it from that list it will be rewritten when I restart Suricata.

one more question please, does the below load look's ok? cause I have Suricata jumps to above 150% and it's only running on WAN interface with no extra settings apart from around 10 ET Rules list enabled, but I do admit that there is alot of traffic passing through this box.

the box is a Dell R430 Server with CPU E5-2603 v4 @ 1.70GHz and 32GB of Memory.

Code: [Select]
last pid: 25566;  load averages:  2.88,  2.85,  2.96                                                                             
41 processes:  2 running, 39 sleeping
CPU:  1.0% user, 14.9% nice,  5.3% system, 17.8% interrupt, 61.0% idle
Mem: 322M Active, 504M Inact, 2270M Wired, 414M Buf, 28G Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
69632 root         11  40   20   768M   687M nanslp  2  95.7H 100.06% suricata
36702 root          1  24    0 12696K  2820K bpf     0  26.6H   7.75% filterlog

 I also noticed that I have a lot of the below lines in suricata.log file:

Code: [Select]
16/1/2018 -- 18:18:56 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123136, ts.tv_usec:1399) flow_spare_q status(): 33% flows at the queue
16/1/2018 -- 18:19:05 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123145, ts.tv_usec:1896) flow_spare_q status(): 39% flows at the queue
16/1/2018 -- 18:19:09 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123149, ts.tv_usec:2995) flow_spare_q status(): 38% flows at the queue
16/1/2018 -- 18:19:14 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123153, ts.tv_usec:997415) flow_spare_q status(): 32% flows at the queue
16/1/2018 -- 18:19:15 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123155, ts.tv_usec:2263) flow_spare_q status(): 42% flows at the queue
16/1/2018 -- 18:19:19 - <Info> -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123159, ts.tv_usec:1779) flow_spare_q status(): 43% flows at the queue

is this something I have to worry about or need to fix?

Thanks again for the help.

Let me answer the question about flow emergency mode errors first.  You need to go read this document (scroll down to the section on Flow):  https://www.aldeid.com/wiki/Suricata/Suricata-yaml-configuration-file.  The short answer is on a high-traffic interface you are going to have to increase the flow memcap value substantially from the default.  Some info is posted at the link given, plus I suggest some additional Google searches for the terms "suricata flow" "suricata flow memcap" and "suricata flow emergency mode".  That search should net you additional information.

I'm sorry I confused the backslash and forward slash in my reply.  If you indeed have a backslash, and you can see that backslash listed when viewing the Pass List on the INTERFACE SETTINGS tab (using the View List button), then you need to test by removing things from the list to see where the backslash is coming from.  I would suspect an alias, if you have those configured, is the source of the backslash.

Bill

Offline The Sky Heart

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #21 on: January 25, 2018, 06:23:40 am »
Hi Guys,


Thanks you so much @bmeeks I was finally able to find out the issue with the passlist, it was this record ::1 causing it, it was manually added to the Alias list, after removing it didn't see the alert in logs anymore, I also fixed the flow alerts by increasing the memory to 512mb.

but I have another issue now, it seem's that after the daily rules update cron pfsense GUI can't figure out if Suricata is running or not, so it's showing in the GUI that it's stopped for the interface, but checking the service and alert and block logs I can see it's Suricata is running, doing a start from the gui shows it's running again.

is anyone else facing this issue?


Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3429
  • Karma: +898/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #22 on: January 25, 2018, 09:12:51 am »
Hi Guys,


Thanks you so much @bmeeks I was finally able to find out the issue with the passlist, it was this record ::1 causing it, it was manually added to the Alias list, after removing it didn't see the alert in logs anymore, I also fixed the flow alerts by increasing the memory to 512mb.

but I have another issue now, it seem's that after the daily rules update cron pfsense GUI can't figure out if Suricata is running or not, so it's showing in the GUI that it's stopped for the interface, but checking the service and alert and block logs I can see it's Suricata is running, doing a start from the gui shows it's running again.

is anyone else facing this issue?

With the latest update I made a change to the GUI code on the INTERFACES tab to make it dynamically show service status.  I was copying the same code over into the latest Snort update (since the two packages share most of the same GUI code) and found a couple of mistakes in the logic.  So could be that the new dynamic GUI code is getting fooled when Suricata restarts from a rules update. It is likely a cosmetic display issue only.

Now one other thing I've seen folks do that I have strongly recommended AGAINST is use the Service Watchdog package.  That package is great, but it does not understand how Suricata and Snort work with multiple processes (one per interface).  The package also does not understand how to cope with Suricata restarting after a rules update, and the instant Service Watchdog sees one of the Suricata processes die it immediately executes the shell script to restart.  That can really confuse things because Suricata is already restarting itself, so you wind up with multiple processes running on the same interface.

Bill

Offline The Sky Heart

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #23 on: January 25, 2018, 09:39:45 am »
Hi @bmeeks

Thank you so much again for the explanation, I actually added Suricata to watchdog service after noticing this issue, but as you mentioned it doesn't really know how Suricata service work so I was noticing the CPU usage is much higher everytime I manually restart Suricata from the interface tab, I removed it from watchdog now.

Thanks. 

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3429
  • Karma: +898/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #24 on: January 25, 2018, 01:08:37 pm »
Hi @bmeeks

Thank you so much again for the explanation, I actually added Suricata to watchdog service after noticing this issue, but as you mentioned it doesn't really know how Suricata service work so I was noticing the CPU usage is much higher everytime I manually restart Suricata from the interface tab, I removed it from watchdog now.

Thanks.

I will fix the GUI issue with showing the status correctly on the INTERFACES tab.  Probably will be sometime next week, though, before I can get it put together and posted.

Bill