pfSense Gold Subscription

Author Topic: Upgrade Suricata 4.0.3  (Read 437 times)

0 Members and 1 Guest are viewing this topic.

Offline The Sky Heart

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Upgrade Suricata 4.0.3
« on: January 12, 2018, 12:30:26 pm »
Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

but I'm getting some errors in the logs:

12/1/2018 -- 18:56:58 - <Notice> -- This is Suricata version 4.0.3 RELEASE
12/1/2018 -- 18:56:58 - <Info> -- CPUs/cores online: 6
12/1/2018 -- 18:56:58 - <Info> -- HTTP memcap: 67108864
12/1/2018 -- 18:56:58 - <Notice> -- using flow hash instead of active packets
12/1/2018 -- 18:56:58 - <Error> -- [ERRCODE: SC_ERR_INVALID_YAML_CONF_ENTRY(139)] - malformed integer value for stream.max-sessions: ''


also I noticed this log:
12/1/2018 -- 18:57:24 - <Info> -- Invalid IP(\) parameter provided in Pass List, skipping...

i'm not sure where (\) came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards

Offline NRgia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #1 on: January 12, 2018, 01:46:15 pm »
Package updated with no issues but I get the same error in the logs:

2/1/2018 -- 20:16:46 - <Notice> -- This is Suricata version 4.0.3 RELEASE
12/1/2018 -- 20:16:46 - <Info> -- CPUs/cores online: 2
12/1/2018 -- 20:16:46 - <Info> -- Netmap: Setting IPS mode
12/1/2018 -- 20:16:46 - <Info> -- HTTP memcap: 67108864
12/1/2018 -- 20:16:46 - <Notice> -- using flow hash instead of active packets
12/1/2018 -- 20:16:46 - <Error> -- [ERRCODE: SC_ERR_INVALID_YAML_CONF_ENTRY(139)] - malformed integer value for stream.max-sessions: ''

I searched the forum and I found this from a long time ago https://forum.pfsense.org/index.php?topic=80224.msg437622#msg437622

The solution was a fix
« Last Edit: January 12, 2018, 02:00:22 pm by NRgia »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #2 on: January 12, 2018, 02:52:42 pm »
At the moment this error appears to be harmless.

Code: [Select]
[ERRCODE: SC_ERR_INVALID_YAML_CONF_ENTRY(139)] - malformed integer value for stream.max-sessions: ''

I can't find where in the binary source code it is coming from.  I did a "grep max-sessions *" in the source directory tree and found only a single line in one source file with the string, but that line prints a warning and not an error; and it prints a message about it being obsolete and does not say anything about it being malformed.

UPDATE EDIT: Never mind, I found the issue. It is a meaningless error, and I will fix it soon.  It is a deprecated parameter and needs to be removed from the /usr/local/pkg/suricata/suricata_yaml_template.inc file.  If you want to implement your own fix, edit that file and remove line number 220.  It reads like this:

Code: [Select]

  max-sessions: {$stream_max_sessions}


Bill
« Last Edit: January 12, 2018, 03:07:28 pm by bmeeks »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #3 on: January 12, 2018, 02:57:25 pm »
Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 -- 18:57:24 - <Info> -- Invalid IP(\) parameter provided in Pass List, skipping...

i'm not sure where (\) came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards

I suspect this error message might be the result of fixing one of the bugs.  There was a logic flaw in how HOME_NET was populated in some circumstances.  That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration.  The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill

Offline NRgia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #4 on: January 12, 2018, 03:42:20 pm »
Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 -- 22:24:01 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #5 on: January 12, 2018, 06:30:41 pm »
Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 -- 22:24:01 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text.  This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file.  It can only write to the pfSense system log.  From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

Offline micropone

  • Jr. Member
  • **
  • Posts: 55
  • Karma: +2/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #6 on: January 13, 2018, 01:31:10 pm »
I got an different issue


Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #7 on: January 13, 2018, 01:37:36 pm »
I got an different issue

This is from a problem with the pkg utility itself that is used to install packages on pfSense.  I have never seen this error in my testing using pfSense virtual machines.  Do you have /var on a RAMDISK by chance?

You can also try to remove the package and then install it again.  I believe that uses a different set of routines within the pkg utiilty.

Bill

Offline NRgia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #8 on: January 14, 2018, 09:00:59 am »
Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 -- 22:24:01 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text.  This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file.  It can only write to the pfSense system log.  From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

You're correct, the log is from /var/log/suricata/suricata_interface_name/suricata.log , and I think it's about the rules that are not parsed correctly, but I did not saw this error with previous version, and the selected rules are almost the same(taking in account the updates):

12/1/2018 -- 20:16:57 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

I attached an unaltered log if you have time to look over it.

If you say it's nothing to worry about, then I can rest easy.

Thank you for your time.

OP @The Sky Heart , sorry for hijacking the thread, did you solve your issue? 

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #9 on: January 14, 2018, 12:52:14 pm »
Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 -- 22:24:01 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text.  This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file.  It can only write to the pfSense system log.  From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

You're correct, the log is from /var/log/suricata/suricata_interface_name/suricata.log , and I think it's about the rules that are not parsed correctly, but I did not saw this error with previous version, and the selected rules are almost the same(taking in account the updates):

12/1/2018 -- 20:16:57 - <Error> -- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

I attached an unaltered log if you have time to look over it.

If you say it's nothing to worry about, then I can rest easy.

Thank you for your time.

OP @The Sky Heart , sorry for hijacking the thread, did you solve your issue?

The error message does not give the SID of the rule where the PCRE parsing failed, so troubleshooting that is a challenge.  However, remember that each update of the Suricata binary means some things changed in the source code.  There are two parts of the Suricata package.  The main piece is the command-line driven binary.  That code is wholly and totally supported by the upstream team at Suricata.org.  The secondary piece of the Suricata packge on pfSense is a GUI program that interacts with the user.  The GUI code ultimately just winds up writing the suricata.yaml file used to send configuration information to the binary.

So why am I going into this detail?  It is to make sure users understand that Suricata, and Snort, along with most every other package on pfSense, has an underlying binary piece that does the real work; and a GUI piece used to gather configuration input and pass it to the underlying binary.  Sometimes, based on questions and bug reports I see, it seems users conflate the two pieces of a package and make them into one.  For example, I can't do anything about issues within the Suricata binary itself.  This particular error is one of those.  It is a problem in the binary that probably got introduced in version 4.0.3 of the binary.  Previously we were running the older 4.0.1 version of the Suricata binary.  Sometimes changes in the binary require changes in the GUI, though.  The other error posted about an invalid integer value for "stream.max-sessions" is one of those.  The newer 4.0.3 binary no longer wants to see that parameter, but the older version did still want to see it.

Bill

Offline NRgia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #10 on: January 14, 2018, 01:36:01 pm »
Actually this is the idea behind pfSense and GUI packages. The user can do the config by CLI also, but it's more work.

IMHO in SDLC when you test integrated pieces of code you don't always know(from returned errors) were the root cause is, if you're not the one who maintained the GUI or the binary. It could be with a bad config from my part, or maybe the config that is passed to the binary is misinterpreted, due to code change in GUI or binary.

I consider you responded my question, by pointing which of the pieces is the culprit.

For binary issues I can take it on the Suricata redmine, or with the maintainer on Fresh Ports.

Thank you
« Last Edit: January 14, 2018, 03:23:57 pm by NRgia »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3225
  • Karma: +835/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #11 on: January 15, 2018, 09:24:58 am »

I consider you responded my question, by pointing which of the pieces is the culprit.

For binary issues I can take it on the Suricata redmine, or with the maintainer on Fresh Ports.

Thank you

Yes, my opinion is the binary is the cause of your error message.  I recommend posting a bug report on Suricata redmine as the Fresh Ports maintainer is really just using the code from Suricata upstream with no changes.  If it is a binary bug, it needs to be reported and fixed by the actual Suricata team.  I guess there is an outside chance it's a problem with the syntax of some rule, but since the GID:SID is not provided in the error it will be hard to locate the offending rule.  Maybe you could try "grep" with some of the content text from the error and locate the rule ???.  You can find the actual file of enabled rules here:

/usr/local/etc/suricata/suricata_interface/rules/suricata.rules

where "interface" will be the physical interface name on your firewall plus a UUID random number.

Bill

Offline drewsaur

  • Jr. Member
  • **
  • Posts: 31
  • Karma: +1/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #12 on: January 15, 2018, 06:51:08 pm »
Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

Offline fgro

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #13 on: January 16, 2018, 06:09:02 am »
Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.


*green shows idle, *yellow shows user

  • Upgrade to recent version / snort with barnyard2 was enabled
  • shutdown of snort barnyard2 & reenabled it
  • shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

Quote
CPU: 47.5% user,  0.0% nice, 16.8% system,  0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
32784 root             1 102    0   561M   489M CPU1    1  11:12  88.10% barnyard2
36442 root             1 102    0   555M   483M CPU2    2  11:11  85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance... due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(
« Last Edit: January 16, 2018, 06:37:38 am by fgro »

Offline fgro

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Upgrade Suricata 4.0.3
« Reply #14 on: January 16, 2018, 07:24:38 am »
Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.


*green shows idle, *yellow shows user

  • Upgrade to recent version / snort with barnyard2 was enabled
  • shutdown of snort barnyard2 & reenabled it
  • shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

Quote
CPU: 47.5% user,  0.0% nice, 16.8% system,  0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
32784 root             1 102    0   561M   489M CPU1    1  11:12  88.10% barnyard2
36442 root             1 102    0   555M   483M CPU2    2  11:11  85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance... due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(


Ok, figured it out - i am using barnyard2 with mysql and snorby - so the fact that some old references where in the table i resettet snorby and voila - barnyard2 is running under normal beheavior.