Netgate SG-1000 microFirewall

Author Topic: Snort + SG-3100 = exited on signal 10  (Read 1059 times)

0 Members and 1 Guest are viewing this topic.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #15 on: November 06, 2017, 07:25:55 am »
My suspicion is the differences between ARM CPU architecture and Intel/AMD CPU architecture and instruction sets are the root cause here, and some sequence of steps during parsing of rules triggers the Signal 10.  If you are familiar with C programming and have access to the Snort binary source code, you will see lots of #ifdef types of statements in the code to detect various environments and adjust for their peculiarities.  These are mostly all related to compilation for different operating systems.  Some detective work would be required to figure out what is needed to compile code reliably for an ARM system and add the appropriate #ifdef statements to bound the changes.  A key step in doing that is having an ARM platform to test with.  I don't have one, and to my knowledge I can't emulate one reliably in a VMware virtual machine either.

Bill

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #16 on: November 06, 2017, 09:56:07 am »
UPDATE

An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware.  Give me a little time to get my environment set up and then do some investigation and testing.

Bill

Offline Valiant

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #17 on: November 13, 2017, 03:49:03 am »
UPDATE

An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware.  Give me a little time to get my environment set up and then do some investigation and testing.

Bill


Hi Bill,

Thanks, any progress on this ?. I dug up an old HP N40L Microserver, installed a NC360T dual NIC card and installed PFsense to compare to the Netgate. So far I have got good results with OpenVPN working nicely and Snort service running reliably with the same oinkcode and ruleset selected.

The N40L has an AMD Turion processor which seems compatible. Not sure which device has more grunt but so far its working ok for me. I do however want to retire this box and stick to the SG-3100 to save on power.



Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #18 on: November 13, 2017, 08:28:37 am »
UPDATE

An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware.  Give me a little time to get my environment set up and then do some investigation and testing.

Bill


Hi Bill,

Thanks, any progress on this ?

Haven't found the offending code yet, but I do have the test/debugging environment set up.  It's weird.  When I run the standard Snort binary I get the crash pretty much immediately upon startup.  However, when I run a Snort binary compiled with debugging symbols it runs just fine and does not crash!  Scratching my head over this one ...  ???.

Bill

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11914
  • Karma: +468/-15
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #19 on: November 13, 2017, 08:42:20 am »
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour.  ;)

Steve

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #20 on: November 13, 2017, 11:41:42 am »
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour.  ;)

Steve

Yep.  Turning on debugging symbols turns off all optimizations done by the compiler.  So at least there is a hint there that maybe something in the compiler optimizations are the cause.  In some subsequent runs I was able to produce a Signal 10 crash using the Snort binary with debug symbols ... but only once so far.  The non-debugging version crashes every single time.

Bill

Offline mcury

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #21 on: November 16, 2017, 12:35:40 pm »
Hi, any luck on this matter?

I`ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I`ll try to help too.

I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.

The question is, which ruleset they have tested? Was VRT rules tested?

If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.

obs: sorry for my english, it`s not my native language.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #22 on: November 16, 2017, 04:58:56 pm »
Hi, any luck on this matter?

I`ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I`ll try to help too.

I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.

The question is, which ruleset they have tested? Was VRT rules tested?

If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.

obs: sorry for my english, it`s not my native language.

Working on it along with one of the pfSense kernel developers.  It's a complex problem, and there are several errors likely in the Snort binary's source code.  Some things were done in the code that are not good programming practice, but Intel processors hide the issue because they silently fix the problem.  ARM processors like the armv7 used in the SG-3100 do not silently fix the problem.  The issue is unaligned memory access done by portions of the Snort binary code.

We have been able to get Snort to run without the Signal 10 error, but it's not properly decoding some of the TCP packets.  It's messing up TCP sequence and ACK numbers for one thing.

Bill

Offline skilbjo

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #23 on: November 24, 2017, 06:04:21 pm »
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it... I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.

Good luck @bmeeks ! May the programming gods bless you on this one!

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3173
  • Karma: +821/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #24 on: November 25, 2017, 11:48:04 am »
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it... I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.

Good luck @bmeeks ! May the programming gods bless you on this one!

Thanks!  This is a tough nut to crack.  It's not an illegal instruction that's causing the Signal 10 in this case.  Instead, it's a problem with something called unaligned memory access.  You can Google that term for details about what it is.  It's down all the way to the register level inside the CPU and how hardware memory access has to work.  The root cause is what many consider poor or bad form C language programming practice when using pointers to reference data in memory.  Intel x86 CPUs swallow these kinds of programmer issues and auto-correct them.  In the old days, before tons of CPU on-die cache memory and all the fancy instruction execution pipelines of modern CPUs, there was a peformance penalty each time the CPU "fixed up" a C programmer's mistake.  Not so much anymore, though.  Modern Intel CPUs just basically instantly fix-up the unaligned memory access and there is no perceivable performance penalty.  Thus there has not been a push to fix these problems in legacy C programming code.  However, other CPUs such as the armv7 used in the SG-3100 don't perform these auto-fixups by default.  So you get the errors.  The preferred fix is to find all the poor programming practices in the C code and fix them at the source.  That is easier to say that it is to actually do... :(.  We're still working on it.  The problems are within sections of the Snort binary and have nothing to do with the GUI package.

Bill

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11914
  • Karma: +468/-15
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #25 on: November 28, 2017, 09:03:19 am »
The preferred fix is to find all the poor programming practices in the C code and fix them at the source.  That is easier to say that it is to actually do... :(.

Wow, I feel that pain.  :-\

Sending you good vibes!  :)

Steve

Offline psylenced

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #26 on: December 02, 2017, 12:45:41 am »
Just jumping into this thread to say that I have the same issue, and await a fix hopefully soon.

I originally thought I had misconfigured something.

Offline RossCaryNC

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #27 on: December 03, 2017, 08:48:12 pm »
OP Here.

Thank you guys for looking into this. I look forward to a fix.

-Ross

Offline mcury

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #28 on: December 07, 2017, 09:53:32 am »
Hey guys, until we wait for the fix, you can use Suricata instead, not a complete replacement due to the lack of appid in my opnion, but it works.
The categories mentioned below worked fine.

The only category I got errors inside Suricata.log was from malware-cnc, so I've disabled it.

I'm currently using the VRT paid rules, with the following categories:

snort_blacklist.rules
snort_browser-chrome.rules
snort_browser-firefox.rules
snort_browser-plugins.rules
snort_file-multimedia.rules
snort_file-office.rules
snort_file-pdf.rules
snort_malware-backdoor.rules
snort_os-windows.rules

Disabled the stream-events rules due to huge false positives.
And a few of the decoder-events due to same reason.

If you guys have any doubts, just ask.

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11914
  • Karma: +468/-15
    • View Profile
Re: Snort + SG-3100 = exited on signal 10
« Reply #29 on: December 10, 2017, 04:38:00 pm »
Hmm, interesting. Is Suricata still running OK for you?

I initially thought this also but found Suricata crashed out after some time. However I'm re-testing it now and it's still running....so far.

Steve