pfSense Support Subscription

Author Topic: No Alerts using Suricata inline mode.  (Read 470 times)

0 Members and 1 Guest are viewing this topic.

Offline dcol

  • Full Member
  • ***
  • Posts: 193
  • Karma: +8/-5
    • View Profile
No Alerts using Suricata inline mode.
« on: November 11, 2017, 03:21:17 pm »
I setup a new PFsense box which seems to function normally except I do not get any alerts using Suricata Inline Mode. When using Legacy mode with 'Set Legacy to Block On Drop Only' checked, I see the proper alerts and blocks being generated. I setup SID mgmt with drop and disable files and did a rebuild. Enabled Auto SID State Mgmt. Using the WAN and LAN interface in Suricata. Firewall logs are normal.

Not using load balancing or traffic shaping. Only other installed package is Cron. Only have 1 WAN and 1 LAN interface active.
System is a Supermicro 5018A-FTN4 with four standard built in Intel NIC's using igb drivers. All offloading is disabled.
Not a NIC issue. Tried an Intel i210T1 card, which worked in another pfsense box using inline.

Tried reinstalling Suricata with no change. There are no suppress rules. No errors in the Suricata log. No netmap messages in the console.

Is there some other setting I may be missing?
« Last Edit: November 11, 2017, 04:52:56 pm by dcol »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #1 on: November 11, 2017, 06:00:47 pm »
I will ask the obvious questions first --

1.  Where are you looking for "alerts and blocks"?  When Inline IPS Mode is used, the BLOCKED tab will always be empty.  That tab is not part of the Inline IPS Mode of operation.  Instead, look on the ALERTS tab for the interface and any alerts that resulted in drops will be highlighted in the "danger" color for the pfSense theme (that will be red using the default theme).

2.  Is your configuration exactly the same with the sole exception being the toggling of Blocking Mode from Legacy to Inline?

3.  Is Suricata running on the same interface in both modes?

You  can easily test Suricata by loading the Emerging Threats Open Rules and enabling the "ET-Scan" category (I forget the exact name, but it has "scan" in it so you can find it).  Next, using a Kali Linux virtual machine or any other machine on the the network with Suricata on it and scan the firewall address with nmap.  You will get some hits for VNC and I think MS-SQL server if I recall correctly.

Bill

Offline dcol

  • Full Member
  • ***
  • Posts: 193
  • Karma: +8/-5
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #2 on: November 12, 2017, 11:56:10 am »
1. Yes, I do know about the blocks tab not showing using inline. I have used inline in the past on another system and I am familiar with its operation.

2.Yes, same config

3. Yes same interface. Even tried changing the WAN interface to an i210T1 NIC I have used with inline in the past.

I have done the scan with ET-Scan category enabled and used Nmap/Zenmap from another system. The firewall picks up the scans and so does Suricata when in Legacy mode. No Alerts when using inline.

One difference with this system is it has 8 cores, so I did up the Stream Memory Cap to 128MB. Using the default cap size generated an error and Suricata would not start up. It starts up fine now with no errors.

I am out of ideas. I even duplicated the entire configuration from a working box. What's up?

Offline teamits

  • Jr. Member
  • **
  • Posts: 35
  • Karma: +4/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #3 on: November 20, 2017, 02:23:55 pm »
dcol, what pfSense version are you using?  I just upgraded a router yesterday from 2.3.4 to 2.4.1, which upgraded the Suricata package, and this morning changed Suricata to Inline mode on WAN (em0), along with setting up the dropsid.conf file.  WAN Rules tab shows the action is Drop for those rules.  We had been getting an alert every couple minutes but have had none in the last 3 hours.  I've stopped and started Suricata, but haven't rebooted the router yet today.

I do see a few SC_ERR_INVALID_SIGNATURE errors in the suricata.log file but those exist yesterday and it was working in legacy mode overnight.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #4 on: November 20, 2017, 05:38:54 pm »
1. Yes, I do know about the blocks tab not showing using inline. I have used inline in the past on another system and I am familiar with its operation.

2.Yes, same config

3. Yes same interface. Even tried changing the WAN interface to an i210T1 NIC I have used with inline in the past.

I have done the scan with ET-Scan category enabled and used Nmap/Zenmap from another system. The firewall picks up the scans and so does Suricata when in Legacy mode. No Alerts when using inline.

One difference with this system is it has 8 cores, so I did up the Stream Memory Cap to 128MB. Using the default cap size generated an error and Suricata would not start up. It starts up fine now with no errors.

I am out of ideas. I even duplicated the entire configuration from a working box. What's up?

So with the identical configuration except for the Blocking Mode changing from "Legacy Mode" to "Inline IPS Mode" on the same hardware box, you get alerts with Legacy Mode but no alerts on the same traffic with Inline IPS Mode?

I'm inclined to say that almost can't happen based on how I understand the underlying Suricata binary source code.  Not saying you are wrong, but I really can't imagine a scenario where that can happen.  The only change between Legacy Mode and Inline IPS Mode is the use of Netmap for Inline.  However, Netmap usually either loads and works or it kills the network at startup.  I've never seen a report like this where it fails to pass traffic for inspection but does not break the network.

Edit:  I will add one more question -- what kind of CPU platform do you have?  Is it an ARM or Intel CPU?

Bill
« Last Edit: November 20, 2017, 05:44:40 pm by bmeeks »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #5 on: November 20, 2017, 05:43:25 pm »
dcol, what pfSense version are you using?  I just upgraded a router yesterday from 2.3.4 to 2.4.1, which upgraded the Suricata package, and this morning changed Suricata to Inline mode on WAN (em0), along with setting up the dropsid.conf file.  WAN Rules tab shows the action is Drop for those rules.  We had been getting an alert every couple minutes but have had none in the last 3 hours.  I've stopped and started Suricata, but haven't rebooted the router yet today.

I do see a few SC_ERR_INVALID_SIGNATURE errors in the suricata.log file but those exist yesterday and it was working in legacy mode overnight.

Those SC_ERR_INVALID_SIGNATURE errors are to be expected if you are using some of the Snort VRT rules with Suricata.  Suricata does not recognize some of the newer rule options and keywords that Snort uses.  Suricata prints the error for those rules, does not load them, and proceeds to load the next rule.

What hardware are you running?  Does it by chance have an ARM CPU (such as the new Netgate SG-3100)?  Problems have been discovered within the Snort binary that prevent it from working on the new ARM hardware platforms.  I've not had a similar report about Suricata unless yours turns out to be the first incidence.

Bill

Offline teamits

  • Jr. Member
  • **
  • Posts: 35
  • Karma: +4/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #6 on: November 21, 2017, 12:23:10 am »
We do have a client with the new Netgate 3100 and have Suricata running on it in legacy mode.

Our hardware is an older PC:
Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
2 CPUs: 1 package(s) x 2 core(s)
2 GB RAM but only 37% used

It's quite possible I missed something but I tried to follow your posts here so I'm not sure what.  I'm not seeing errors, just not seeing alerts.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #7 on: November 21, 2017, 07:22:31 am »
We do have a client with the new Netgate 3100 and have Suricata running on it in legacy mode.

Our hardware is an older PC:
Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
2 CPUs: 1 package(s) x 2 core(s)
2 GB RAM but only 37% used

It's quite possible I missed something but I tried to follow your posts here so I'm not sure what.  I'm not seeing errors, just not seeing alerts.

Need some clarification on details please.  There are some subtle differences between your two posts that left me a little confused --

1.  This same hardware (SG-3100, I assume) was working fine with Legacy Mode Suricata on pfSense 2.3.x?

2.  Do you know what version of Suricata you were running prior to the upgrade?  Version 4.0.0 has been out for quite some time.

3.  You upgraded the same hardware to 2.4.x and the Suricata package updated automatically (if so, to what version?) and now does not produce alerts.

4.  Do you now get no alerts with either mode (Legacy or Inline), or just no alerts with Inline IPS Mode?

Bill

Offline teamits

  • Jr. Member
  • **
  • Posts: 35
  • Karma: +4/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #8 on: November 21, 2017, 09:44:49 am »
Hi Bill,

(to perhaps clarify I'm not the OP, I'm "me too"-ing the thread)

Ignore the SG-3100, that's at a client site.

We have the PC hardware I detailed.  It works (alerts and blocks) if I switch to Legacy mode.  In inline mode I get no alerts.

I am not sure about the Suricata version.  It was from when we upgraded to 2.3.4 in I think early July.

I am seeing no alerts in Inline mode and alerts in Legacy mode.

If netmap wasn't working right in the driver what would be the symptom?  No traffic at all?  (the wan NIC is detected as em0, the LAN - w/o Suricata is bge0)

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #9 on: November 21, 2017, 10:40:47 am »
Hi Bill,

(to perhaps clarify I'm not the OP, I'm "me too"-ing the thread)

Ignore the SG-3100, that's at a client site.

We have the PC hardware I detailed.  It works (alerts and blocks) if I switch to Legacy mode.  In inline mode I get no alerts.

I am not sure about the Suricata version.  It was from when we upgraded to 2.3.4 in I think early July.

I am seeing no alerts in Inline mode and alerts in Legacy mode.

If netmap wasn't working right in the driver what would be the symptom?  No traffic at all?  (the wan NIC is detected as em0, the LAN - w/o Suricata is bge0)

I would expect that if Netmap was an issue there would be no connectivity (i.e., the network would be broken).

Last time I tested on a VM, Inline IPS mode worked fine.  There are also a number of other users here that make use of that mode and are not reporting any problems.

Some things to check -- are $HOME_NET and $EXTERNAL_NET properly defined when Inline IPS Mode is enabled.  You can view the values on the INTERFACE SETTINGS tab for the interface where Suricata is enabled.  Most all rules depend on the HOME_NET and EXTERNAL_NET variables being correctly configured in order for the rules to trigger.

Have you tried throwing some scans at the machine in question from nmap or a simlar tool?

Examine the SID MGMT log file to see what the tally is for rules processed by that logic.  Does it show it really changed the rules you think it did? (Go by numbers, of impacted rules since that is what is displays in the log).

You can examine the actual rules file used by an interface here using the DIAGNOSTICS > EDIT FILE menu option in pfSense --

/usr/local/etc/suricata/suricata_xxxxx/rules/suricata.rules

Note -- the xxxxx part of the path will be a GUID along with the physical interface name.

Open that file and check which rules are enabled and what their actions are.

Bill


Offline teamits

  • Jr. Member
  • **
  • Posts: 35
  • Karma: +4/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #10 on: December 21, 2017, 03:01:31 pm »
Long delay on my end here but I finally got back to this today.  I upgraded the Suricata package 4.0.0_2 -> 4.0.1_1 which has the side effect of wiping out prior alerts and logs, apparently.  The sid_changes.log file has:

Processing drop_sid file: dropsid.conf
    Parsed 19356 potential SIDs to match from the provided list of tokens.
    Found 19356 matching SIDs in the active rules.
    Changed state for 19356 SIDs to 'drop'.

...so it seems like it's finding all the rules.

Should "Promiscuous Mode" be checked/on for Inline mode?  In a quick test that didn't seem to matter.  (it was On for Legacy mode)

/usr/local/etc/suricata/suricata_57335_em0/rules/suricata.rules exists...7 MB.  /var/log/suricata/suricata_em057335/alerts.log exists, size 0.

I've not tried scanning ourselves, but the flow of alerts in Legacy mode is pretty constant in Legacy mode.  Per https://forum.pfsense.org/index.php?topic=108010.0 I read your post that in Inline mode Suricata is before the firewall (Internet->Suricata->firewall) so it should still be seeing all traffic...?

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3216
  • Karma: +835/-0
    • View Profile
Re: No Alerts using Suricata inline mode.
« Reply #11 on: December 22, 2017, 10:45:50 am »
Are you 100% positive the Suricata binary continues to run after the initial startup?  It's just not really logical that it runs with the same rule set yet fails to alert on the same traffic as it does in Legacy Mode.  It could be that something happens during the last stages of startup that kills the Suricata process.  If true, then I could make sense of no alerts.

Post up the actual suricata.log file's contents from a startup in Inline IPS Mode.  You can find it under the LOGS VIEW tab.

Bill