pfSense Gold Subscription

Author Topic: Pfsense has 40 percent free ram and 10 percent swap usage  (Read 441 times)

0 Members and 1 Guest are viewing this topic.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Pfsense has 40 percent free ram and 10 percent swap usage
« on: November 14, 2017, 05:18:11 am »
Quick question.  I'm not sure if this is the right category to post in.  Anyway.

I have a small VM with only about 1.25GB Ram allocated and only added package is Suricata.

It is running wonderfully and not showing any ill effects at all but for some reason it shows between 8% and 10% swap usage even though its not acting like any machine I've ever seen using swap.  No problems at all.  Just wondering if this is something weird or often seen?  I'm not used to seeing swap usage when nearly half the RAM is free on a machine.  Not complaining at all.  Its running fine.

last pid: 34413;  load averages:  1.01,  0.86,  0.72  up 0+02:00:17    11:43:12
41 processes:  1 running, 40 sleeping

Mem: 156M Active, 386M Inact, 82M Laundry, 486M Wired, 77M Buf, 94M Free
Swap: 410M Total, 30M Used, 380M Free, 7% Inuse


  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
77375 root          7  21    0  1000M   539M nanslp  0   8:20   1.86% suricata
 7123 root          1  27    0   263M 30064K piperd  0   0:00   1.86% php-fpm
14329 root          1  20    0 20348K  5308K select  1   2:10   0.00% openvpn
  306 root          1  20    0   261M 18460K kqread  1   0:10   0.00% php-fpm
43365 root          1  52   20 13084K   968K wait    0   0:08   0.00% sh
22324 root          5  24    0 13028K  2240K uwait   1   0:07   0.00% dpinger
31128 root          1  20    0 37704K  7644K kqread  0   0:06   0.00% nginx
17601 root          1  20    0 12696K  1996K bpf     0   0:04   0.00% filterlog
30837 root          1  20    0 37704K  4872K kqread  0   0:03   0.00% nginx
32150 root          1  20    0 24604K 12424K select  1   0:02   0.00% ntpd
16644 root          1  20    0 20348K  5252K select  0   0:02   0.00% openvpn
87639 root          1  20    0 10472K  2512K select  0   0:01   0.00% syslogd
92561 dhcpd         1  20    0 16648K  7696K select  1   0:01   0.00% dhcpd
10824 _dhcp         1  20    0 10528K  2052K select  1   0:01   0.00% dhclient
31597 root          1  52    0 12496K   664K nanslp  1   0:01   0.00% cron
23110 nobody        1  20    0 31868K  2860K select  1   0:00   0.00% dnsmasq
11435 root          1  52    0 39432K     0K wait    1   0:00   0.00% <login>
  320 root          1  40   20 19436K  1144K kqread  0   0:00   0.00% check_reload_status
« Last Edit: November 14, 2017, 05:45:04 am by kejianshi »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3103
  • Karma: +800/-0
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #1 on: November 14, 2017, 06:41:19 am »
Quick question.  I'm not sure if this is the right category to post in.  Anyway.

I have a small VM with only about 1.25GB Ram allocated and only added package is Suricata.

It is running wonderfully and not showing any ill effects at all but for some reason it shows between 8% and 10% swap usage even though its not acting like any machine I've ever seen using swap.  No problems at all.  Just wondering if this is something weird or often seen?  I'm not used to seeing swap usage when nearly half the RAM is free on a machine.  Not complaining at all.  Its running fine.

last pid: 34413;  load averages:  1.01,  0.86,  0.72  up 0+02:00:17    11:43:12
41 processes:  1 running, 40 sleeping

Mem: 156M Active, 386M Inact, 82M Laundry, 486M Wired, 77M Buf, 94M Free
Swap: 410M Total, 30M Used, 380M Free, 7% Inuse


  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
77375 root          7  21    0  1000M   539M nanslp  0   8:20   1.86% suricata
 7123 root          1  27    0   263M 30064K piperd  0   0:00   1.86% php-fpm
14329 root          1  20    0 20348K  5308K select  1   2:10   0.00% openvpn
  306 root          1  20    0   261M 18460K kqread  1   0:10   0.00% php-fpm
43365 root          1  52   20 13084K   968K wait    0   0:08   0.00% sh
22324 root          5  24    0 13028K  2240K uwait   1   0:07   0.00% dpinger
31128 root          1  20    0 37704K  7644K kqread  0   0:06   0.00% nginx
17601 root          1  20    0 12696K  1996K bpf     0   0:04   0.00% filterlog
30837 root          1  20    0 37704K  4872K kqread  0   0:03   0.00% nginx
32150 root          1  20    0 24604K 12424K select  1   0:02   0.00% ntpd
16644 root          1  20    0 20348K  5252K select  0   0:02   0.00% openvpn
87639 root          1  20    0 10472K  2512K select  0   0:01   0.00% syslogd
92561 dhcpd         1  20    0 16648K  7696K select  1   0:01   0.00% dhcpd
10824 _dhcp         1  20    0 10528K  2052K select  1   0:01   0.00% dhclient
31597 root          1  52    0 12496K   664K nanslp  1   0:01   0.00% cron
23110 nobody        1  20    0 31868K  2860K select  1   0:00   0.00% dnsmasq
11435 root          1  52    0 39432K     0K wait    1   0:00   0.00% <login>
  320 root          1  40   20 19436K  1144K kqread  0   0:00   0.00% check_reload_status

Suricata can be a bit RAM hungry, especially during rule updates when it basically loads two copies of the rules in memory (the current set and the new set) and then switches over to using the newest set and dumps the old ones from RAM.  Not saying that is definitely the cause here, but it is a suspect.  1.25 GB is really not what I would consider ideal for Suricata unless you are running a very limited rule set.  I would want at least 2 GB, and really more like 4 GB for most cases.

Bill

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #2 on: November 14, 2017, 06:59:14 am »
I think what you said makes perfect sense.  If it gets anywhere near 1.25gb its only for the tinyest split fraction of a second because I've never seen it exhaust memory.

So, it seems that whatever gets into that swap is very very very temporary.  Thanks for the Hypothesis. 

I'm going to keep running it this way unless I actually experience some ill effects. 

The inline IPS is working very well by the way.  I've had no hangs, drops or slow-downs. 


Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #3 on: November 14, 2017, 07:32:53 am »
BTW - I have a lot of rules...   I'm not worried about the fraction of a second pause that might occur every 12 or 24 hours, just as long as 99.99% of the time it is humming along fine.  The only thing I don't like about inline mode so far is I can't tell easily what is dropped and what is just alerted.

I know that things that generate a drop should turn red in the alerts. 

I also know that I told it to drop anything that generated an alert, so I'm wondering which of those two things it isn't doing / showing me.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #4 on: November 14, 2017, 08:34:49 am »
In general that also isn't anything to worry about. One of the common BSD mantras is "Free RAM is wasted RAM". The OS will swap things that are not in use so it can use RAM to speed up other processes, for caching, etc. It will reallocate things as needed when the requirements shift.


Now if you see no free RAM and your swap space is also nearly/completely full, then it's time to worry.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #5 on: November 14, 2017, 08:38:20 am »
Yep - Thats also my mantra.  And free cycles are wasted cycles...  I'm actually experimenting to see how small I can make it and have it work well.  I think I'm there already. 

Still wondering if those alerts are getting blocked since thats what I told it to do.  No red in the alerts so far. 

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile

Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2195
  • Karma: +202/-12
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #7 on: November 14, 2017, 10:50:43 am »
Mem: 156M Active, 386M Inact, 82M Laundry, 486M Wired, 77M Buf, 94M Free
Swap: 410M Total, 30M Used, 380M Free, 7% Inuse


  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
77375 root          7  21    0  1000M   539M nanslp  0   8:20   1.86% suricata

Looks like suricata is trying to use nearly 1GiB of memory. Your total memory usage may be low right now, but most OS's only page in data when the data is requested and only page out when there is not enough free space. If at any moment there was not enough memory, some of the data would get paged out, and assuming the data never got referenced again, it would forever live in swap.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #8 on: November 14, 2017, 10:57:22 am »
Yep - Saw that.  But throughput seems fine and its definitely throwing all the alerts you would expect. 

Still if anyone can enlighten me.  Is the alerted traffic being dropped like it should be?
« Last Edit: November 14, 2017, 11:20:08 am by kejianshi »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3103
  • Karma: +800/-0
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #9 on: November 14, 2017, 03:30:28 pm »
Yep - Saw that.  But throughput seems fine and its definitely throwing all the alerts you would expect. 

Still if anyone can enlighten me.  Is the alerted traffic being dropped like it should be?

If it's red on the ALERTS tab, then it was dropped.  If you look at the Suricata alerts log for the interface you will see the word "[drop]" as the first word of the alert message.  The Suricata binary does not log drops unless you enable the drop log in the EVE JSON output options.  However, those options are for exporting to an external log collector and the Suricata GUI package does not use them for display.

Bill

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #10 on: November 14, 2017, 05:06:49 pm »
OK, I guess I'm a little slow. 

So, when something like this appears in my lalerts ist:


Date   Pri   Proto   Class   Src   SPort   Dst   DPort   GID:SID   Description
11/14/2017
22:19:57   2   TCP   Misc Attack   46.17.46.77
     57669   192.168.10.14
     7777   1:2403325
     ET CINS Active Threat Intelligence Poor Reputation IP group 26
11/14/2017
22:19:57   2   TCP   Misc Attack   46.17.46.77
     57669   192.168.10.14
     7777   1:2402000
     ET DROP Dshield Block Listed Source group 1
11/14/2017
22:01:53   2   TCP   Misc Attack   109.248.9.241
     58625   192.168.10.14
     7777   1:2402000
     ET DROP Dshield Block Listed Source group 1

Its not dropped?  Even Though its on the "Block List"?

And even though this block is check?




OK - I mean, if its not dropping packets under those conditions, what must one do to get a packet dropped?
« Last Edit: November 14, 2017, 05:10:06 pm by kejianshi »

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #11 on: November 14, 2017, 05:35:57 pm »
Now, the very instant I take it out of "Inline" mode, it starts dropping that very same traffic that was listed in my alerts previously, but not highlighted in red.



So I guess my question is now, in inline mode is it dropping the packets and simply not showing them in red?

Or

Is it not dropping the packets in inline mode because there is some difference between inline and legacy mode that makes legacy mode work better?

Basically, it seems that legacy mode is blocking and inline mode isn't.  No crashes, no hangs, no weird symptoms.  No stopped suricata services.  I'm just not seeing the blocks in inline mode.
« Last Edit: November 14, 2017, 05:52:53 pm by kejianshi »

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #12 on: November 14, 2017, 06:05:19 pm »
I notice that when you are in legacy mode, you have a choice to drop source, destination or both. 

In inline mode you don't have that choice apparently.  Anyway.  Am I missing something?

Clearly inline mode would be the way to go, but without seeing some indication of drops somewhere, how can I feel confident with it?

I expected crashes and hangs and stopped suricata process if there is a problem.  Didn't expect it to just ignore the rules and settings.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4928
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #13 on: November 14, 2017, 07:17:02 pm »
I'm assuming there is a way to get expected results in inline mode with suricada.  That 1 check box is the only difference between a config that seems to block well and one that doesn't. 

I've seen this work and I've seen this stop the suricata process on other physical machines.

This is a VM and detection seems fine.  Just nothing is being blocked.

Could it be that in inline mode that the packet is only getting dropped if the rule specifically includes drop?

I see that "Block on Drop" option disappears when inline mode is enabled. 

"Checking this option will insert blocks only when rule signatures having the DROP action are triggered."

Does this get automagically and invisibly applied in inline mode?  If so, that would explain alot. 
« Last Edit: November 14, 2017, 07:21:08 pm by kejianshi »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3103
  • Karma: +800/-0
    • View Profile
Re: Pfsense has 40 percent free ram and 10 percent swap usage
« Reply #14 on: November 14, 2017, 07:57:20 pm »
I suspect you have a severe misconfiguration of your Suricata package when using Inline IPS Mode.  I believe you have a misunderstanding of rule names versus rule actions and the steps you must take as an admin to properly configure Inline IPS Mode.  If you have it configured like I think you do based on what you've posted, then you are in fact blocking nothing when using Inline IPS Mode.  Let me explain below.

Rules have a particular syntax they are written in.  The very first word of the rule text is the "action" keyword and it can have one of four possible values.  Those four values are "pass, drop, reject or alert".  The default value for all rules from the vendors is "alert".  Here is an actual rule from the Emerging Threats rule set --

Code: [Select]

alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE 180solutions (Zango) Spyware Local Stats Post"; flow:to_server,established; content:"/php/rpc_uci.php"; nocase; http_uri; reference:url,securityresponse.symantec.com/avcenter/venc/data/pf/adware.180search.html; reference:url,doc.emergingthreats.net/bin/view/Main/2003060; classtype:trojan-activity; sid:2003060; rev:5; metadata:created_at 2010_07_30, updated_at 2010_07_30;)


Notice the very first word of the rule is "alert".  This means this rule will generate alerts.  When you use Inline IPS Mode and you want rules to drop or "block" traffic, you must change the action keyword from "alert" to "drop".  You can do this manually, but most folks use the tools on the SID MGMT tab to accomplish that automatically in batches.  You would put the rules (either by SID or category name) you want to drop traffic in the dropsid.conf file.

Legacy Mode is totally different.  It is really a sort of kluge that leverages the pfSense firewall packet engine (pf) to block traffic.  It uses a custom plugin I wrote to insert offending IP addresses into a special firewall table.  More on that can be found here:  https://forum.pfsense.org/index.php?topic=135331.0.  Review the section describing how Legacy Mode and Inline IPS Mode operate.

So in Legacy Mode, every single alert will always generate a block.  Legacy Mode never looks at the rule action keyword.  With Inline IPS Mode, only rules where you have changed the action keyword from "alert" to "drop" will actually block traffic.  Decoder rules (the ones in your list of stream alerts) only generate alerts (by default).  So using Inline IPS Mode they will alert but not block.  However, using Legacy Mode, since every alert equals a block, those decoder rules generate blocks.

In your provided example you seem to imply that because you enabled the "ET DROP" rules category that those rules will generate drops.  They default to "alert" from the vendor.  You will need to put that category name in a dropsid.conf file on the SID MGMT tab, enable SID MGMT, and restart Suricata to have them actually drop traffic.

Bill

Edit:  fixed some typos
« Last Edit: November 15, 2017, 06:59:33 am by bmeeks »