The pfSense Store

Author Topic: ELK + pfSense 2.3 Working  (Read 10124 times)

0 Members and 1 Guest are viewing this topic.

Offline bubbawatson

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #15 on: March 07, 2017, 12:34:07 pm »
I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java  >:( ::))

I run elk stack on 1.5  ;D

Small office though. Thx for the info on auth.. I've been wondering how to do that.

Offline BrunoCAVILLE

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #16 on: May 02, 2017, 07:57:17 am »
I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

Offline BrunoCAVILLE

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #17 on: May 05, 2017, 08:46:47 am »
Eveything works well except the maps visualization, someone can help?

Offline BrunoCAVILLE

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #18 on: May 09, 2017, 06:38:57 am »
Up

Logstash stops after a few seconds (rising heap size didn't help).

Offline AMizil

  • Newbie
  • *
  • Posts: 22
  • Karma: +1/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #19 on: May 13, 2017, 03:26:57 pm »
I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

Status Menu - System Logs - Settings  - and jump to :  Remote log servers - and you can add another 2 Syslog Servers you have ; ex syslog-ng, Splunk etc

Offline ronv

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #20 on: June 21, 2017, 09:19:43 am »
Hi all,

trying to get this going with PFsense 2.3.4 and ELK 5.4 - all components are talking ok, and I can get the JSON Dashboard, Search and Visualization up and running - almost...:

- when I import the visualizations, Kibana complains that the tags geoip.country_name and geoip.city_name are not available.
- I checked 11-pfsense.conf (which I used from this site) against the spec at https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html, and there does not appear to be any issue with this - that is, it looks like those tags should be returned.

Anything else I could check, or logs I could provide?

kind regards

Ron

Offline hamed_forum

  • Jr. Member
  • **
  • Posts: 37
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #21 on: June 24, 2017, 11:57:25 pm »
the log send from pfsense where is save on elk?
i change the elk server and how to export import log on prvise server?

Offline pfBasic

  • Hero Member
  • *****
  • Posts: 1021
  • Karma: +132/-19
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #22 on: July 08, 2017, 03:35:07 am »
Any differences to get this running on 2.4.0 BETA?

Offline pfBasic

  • Hero Member
  • *****
  • Posts: 1021
  • Karma: +132/-19
    • View Profile
Re: ELK + pfSense 2.3 Working - On 2.4.0 BETA
« Reply #23 on: July 12, 2017, 11:54:06 am »
I finally got this up & running on pfSense 2.4.0 BETA with the help of AR15USR and some people on IRC.

Initially I was having trouble getting the Index Patterns to populate in the first step of Kibana. I had followed doktornotor's advice for setting up MaxMind. For whatever reason that didn't work for me so I just did it according to http://pfelk.3ilson.com/ and it worked.

Next, I had everything stable and logs being imported, but all logs were being tagged "_grokparsefailure" & "_geoip_lookup_failure" and since the pattern wasn't matching, it wasn't putting out any useful fields/information. This was also preventing me from importing the Visualizations.json due to not having the applicable fields available.

After way too much time troubleshooting and trying to figure out what was happening and why I was given some direction and pointed to the grok debugger by a kind IRC user. https://grokdebug.herokuapp.com/
For anyone looking to troubleshoot or modify their own grok pattern files, here's what I could make of the fields in 2.4.0 BETA's Rsyslog format. https://forum.pfsense.org/index.php?topic=133354.msg733494#msg733494
Run a pcap to see exactly what your pfSense box is sending to your ELK server.

It turned out that all I needed to do was change one character in /etc/logstash/conf.d/patterns/pfsense2-3.grok and reboot.

I changed line 16 (PFSENSE_LOG_DATA)
From:
Code: [Select]
PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),To:
Code: [Select]
PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
That's it, one "?".

After that, log files were parsing successfully, I refreshed my Index Pattern Field List to pull in all of the new fields, imported the Visualizations.json and opened up the Dashboard. All is working now on my single core atom with 2GB DDR2!


I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java  >:( ::))

I have this up and running (for home use) on an old netbook with an atom N450 (Pineview ~2010, single core 1.66GHz) with 2GB DDR2. I had to significantly lower RAM usage in the following two files to get it working. Currently using <1.5GB RAM, the OS is lubuntu with GUI service disabled. It's also running a Unifi controller. Dashboard is slow to load even for a small home network but it works! I couldn't justify buying anything to get an ELK stack for my home network.

Code: [Select]
/etc/elasticsearch/jvm.options
Code: [Select]
/etc/logstash/jvm.options
« Last Edit: July 12, 2017, 12:16:06 pm by pfBasic »

Offline idealanthony

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #24 on: July 16, 2017, 03:33:31 am »
Eveything works well except the maps visualization, someone can help?

@BrunoCAVILLE - I'm having the same problem as you did.  I used the revised visualization file due to the .keyword issue.  I've attempted to merge back in the country sections from the http://pfelk.3ilson.com/ visualization file, but still no luck.  Just wanted to know if you were able to identify/ resolve the issue?

https://forum.pfsense.org/index.php?topic=125376.0

Offline pfBasic

  • Hero Member
  • *****
  • Posts: 1021
  • Karma: +132/-19
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #25 on: July 16, 2017, 06:47:55 pm »
Did you refresh your fields list (Management / Index Patterns) after a number of your log files were successfully parsed?

If not, do that first then try to import the pf3lk visulization.json.

The import fails if you don't have the appropriate fields available.

Offline Steven.DeZalia

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +2/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #26 on: September 13, 2017, 08:57:25 pm »
I just wanted to add that the Kibana4 init script from the OP is no longer listed via a link as others were so I wanted to copy it here in text form as it did take me a moment to realize that the scripts were all included in a zip file in the op as well.

Kibana4 init script:

      #!/bin/sh
      #
      # /etc/init.d/kibana4 -- startup script for kibana4
      # bsmith@the408.com 2015-02-20; used elasticsearch init script as template
      # https://github.com/akabdog/scripts/edit/master/kibana4_init
      #
      ### BEGIN INIT INFO
      # Provides:          kibana4
      # Required-Start:    $network $remote_fs $named
      # Required-Stop:     $network $remote_fs $named
      # Default-Start:     2 3 4 5
      # Default-Stop:      0 1 6
      # Short-Description: Starts kibana4
      # Description:       Starts kibana4 using start-stop-daemon
      ### END INIT INFO
      
      #configure this with wherever you unpacked kibana:
      KIBANA_BIN=/opt/kibana4/bin
      
      PID_FILE=/var/run/$NAME.pid
      PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
      DAEMON=$KIBANA_BIN/kibana
      NAME=kibana4
      DESC="Kibana4"
      
      if [ `id -u` -ne 0 ]; then
              echo "You need root privileges to run this script"
              exit 1
      fi
      
      . /lib/lsb/init-functions
      
      if [ -r /etc/default/rcS ]; then
              . /etc/default/rcS
      fi
      
      case "$1" in
        start)
              log_daemon_msg "Starting $DESC"
      
              pid=`pidofproc -p $PID_FILE kibana`
              if [ -n "$pid" ] ; then
                      log_begin_msg "Already running."
                      log_end_msg 0
                      exit 0
              fi
      
              # Start Daemon
              start-stop-daemon --start --pidfile "$PID_FILE" --make-pidfile --background --exec $DAEMON
              log_end_msg $?
              ;;
        stop)
              log_daemon_msg "Stopping $DESC"
      
              if [ -f "$PID_FILE" ]; then
                      start-stop-daemon --stop --pidfile "$PID_FILE" \
                              --retry=TERM/20/KILL/5 >/dev/null
                      if [ $? -eq 1 ]; then
                              log_progress_msg "$DESC is not running but pid file exists, cleaning up"
                      elif [ $? -eq 3 ]; then
                              PID="`cat $PID_FILE`"
                              log_failure_msg "Failed to stop $DESC (pid $PID)"
                              exit 1
                      fi
                      rm -f "$PID_FILE"
              else
                      log_progress_msg "(not running)"
              fi
              log_end_msg 0
              ;;
        status)
              status_of_proc -p $PID_FILE kibana kibana && exit 0 || exit $?
          ;;
        restart|force-reload)
              if [ -f "$PID_FILE" ]; then
                      $0 stop
                      sleep 1
              fi
              $0 start
              ;;
        *)
              log_success_msg "Usage: $0 {start|stop|restart|force-reload|status}"
              exit 1
              ;;
      esac

Offline Steven.DeZalia

  • Jr. Member
  • **
  • Posts: 27
  • Karma: +2/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #27 on: September 13, 2017, 09:33:33 pm »
ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi...

How did you make out with this? I'm running into the same issue and it doesn't look like there was a resolution to the issue.

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #28 on: September 16, 2017, 08:18:15 am »
I'm starting to get this error when my Dashboard refreshes: "Courier Fetch: 28 of 325 shards failed."

I've noticed that I'm seeing yellow health and replications:

Code: [Select]
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-2017.09.09 yhdtjrKHQVycMOCfBmssWQ   5   0     347962            0    150.8mb        150.8mb
green  open   logstash-2017.09.10 Yg98wyN5SYav2dnc8OmFxA   5   0     359406            0      158mb          158mb
green  open   logstash-2017.09.11 mG66BkrDQOSnyJCqI5Ir-w   5   0     380644            0    164.2mb        164.2mb
green  open   logstash-2017.09.12 y26fNsoORtW6cSx1QcE7ZQ   5   0     390537            0    169.2mb        169.2mb
green  open   logstash-2017.09.13 MxyncENMRXqxLnMuJIw1rw   5   0     353464            0    152.2mb        152.2mb
yellow open   logstash-2017.09.14 Gp3dZ-uUTeWv9YIS4calhw   5   1     376975            0    163.5mb        163.5mb
yellow open   logstash-2017.09.15 cq8n4mYYSWGZZrzrb50B-g   5   1     392566            0    165.2mb        165.2mb
yellow open   logstash-2017.09.16 u7aF2fGSSmOJmJCU4odO5w   5   1     210728            0     94.5mb         94.5mb

Anyone know why this is happening all the sudden?
« Last Edit: September 16, 2017, 09:40:06 am by AR15USR »
_________________________

Release: pfSense 2.3.4

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #29 on: September 17, 2017, 12:01:56 pm »
Anyone able to give me any clues or places to start with the above?
_________________________

Release: pfSense 2.3.4