The pfSense Store

Author Topic: ELK + pfSense 2.3 Working  (Read 12634 times)

0 Members and 1 Guest are viewing this topic.

Offline ando1

  • Jr. Member
  • **
  • Posts: 25
  • Karma: +5/-0
    • View Profile
ELK + pfSense 2.3 Working
« on: November 10, 2016, 04:55:09 am »
OK after a lot of reading and researching, I have successfully created an ELK stack and can monitor my pfsense 2.3 firewall. I am posting the steps I used below along with the files needed. You may need to modify some of the files to fit your IP address and environment. Also I posted the reference links I used to create the steps.
 
I wanted to give credit to the sites that I got most of this information from as it helped me in figuring out how to make this work.

UPDATE 11/17: I also found this site and was able to get version 5 working with Ubuntu server 16+: http://pfelk.3ilson.com/

My original post on Reddit: https://www.reddit.com/r/PFSENSE/comments/5axoaj/getting_elk_to_work_with_pfsense_23/

Reference Links:
http://secretwafflelabs.com/2015/11/06/pfsense-elk/
https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/
https://elijahpaul.co.uk/updated-monitoring-pfsense-logs-using-elk-elasticsearch-logstash-kibana-part-1/

Prerequisites:
   • Ubuntu 14.04 Desktop - http://releases.ubuntu.com/14.04/
   • Kibana 4.5.4
   • Logstash 2.2.4
   • Elasticsearch 2.4.0
   • pfSense 2.3.2

Files Needed (also in attached zip file)
(You will need to modify some of these to fit your environment)
   • Kibana4 init script
   • Pfsense 2.2+ grok file - http://secretwafflelabs.com/files/pfsense2-2.grok
   • 02-syslog-input.conf - http://secretwafflelabs.com/files/02-syslog-input.conf
   • 20-syslog-filter.conf - http://secretwafflelabs.com/files/20-syslog-filter.conf
   • 81-pfsense-filter.conf - http://secretwafflelabs.com/files/81-pfsense-filter.conf
   • 99-elasticsearch-output.conf - http://secretwafflelabs.com/files/99-elasticsearch-output.conf
   • Dashboard - http://secretwafflelabs.com/files/Firewall_External_Dash.json
   • Visualizations Export - http://secretwafflelabs.com/files/Firewall_External_Visual.json
   • Saved Searches Export  - http://secretwafflelabs.com/files/export.json   
   

Steps:
   1. In order to be able to run the below commands as root, log into the Ubuntu desktop and type sudo - i
   2. Install Java
Code: [Select]
apt-get remove --purge openjdk*

add-apt-repository -y ppa:webupd8team/java

apt-get update

apt-get -y install oracle-java8-installer
   
   3. Verify java version
Code: [Select]
java -version      
   Output
   java version "1.8.0_111"
        Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
        Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
      
   4. Install ElasticSearch
   
Code: [Select]
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.0/elasticsearch-2.4.0.deb


dpkg -i elasticsearch-2.3.4.deb
      
   5. Download and install Logstash
   
Code: [Select]
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.2.4-1_all.deb

dpkg -i logstash_2.2.4-1_all.deb
      
   6. Create a patterns directory for Geo_IP
   
Code: [Select]
cd /etc/logstash/conf.d

mkdir patterns
      
   7. Create pfsense grok file
   
Code: [Select]
cd /etc/logstash/conf.d/patterns

nano pfsense2-2.grok
   
   8. Download the GEO_IP database
   
Code: [Select]
cd /etc/logstash

curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"

gunzip GeoLiteCity.dat.gz
   
   
   9. Create the logstash conf files
      
        02-syslog-input.conf
   
Code: [Select]
nano /etc/logstash/02-syslog-input.conf

Copy the contents of 02-syslog-input.conf and save

Modify the port if needed
   
   
   20-syslog-filter.conf
   
Code: [Select]
nano /etc/logstash/20-syslog-filter.conf

Copy the contents of20-syslog-filter.conf and save

Modify the section "#change to pfSense ip address" to reflect your pfsense IP address
      
   
   81-pfsense-filter.conf
   
Code: [Select]
nano /etc/logstash/81-pfsense-filter.conf

Copy the contents of81-pfsense-filter.conf  and save
   

       99-elasticsearch-output.conf

Code: [Select]
nano /etc/logstash/99-elasticsearch-output.conf

Copy the contents of99-elasticsearch-output.conf and save
      
         
   10. Download and install Kibana
   
Code: [Select]
wget https://download.elastic.co/kibana/kibana/kibana-4.5.4-linux-x64.tar.gz

untar -xzvf  kibana-4.4.2-linux-x64.tar.gz

mv kibana-4.4.2-linux-x64 /opt/kibana4/

sed -i 's/#pid_file/pid_file/g' /opt/kibana4/config/kibana.yml
      
   11. Create "kibana4.sh" init script and save in /etc/init.d/
   
Code: [Select]
cd /etc/init.d

nano kibana4.sh

Copy the contents of the kibana script and save
   
   
   12. Ensure services are running. Start if necessary.
   
   Start elasticsearch:
   
Code: [Select]
service elasticsearch start   
   Start logstash:
   
Code: [Select]
service logstash start   
   Start kibana:
   
Code: [Select]
/opt/kibana4/bin/kibana &


   13. Log into your pfsense system and point your logs to the ELK IP address:
   Status --> System Logs
   
   14. Log into http://<IP_ADDRESS>:5601
   15. Click "Create Index"
   
   
      
   16. On the kibana interface, go to Settings --> Objects and click Import. Import each file.
     • Dashboard - http://secretwafflelabs.com/files/Firewall_External_Dash.json
     • Visualizations Export - http://secretwafflelabs.com/files/Firewall_External_Visual.json
     • Saved Searches Export - http://secretwafflelabs.com/files/export.json
   
   17. On the kibana interface, go to Settings --> Objects and click the icon to view the new dashboard.

Troubleshooting

NOTE: For some reason my logstash doesn’t start at boot. I have to look into this, but haven't had time yet so I just start it manually

Here are some good troubleshooting commands:

Ensure logstash and elasticsearch are running and did not error out:
Code: [Select]
/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --debug
View the logstash stdout in realtime to see if you are receiving syslog messages from pfsense:
Code: [Select]
tail -f /var/log/logstash/logstash.stdout
Check the logstash configuration files:
Code: [Select]
/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/
If you do not see "Create Index" in step 12, see if logstash created one
Code: [Select]
curl http://localhost:9200/_cat/indices
« Last Edit: November 17, 2016, 05:05:00 am by ando1 »

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14265
  • Karma: +1329/-191
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #1 on: November 10, 2016, 05:53:44 am »
While this great, sure many people will be happy.  Why are you using old versions of stuff?

The current is 5 is it not?  And why such an old version of java?  I just looked on my ubuntu 14.04 vm and 111 is current

user@uc:~$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
user@uc:~$



- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.1-RELEASE on VM esxi 6.5 (home)

Offline ando1

  • Jr. Member
  • **
  • Posts: 25
  • Karma: +5/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #2 on: November 10, 2016, 06:07:08 am »
While this great, sure many people will be happy.  Why are you using old versions of stuff?

The current is 5 is it not?  And why such an old version of java?  I just looked on my ubuntu 14.04 vm and 111 is current

user@uc:~$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
user@uc:~$

I used these versions because these were the ones that worked for me. I asked several times on this forum and received no help so I decided to share a working config with others here as I have read many posts where people said they had tried and could not get it going. If you got a newer version to work, then that's great. Post the instructions so everyone can also enjoy.
« Last Edit: November 10, 2016, 06:15:51 am by ando1 »

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #3 on: November 10, 2016, 07:55:01 am »
Thanks a bunch for this post ando1. Been looking forward to getting ELK going, will try it out when I get some free time...
_________________________

Release: pfSense 2.3.4

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #4 on: November 13, 2016, 10:45:21 am »
I see no Create Index button. The output from your trouble shooting section is:

Code: [Select]
yellow open .kibana 1 1 1 0 3.1kb 3.1kb

Also, when importing the 3 .json files, the "Firewall External" imports fine but I get this error on the other two:

Code: [Select]
Error: Could not locate that index-pattern (id: logstash-*)
KbnError@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:57463:21
SavedObjectNotFound@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:57592:6
applyESResp@http://0.0.0.0:5601/bundles/kibana.bundle.js?v=10000:79296:37
processQueue@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:42404:29
scheduleProcessQueue/<@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:42420:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:43648:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:43459:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:43756:14
done@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38205:37
completeRequest@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38403:8
requestLoaded@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38344:10


Also, in steps 4 & 10, the file version numbers don't match fyi...
« Last Edit: November 14, 2016, 07:41:59 am by AR15USR »
_________________________

Release: pfSense 2.3.4

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #5 on: November 16, 2016, 07:52:19 am »
ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi...
« Last Edit: November 16, 2016, 07:56:12 am by AR15USR »
_________________________

Release: pfSense 2.3.4

Offline ando1

  • Jr. Member
  • **
  • Posts: 25
  • Karma: +5/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #6 on: November 16, 2016, 12:10:55 pm »
ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi...


Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --debug


Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/




Andy

Offline ando1

  • Jr. Member
  • **
  • Posts: 25
  • Karma: +5/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #7 on: November 17, 2016, 05:07:36 am »
For anyone interested in getting the newest version of ELK (v5) working with pfSense, I was able to get do it using the instructions on this siye: http://pfelk.3ilson.com/

You need at least Ubuntu server vv16.04.01

Offline AR15USR

  • Full Member
  • ***
  • Posts: 266
  • Karma: +10/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #8 on: November 17, 2016, 07:44:47 am »

Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --debug


Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/




Andy


/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --debug

Code: [Select]
Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}


/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Code: [Select]
Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}
_________________________

Release: pfSense 2.3.4

Offline ando1

  • Jr. Member
  • **
  • Posts: 25
  • Karma: +5/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #9 on: November 18, 2016, 02:53:36 pm »



/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ --debug

Code: [Select]
Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}


/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Code: [Select]
Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}
[/quote]

You definitely have a config file issue. Logstash combines all the configuration files into one and then processes them. Since the error is at Line 1 column 1 it sounds like the problem may be in the 02-inputs file. Have a look at all config files and double check they are OK.

Offline hamed_forum

  • Jr. Member
  • **
  • Posts: 37
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #10 on: November 21, 2016, 10:56:17 pm »
tanks
if can creat ova or ovf from vm machine and upload it its very good :)

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #11 on: November 22, 2016, 12:43:54 pm »
http://pfelk.3ilson.com/ basically works, but some pointers:

1/ There's a PPA for MaxMind:

Code: [Select]
sudo add-apt-repository ppa:maxmind/ppa
+ see http://dev.maxmind.com/geoip/geoipupdate/ for /etc/GeoIP.conf and run geoipupdate after that. The DB is located in /usr/share/GeoIP/GeoLite2-City.mmdb

2/ You really should set up some authentication:

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-package-installation
https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html
https://www.elastic.co/guide/en/x-pack/current/logstash.html
Do NOT PM for help!

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14265
  • Karma: +1329/-191
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #12 on: November 22, 2016, 03:18:12 pm »
Yeah I had issues with the date stuff in logstash config as well.. had to remove the +0400 and timezone..

I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.  And have not had any time to do any visualizations - which is what everyone wants ;)
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.1-RELEASE on VM esxi 6.5 (home)

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #13 on: November 22, 2016, 03:42:47 pm »
I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java  >:( ::))
Do NOT PM for help!

Offline hamed_forum

  • Jr. Member
  • **
  • Posts: 37
  • Karma: +0/-0
    • View Profile
Re: ELK + pfSense 2.3 Working
« Reply #14 on: February 03, 2017, 02:59:12 pm »
Elasticsearch after 10 sec  start its stop
« Last Edit: February 03, 2017, 03:04:04 pm by hamed_forum »