Netgate SG-1000 microFirewall

Author Topic: Nginx with IPsec widget causes 504?  (Read 2372 times)

0 Members and 1 Guest are viewing this topic.

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Nginx with IPsec widget causes 504?
« on: May 07, 2016, 02:36:05 am »
When having IPSec widget up on the dashboard I get a 504 Gateway Timeout with Nginx and been having this since 2.3 release. Seems if I remove the widget everything is fine.

Although regardless I see this being output when entering the command "ps auwwx"
Code: [Select]
root    35838   0.3  1.0 268244 38740  -  S     3:26AM   0:00.31 php-fpm: pool nginx (php-fpm)
root    41479   0.0  0.2  38844  6324  -  Is   10:43PM   0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root    41650   0.0  0.2  38844  7380  -  S    10:43PM   0:04.95 nginx: worker process (nginx)
root    42060   0.0  0.2  38844  7380  -  S    10:43PM   0:05.05 nginx: worker process (nginx)
root    42294   0.0  0.2  38844  7364  -  S    10:43PM   0:06.86 nginx: worker process (nginx)

Not sure if theres supposed to be three worker process or if the problem lies elsewhere this is what i've have found while on the 2.3.1 snapshot. Does anyone have a similar result?
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Nginx with IPsec widget causes 504?
« Reply #1 on: May 09, 2016, 01:23:49 pm »
What do you see for php-fpm when that happens? The timeout you see is not nginx timing out, but nginx saying that what it was trying to load timed out. Usually there is something stuck in PHP when that happens. If you can show the full "ps uxawww" output it may be possible to spot.

Or, even better:
Code: [Select]
pkg install pstree
rehash
pstree

Similar output but it makes it easier to spot which processes are children of others.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: Nginx with IPsec widget causes 504?
« Reply #2 on: May 09, 2016, 03:37:27 pm »
Thank you for the reply
Give me some time to get this to reproduce and i'll provide the necessary information.

Edit:
pstree output displays the following

Code: [Select]
-+= 00001 root /sbin/init --
 |-+= 00318 root php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fp
 | |--- 37870 root php-fpm: pool nginx (php-fpm)
 | |--- 39664 root php-fpm: pool nginx (php-fpm)
 | |--- 43229 root php-fpm: pool nginx (php-fpm)
 | \--- 83249 root php-fpm: pool nginx (php-fpm)
 |-+= 00356 root /usr/local/sbin/check_reload_status
 | \--- 00358 root check_reload_status: Monitoring daemon of check_reload_stat
 |--= 00371 root /sbin/devd -q
 |-+= 12981 root /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/
 | \--= 40178 root /usr/local/sbin/sshlockout_pf 15
 |-+= 14052 root /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/loca
 | \--- 14294 root minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
 |-+= 14504 root /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /us
 | \-+- 14756 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expirea
 |   \--- 76576 root /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
 |-+= 14678 root /usr/sbin/sshd
 | |-+= 12555 root sshd: admin@notty (sshd)
 | | \-+= 12838 root /bin/sh /etc/rc.initial -c /usr/libexec/sftp-server
 | |   \--- 13111 root /usr/libexec/sftp-server
 | |-+= 57044 root sshd: root@pts/0 (sshd)
 | | \-+= 57647 root -sh (sh)
 | |   \-+= 57715 root /bin/sh /etc/rc.initial
 | |     \--= 59472 root /bin/tcsh
 | |-+= 57334 root sshd: root@notty (sshd)
 | | \--= 57548 root /usr/libexec/sftp-server
 | \-+= 84179 root sshd: admin@pts/1 (sshd)
 |   \-+= 13338 root /bin/sh /etc/rc.initial
 |     \-+= 14582 root /bin/tcsh
 |       \-+= 15630 root pstree
 |         \--- 15653 root ps -axwwo user,pid,ppid,pgid,command
 |--= 14939 root /usr/local/sbin/sshlockout_pf 15
 |-+= 14991 root /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.
 | \--- 14999 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_
 |--= 15732 root dhclient: em0 [priv] (dhclient)
 |--= 21103 _dhcp dhclient: em0 (dhclient)
 |--= 24559 root /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
 |--= 25401 root /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.con
 |--= 27077 root /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf
 |--= 40849 root /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B xxx.xxx.xxx.xxx -
 |-+= 41267 root nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx
 | |--- 41426 root nginx: worker process (nginx)
 | |--- 41427 root nginx: worker process (nginx)
 | \--- 41717 root nginx: worker process (nginx)
 |--= 42091 root /usr/sbin/cron -s
 |--= 43657 unbound /usr/local/sbin/unbound -c /var/unbound/unbound.conf
 |--= 44423 root /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases
 |--= 44782 root /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
 |--= 58083 dhcpd /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/
 |-+= 62095 root /usr/local/libexec/ipsec/starter --daemon charon
 | \--= 62399 root /usr/local/libexec/ipsec/charon --use-syslog
 |--= 70428 root /usr/sbin/powerd -b min -a adp -n adp
 |--= 75906 root /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/
 |--= 39550 root /usr/libexec/getty Pc ttyv0
 \-+- 66263 root /bin/sh /var/db/rrd/updaterrd.sh
   \--- 12539 root sleep 60

if it helps here is the ps auwwx output

Code: [Select]
USER      PID  %CPU %MEM    VSZ   RSS TT  STAT STARTED      TIME COMMAND
root       11 200.0  0.0      0    32  -  RL    4:59PM 573:26.08 [idle]
root        0   0.0  0.0      0   240  -  DLs   4:59PM   0:53.63 [kernel]
root        1   0.0  0.0   9136   812  -  ILs   4:59PM   0:00.00 /sbin/init --
root        2   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [crypto]
root        3   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [crypto returns]
root        4   0.0  0.0      0    32  -  DL    4:59PM   0:00.00 [cam]
root        5   0.0  0.0      0    16  -  DL    4:59PM   0:04.43 [pf purge]
root        6   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [sctp_iterator]
root        7   0.0  0.0      0    16  -  DL    4:59PM   0:00.01 [enc_daemon0]
root        8   0.0  0.0      0    32  -  DL    4:59PM   0:00.23 [pagedaemon]
root        9   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [vmdaemon]
root       10   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [audit]
root       12   0.0  0.0      0   528  -  WL    4:59PM   0:36.50 [intr]
root       13   0.0  0.0      0    32  -  DL    4:59PM   0:00.00 [ng_queue]
root       14   0.0  0.0      0    48  -  DL    4:59PM   0:00.64 [geom]
root       15   0.0  0.0      0    16  -  DL    4:59PM   0:14.76 [rand_harvestq]
root       16   0.0  0.0      0   560  -  DL    4:59PM   0:00.37 [usb]
root       17   0.0  0.0      0    16  -  DL    4:59PM   0:00.00 [pagezero]
root       18   0.0  0.0      0    16  -  DL    4:59PM   0:00.01 [idlepoll]
root       19   0.0  0.0      0    32  -  DL    4:59PM   0:00.24 [bufdaemon]
root       20   0.0  0.0      0    16  -  DL    4:59PM   0:02.70 [syncer]
root       21   0.0  0.0      0    16  -  DL    4:59PM   0:00.04 [vnlru]
root       56   0.0  0.0      0    16  -  DL    4:59PM   0:00.02 [md0]
root      318   0.0  0.7 268244 26916  -  Ss    4:59PM   0:00.75 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root      356   0.0  0.1  18888  2504  -  INs   4:59PM   0:00.00 /usr/local/sbin/check_reload_status
root      358   0.0  0.1  18888  2392  -  IN    4:59PM   0:00.00 check_reload_status: Monitoring daemon of check_reload_status
root      371   0.0  0.1  13624  5200  -  Is    4:59PM   0:00.01 /sbin/devd -q
root    12555   0.0  0.2  82264  8412  -  Ss   10:07PM   0:00.14 sshd: admin@notty (sshd)
root    12838   0.0  0.1  17000  2512  -  Is   10:07PM   0:00.00 /bin/sh /etc/rc.initial -c /usr/libexec/sftp-server
root    12981   0.0  0.1  14516  2316  -  Ss    5:00PM   0:01.73 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /var/etc/syslog.conf
root    13111   0.0  0.2  50292  7412  -  I    10:07PM   0:00.01 /usr/libexec/sftp-server
root    14052   0.0  0.0  12268  1872  -  Is    5:00PM   0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root    14294   0.0  0.0  12268  1884  -  I     5:00PM   0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root    14504   0.0  0.0  12268  1872  -  Is    5:00PM   0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root    14678   0.0  0.2  59064  8072  -  Is    4:59PM   0:00.00 /usr/sbin/sshd
root    14756   0.0  0.0  12268  1884  -  I     5:00PM   0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts  (minicron)
root    14939   0.0  0.1  14612  2180  -  Is    4:59PM   0:00.00 /usr/local/sbin/sshlockout_pf 15
root    14991   0.0  0.0  12268  1872  -  Is    5:00PM   0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
root    14999   0.0  0.0  12268  1884  -  I     5:00PM   0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data  (minicron)
root    15732   0.0  0.1  14564  2264  -  Is    4:59PM   0:00.00 dhclient: em0 [priv] (dhclient)
_dhcp   21103   0.0  0.1  14564  2344  -  Is    4:59PM   0:00.02 dhclient: em0 (dhclient)
root    24559   0.0  0.1  16676  2428  -  Ss    4:59PM   0:00.48 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root    25401   0.0  0.1  21624  5548  -  Ss    4:59PM   0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
root    27077   0.0  0.1  18896  2488  -  Is    4:59PM   0:00.00 /usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid
root    37870   0.0  0.9 281056 36780  -  I     5:20PM   0:00.22 php-fpm: pool nginx (php-fpm)
root    39664   0.0  0.9 281056 36932  -  I     7:49PM   0:00.16 php-fpm: pool nginx (php-fpm)
root    40178   0.0  0.1  14612  2180  -  Is    5:00PM   0:00.00 /usr/local/sbin/sshlockout_pf 15
root    40849   0.0  0.1  15012  2292  -  Is    4:59PM   0:01.34 /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B xxx.xxx.xxx.xxx -p /var/run/dpinger_WAN_DHCP_xxx.xxx.xxx.xxx_xxx.xxx.xxx.xxx.pid -u /var/run/dpinger_WAN_DHCP_xxx.xxx.xxx.xxx_xxx.xxx.xxx.xxx.sock -C /etc/rc.gateway_alarm -d 0 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 xxx.xxx.xxx.xxx
root    41267   0.0  0.2  38844  6324  -  Is    5:00PM   0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root    41426   0.0  0.2  38844  7312  -  S     5:00PM   0:03.70 nginx: worker process (nginx)
root    41427   0.0  0.2  38844  7336  -  S     5:00PM   0:04.04 nginx: worker process (nginx)
root    41717   0.0  0.2  38844  7308  -  I     5:00PM   0:04.51 nginx: worker process (nginx)
root    42091   0.0  0.1  16532  2288  -  Ss    5:00PM   0:00.07 /usr/sbin/cron -s
root    43229   0.0  0.9 281056 37000  -  I     5:35PM   0:00.25 php-fpm: pool nginx (php-fpm)
unbound 43657   0.0  0.5  43220 21020  -  Ss    5:00PM   0:01.97 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root    44423   0.0  0.1  12272  2032  -  Is    5:00PM   0:00.00 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d mauro.manor -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /var/etc/hosts
root    44782   0.0  0.4  30136 17964  -  Ss    5:00PM   0:00.72 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
root    57044   0.0  0.2  82264  8668  -  Ss   10:02PM   0:00.03 sshd: root@pts/0 (sshd)
root    57334   0.0  0.2  82264  8412  -  Ss   10:02PM   0:00.14 sshd: root@notty (sshd)
root    57548   0.0  0.2  50292  7412  -  Is   10:02PM   0:00.01 /usr/libexec/sftp-server
dhcpd   58083   0.0  0.3  24804 13564  -  Ss    5:00PM   0:00.80 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
root    62095   0.0  0.1  30380  3460  -  Is    5:00PM   0:00.00 /usr/local/libexec/ipsec/starter --daemon charon
root    62399   0.0  0.4 222312 14172  -  Is    5:00PM   0:00.54 /usr/local/libexec/ipsec/charon --use-syslog
root    65977   0.0  0.0   8168  1824  -  IN   10:12PM   0:00.00 sleep 60
root    70428   0.0  0.0  14408  1956  -  Ss    5:00PM   0:01.03 /usr/sbin/powerd -b min -a adp -n adp
root    75906   0.0  0.1  21032  4904  -  Is    5:00PM   0:00.32 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
root    76576   0.0  0.0  14328  1964  -  I     8:00PM   0:00.00 /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root    83249   0.0  0.9 281052 38048  -  I     5:14PM   0:00.24 php-fpm: pool nginx (php-fpm)
root    84179   0.0  0.2  82264  8660  -  Ss   10:07PM   0:00.03 sshd: admin@pts/1 (sshd)
root    39550   0.0  0.0  14428  1988 v0  Is+   5:00PM   0:00.00 /usr/libexec/getty Pc ttyv0
root    66263   0.0  0.1  17000  2580 v0- IN    5:00PM   0:02.44 /bin/sh /var/db/rrd/updaterrd.sh
root    57647   0.0  0.1  17000  2632  0  Is   10:02PM   0:00.00 -sh (sh)
root    57715   0.0  0.1  17000  2540  0  I    10:02PM   0:00.00 /bin/sh /etc/rc.initial
root    59472   0.0  0.1  17340  3660  0  I+   10:02PM   0:00.01 /bin/tcsh
root    13338   0.0  0.1  17000  2536  1  Is   10:07PM   0:00.00 /bin/sh /etc/rc.initial
root    14582   0.0  0.1  17340  3572  1  S    10:07PM   0:00.01 /bin/tcsh
root    67673   0.0  0.1  18676  2264  1  R+   10:13PM   0:00.00 ps auwwx

I've replaced my public ip address with xxx.xxx.xxx.xxx
« Last Edit: May 09, 2016, 09:18:23 pm by crisdavid »
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Nginx with IPsec widget causes 504?
« Reply #3 on: May 10, 2016, 02:53:05 pm »
For future reference, when you edit a post, it doesn't notify that you've edited, so I didn't see that until I randomly stumbled back on this thread -- make a new reply rather than editing and it will show up easier for those who have already replied and visited the thread.

The only thing that stands out is this:
Code: [Select]
|-+= 14504 root /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /us
 | \-+- 14756 root minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expirea
 |   \--- 76576 root /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts

Code: [Select]
root    76576   0.0  0.0  14328  1964  -  I     8:00PM   0:00.00 /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
It appears to have gotten stuck in that script expiring user accounts.

Do you have any other accounts defined besides "admin"? Any groups? Can you share the account usernames and group names?

If that happens again and you see that specific process in the ps output, try to kill it and see if the GUI works again. The fcgicli process itself, that is, not the minicron or helper thread.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: Nginx with IPsec widget causes 504?
« Reply #4 on: May 10, 2016, 04:46:44 pm »
My apologies about that. I created an admin group for myself and another member so I could keep the actual admin account better secured; other than that I created three other users in another group with only access to the WOL page and the dashboard. I remember accidentally disabling or deleting one of the users in my admin group that I use to login most of the time but was able to restore it later. Would the reason be that or does that process refer to something else? Oddly once I remove the IPSec widget everything returns to normal but once I add it to the dashboard it'll return with the 504 later on until I restart php.
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21388
  • Karma: +1432/-26
    • View Profile
Re: Nginx with IPsec widget causes 504?
« Reply #5 on: May 11, 2016, 02:04:48 pm »
Extra users and groups are fine, I was mostly curious since before 2.3 released we had observed an issue with spaces in group names causing a problem with accounts being synchronized, I thought it might have been related.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!