The pfSense Store

Author Topic: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1  (Read 13817 times)

0 Members and 1 Guest are viewing this topic.

Offline jahonix

  • Hero Member
  • *****
  • Posts: 2495
  • Karma: +150/-16
  • volunteer since 2006
    • View Profile
504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« on: April 15, 2016, 11:37:03 pm »
I have quite strange a phenomena and don't know exactly where to start troubleshooting.

Updated an APU1 yesterday afternoon from 2.2.6 to 2.3 with Nano install on an SD card.
Today the unit wasn't accessible via HTTPS (504 Gateway Time-out). Doesn't route also but still hands out DHCP leases to clients.

To get to production again I installed my config.xml to a different APU1 which I just updated to 2.3 from 2.3RC. This device has an mSATA disk.
Well, now, nearly 10h after working flawlessly the second device goes down as well. Cannot login via HTTPS or ssh from LAN. I can, however, ssh via an OPT interface - but HTTPS doesn't work there as well.

When I had console access via serial I tried to restart the webConfigurator and rebooted the device. No change.
Reverting back to HTTP didn't help either, I still get redirected to HTTPS, even after a reboot.

WAN is PPPoE on re0_vlan7, LAN and other OPT interfaces are re1_vlan10 / 20 / 30 / 40 / 50 / 60.
Installed package was Backup/Restore only.

Anyone got an idea? I'm a bit clueless.
Chris

The issue with IPv6 jokes is that almost no one understands them and no one is using them yet.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +894/-7
    • View Profile
    • Chris Buechler
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #1 on: April 16, 2016, 01:12:14 am »
The remaining HTTPS redirect is most likely cached by your browser, they're pesky about that.

Only thing I'm aware of there is if the system doesn't have Internet access, update checks can pile up and hang the GUI, resulting in a 504.
https://redmine.pfsense.org/issues/6177

Any indication it's having trouble checking for updates?

Offline jahonix

  • Hero Member
  • *****
  • Posts: 2495
  • Karma: +150/-16
  • volunteer since 2006
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #2 on: April 16, 2016, 06:39:38 am »
HTTPS redirect: maybe, i'm unsure if I checked this with a different browser as well.

System has Internet access most all the time. I always listen to streaming radio and it just went on.

The first APU going down did not have troubles checking for updates (I remember having looked at it). I don't know about the alternative unit but since it received the same config and was a drop-in replacement I doubt it being different.

Any idea why I can ssh from OPT1 and not from LAN, both being VLANs on the same trunk? Rules surely don't permit it.
Chris

The issue with IPv6 jokes is that almost no one understands them and no one is using them yet.

Offline jahonix

  • Hero Member
  • *****
  • Posts: 2495
  • Karma: +150/-16
  • volunteer since 2006
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #3 on: April 18, 2016, 06:57:01 am »
Sorry, Chris, from the console I could see that pkg update indeed doesn't work.
ssh from Lan is affected as well, vie OPT1 it's ok. Strange...
Chris

The issue with IPv6 jokes is that almost no one understands them and no one is using them yet.

Offline dusan

  • Full Member
  • ***
  • Posts: 216
  • Karma: +10/-0
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #4 on: April 18, 2016, 12:18:10 pm »
I think I have the same situation here.

After a few days pfSense has basically stopped responding to clients (from any interface). I can't access it via HTTPS (nginx error 504). It doesn't respond to OpenVPN clients. It doesn't reply on ping.

But it's still passing traffic _through it_. Clients in the LAN can still access Internet. Clients from the Internet can still access every e-mail server and website located in the LAN.

It is a single-LAN multi-WAN system with PPPoE on all WAN OPT1 OPT2 etc. pfSense v.2.3-RELEASE (i386), upgraded from 2.2.6. The system is a virtual machine on IBM x3650 host server running VMware ESXi v. 4.1.

EDIT: PPPoE on OPT1 OPT2 etc, but DHCP on WAN.

EDIT 2: it still replies to ping and responds to IPsec remote peers (which are also pfSense v.2.3).

EDIT 3: sometimes nginx reports error 502 (Bad Gateway).

EDIT 4: it still responds to SSH connection. Tested on remote pfSense 2.3 devices that stopped responding.
« Last Edit: April 19, 2016, 03:15:29 am by dusan »

Offline dusan

  • Full Member
  • ***
  • Posts: 216
  • Karma: +10/-0
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #5 on: April 19, 2016, 03:19:04 am »
It can be fixed by restarting PHP-FPM (console command #16), but not permanently. After a few hours it stops responding to HTTPS (nginx error 504 or 502) again.

EDIT: I think I've found the root cause here: https://forum.pfsense.org/index.php?topic=110070.0. It's the IPsec widget.
« Last Edit: April 19, 2016, 03:30:47 am by dusan »

Offline jahonix

  • Hero Member
  • *****
  • Posts: 2495
  • Karma: +150/-16
  • volunteer since 2006
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #6 on: April 19, 2016, 05:30:57 am »
Could be.
In vanilla mode it runs flawlessly, my restored config has the IPsec widget in dashboard. I'll try to remove that and test again (when I have the time to).
Chris

The issue with IPv6 jokes is that almost no one understands them and no one is using them yet.

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #7 on: April 25, 2016, 08:23:50 am »
I had this problem (not exactly but same effect). I tried updating to the 2.3.1 snapshot and problem persisted. I removed every widget (as I saw there were multiple widgets along with IPSec that was being logged), rebooted, waited it out for a couple minutes, and slowly re added widgets. IPSec did pop back up with an nginx error but removed it again and just waited a couple hours longer. As far as I can tell my system running fine for the past  +12 hours (and it usually breaks even during the day despite it waiting till midnight to break.

My post is here if you want to see what me and others were getting on this issue.

https://forum.pfsense.org/index.php?topic=110121.0

Edit: Seems it came back to haunt me once again, if you haven't done the above yet try to also clear your browser cache. It would seem the IPSec widget isn't playing well but I have reduced the amount of times this problem occurs (as far as Im experiencing so far). But it seems like its varying in occurrences, widget, and when it brings the webGUI down.
« Last Edit: April 25, 2016, 11:12:47 pm by crisdavid »
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline h0tf1r3

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #8 on: May 17, 2016, 08:16:47 pm »
Look also here discussing the same problem! https://forum.pfsense.org/index.php?topic=110121.0

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +894/-7
    • View Profile
    • Chris Buechler
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #9 on: May 25, 2016, 09:37:45 pm »
This was likely fixed in either 2.3.1 or 2.3.1_1 depending on which instance of the issue is responsible.

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #10 on: October 13, 2016, 11:05:31 am »
This was likely fixed in either 2.3.1 or 2.3.1_1 depending on which instance of the issue is responsible.

Yes, I can confirm that this issue is resolved (at least I haven't noticed the problem arising lately). Latest updates have been doing wonders  :)
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline toby-rdc

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #11 on: November 27, 2016, 06:01:55 am »
Hello

This issue is NOT resolved in version 2.3.1_1 . I contantly gett 502 BAD gateway after some time of usage.
I have removed IPSEC gadgets etc and the problem still persists.
I have to reboot the pfsense everyday because of this.

Best regards
Toby

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #12 on: November 27, 2016, 06:25:19 am »
There's no need to reboot, simply restart PHP-FPM and the webconfigurator from the shell menu. Plus you are two releases behind.
Do NOT PM for help!

Offline toby-rdc

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #13 on: November 27, 2016, 07:09:09 am »
Hello

Running 2.3.2 P1 , sorry i did not check. But anyway the error is still there. I have it on several units

/Toby

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: 504 Gateway Time-out (nginx) - 2.3 upgrade on 2x APU1
« Reply #14 on: November 27, 2016, 08:01:33 am »
Hello

Running 2.3.2 P1 , sorry i did not check. But anyway the error is still there. I have it on several units

/Toby

If you can please run this command, and try to capture the output; when the issue happens ps uxawww either by ssh, or the local terminal. Install pstree for an even better way to find the issue. One of the devs instructed me to do this in order to see what was the cause.

pkg install pstree
rehash
pstree
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB