The pfSense Store

Author Topic: 502 Bad Gateway (nginx) after Update to 2.3  (Read 21728 times)

0 Members and 2 Guests are viewing this topic.

Offline igpit

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #45 on: April 20, 2017, 01:58:48 am »
It just happened again. "Restart php-fpm" solved it, but there is definitely some bug.

Offline weehooey

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #46 on: May 15, 2017, 08:06:28 am »
- Have same issue on 2.3.4
- Restarted PHP-FPM restored GUI and OpenVPN
- Removed IPsec widget from dashboard, hopefully will help

Offline AlexMex

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #47 on: May 17, 2017, 08:19:18 am »
Hello,

I'm getting the 502 bad gateway too. I have just installed pfsense 2.3.4.
I start getting the issue after setting up four VLANs on my OPT1 interface. Using option 16 Restart PHP-FPM is working sometimes on first shot but more frequently after second or third attempt.
CPU usage was arround 1.5%


Today I install SQUID and activate transparent proxy mode on my LAN and four OPT1 VLANs.
After I login in webconfigurator, I cannot access any page. I immediately fall on the nginx 502 bad gateway error. I have tried to use option 16 as before but it does not work anymore. Only a reboot of pfSense box allow me to login again in the web configurator.

After several reboots I saw that  sometimes the OPT1 appears down and the VLANs are up, , sometimes OPT1 and VLANs are down, sometimes everything is up as expected  :-[
Each time CPU usage increase to 100% and then in a few seconds I got the 502 message.

I have made some tests and when I disconnect cable on OPT1, issue does not happen.
I have checked option "Do not forward traffic to Private Address Space (RFC 1918) destinations" and plug in the cable on opt1. It lloks like I am back to initial situation now.

It fails less often but is still annoying since I have to reboot the box to continue my configuration.

If you have any suggestion with this I will really apreciate. For now, I will stay on pfSense 2.2


Offline costasppc

  • Full Member
  • ***
  • Posts: 233
  • Karma: +2/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #48 on: June 08, 2017, 03:39:23 am »
Ι can confirm the issue with 2.3.4-RELEASE (amd64). When the gui is not accessible, OpenVPN users cannot login. When I used option 16, the users could login again.

I disabled the widgets mentioned before (although I need the OpenVPN widget...) and see what's happen.

Best regards

Kostas

Offline edmund

  • Jr. Member
  • **
  • Posts: 99
  • Karma: +4/-1
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #49 on: July 01, 2017, 05:46:36 pm »
I believe that the root of all my problems has been an auto-negotiate failure on the WAN interface ...

Update:  I'm convinced that my problems all stem from using a cheap Chinese "pfSense" system that I purchased on Amazon - it was about half the price of a comparable Netgate unit but it has continuously generated errors on the WAN interface and has never managed to auto-negotiate the link speed.  Recently I had the cable company at the house after complaining that the cable speed (at 10Mbs) was too low - I'm paying for 150Mbs.  After about an hour of trying everything and failing to fix the problem I put a switch in between the cable modem and the firewall wan port - which boosted the speed to about 55Mbs.

I have replaced the "el cheapo" Chinese box with a Netgate box - auto-negotiated works, the cable modem instantly supports 1Gbs and I'm not getting 160Mbs on the cable connection.  I'm no longer seeing any errors on the WAN interface.

Offline hdejongh

  • Newbie
  • *
  • Posts: 17
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #50 on: October 19, 2017, 10:57:22 am »
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.

Only way to solve it is rebooting the firewall

btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).

My physical pfsense's are not infected...

Offline jahonix

  • Hero Member
  • *****
  • Posts: 2433
  • Karma: +145/-14
  • volunteer since 2006
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #51 on: October 19, 2017, 11:00:53 am »
Search in this forum or on redmine.pfsense.org - this was seen before.
IIRC it may come from an associated CD/DVD drive to your pfSense VM. Get rid of that and it might work. Other problem I don't remember off of my head is discussed on redmine.
Chris

The issue with IPv6 jokes is that almost no one understands them and no one is using them yet.

Offline rightnow

  • Newbie
  • *
  • Posts: 16
  • Karma: +1/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #52 on: October 20, 2017, 01:39:47 pm »
sunday i updated 6 firewalls from 2.3 to 2.4.
All suddenly are showing 502 bad gateway problems.
Besides that 2 of them become unresponsive after around 20 hours.
Its hard to access the firewall then but the one time i got lucky i can see that the memory is completely full.
So i doubled the memory and still same problem occur.

Only way to solve it is rebooting the firewall

btw, all are vm's
2 of them were a complete new install with a config restore (i had to go from 32bits to 64 bits).

My physical pfsense's are not infected...

Exactly same problem here! Firewall dies, and 502 bad gateway.

Offline mikael.andre

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #53 on: October 21, 2017, 03:12:51 am »
Hello,

I experienced the same problem on 2.4.0 release.
I have no access to my firewall by :
- SSH
- WebGUI
- Console

BUT, I still continue to surf on the Internet... Very strange...
This issue occurs at the end of 20 hours uptime.
It's hardening to identify the root cause.
The only way to resolve this problem is rebooting my hardware appliance.
Once the reboot process is done, there is no event logs...

I have the following widget on my dashboard :
- System Informations
- NTP Status
- SMART Status
- pfBlockerNG
- OpenVPN
- Gateways
- Interfaces
- Interfaces Statistics
- Traffic Graphs
- Services status
- Firewall logs
- Thermal sensors

Heres my hardware configuration :
- MotherBoard : APU2C4
- CPU : AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support
- RAM : 4 GByte DDR3-1333 DRAM
- Storage Type : mSATA SSD
- Storage Size : 120GB
- Ethernet ports : 3 x 1Gbit/s
- Wireless card : WLE200NX with two antenna
Best regards,

MikaŽl ANDRE

Offline BBcan177

  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #54 on: October 21, 2017, 07:13:26 am »
Update pfBlockerNG to v2.1.2 and reboot.

https://forum.pfsense.org/index.php?topic=137103.0
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline mikael.andre

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #55 on: October 22, 2017, 02:36:47 am »
Hi BBcan177,

Many thanks.
I was already in this version. But, I deleted some widget and increased the number of webconfigurator process up to 4.
I also disabled autocheck updates in Dashboard and deinstalled NTOPNG due to some unstabilities.
I keep an eye on it.
« Last Edit: October 22, 2017, 04:59:41 am by mikael.andre »
Best regards,

MikaŽl ANDRE

Offline pdrass

  • Jr. Member
  • **
  • Posts: 65
  • Karma: +1/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #56 on: November 21, 2017, 08:48:15 pm »
This has reared its ugly head again on the latest 2.4.1 release.  I have about 10% of my FW's doing this.

I tried to update the packages:
Code: [Select]
pkg update
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
On the webUI it sits and spins.

So...apparently the repo's are really down and you can't get to them?

https://forum.pfsense.org/index.php?topic=139276.0

I updated two of the bad apples today in hopes they'd be fixed form the nginx problem.

I can't ssh to it, when I login it just says "pfsense -" <-- I can't do the /etc/init....whatever to start the console number list, I can't shutdown -r now, nothing...it just sites there.  I also can't OpenVPN into the box which is when I start getting the phone calls then someone has to go in and reboot the firewall.

Is this PFBng doing this?  Perish the thought!  :0)

1 - When will the repos be back up?
2 - I swear I remembered to update the packages on each box after updating but maybe I missed these two

Once again...thanks PFSense for all you do, the ONLY firewall I use and deploy in production!

Offline BBcan177

  • Hero Member
  • *****
  • Posts: 2553
  • Karma: +796/-5
    • View Profile
    • Click for Support
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #57 on: November 21, 2017, 09:07:15 pm »
Try to hit: CTRL - z and then enter "/bin/tcsh"
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline brainyron

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #58 on: November 22, 2017, 12:49:43 pm »
I"ve been seeing this off and on with 2.4.  It seemed to be related to a pfBlockerNG bug that was patched -- but with 2.4.2 it's happening again.  I tried the Control-Z trick as suggested and get no response

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 387
  • Karma: +20/-50
    • View Profile
Re: 502 Bad Gateway (nginx) after Update to 2.3
« Reply #59 on: November 22, 2017, 02:40:23 pm »
I did briefly look into this and have a theory of whats going on, but its only a theory.

pfSense backend scripts, a lot of them run on PHP.

Instead of using PHP cli, they seem to utilise PHP FPM instead of PHP CLI for the cli scripts, which in turn will utilise the FPM server processes, if you have some of these background scripts been processed "whilst" trying to access the GUI, then none may be available for nginx to use and you will get the gateway error.

If I am correct there is a number of possible resolutions, each with their own downsides.

1 - Stop using FPM to run background PHP stuff.
2 - Increase FPM processes.
3 - Retune FPM so its more scalable for higher load and avoid gateway errors.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK