pfSense Gold Subscription

Author Topic: New 502 Bad Gateway  (Read 11542 times)

0 Members and 2 Guests are viewing this topic.

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #30 on: October 10, 2017, 10:35:05 am »
Its something with Pfblocker and the new 2.4.0 & 2.4.1.  2.3 works fine.  I  can uninstall pfblockerng and my systems stay up for days, weeks, even longer.. if I put pfblockerng in.. it locks in less than 12 hours.. or less.  The 502 is just a symptom of the issue, but its the symptom you notice first.  Nginix is not directly related to the issue, its the system locking up internally, and not executing commands.  Causes everything to go nuts.

Offline seanr22a

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #31 on: October 10, 2017, 11:40:45 am »
Strange thing is that I run PFblocker on the SG-2440 boxes without issues. They have been up since I installed the 1006.2203 release .... 4 days now. Same install on the SG-8660 locks up in a couple of hours.  ???

Offline MaxPF

  • Full Member
  • ***
  • Posts: 259
  • Karma: +1/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #32 on: October 10, 2017, 05:10:35 pm »
Just happened again. Here are my symptoms:

- Web GUI inaccessible
- ssh and local console do no show the menu and do not accept any input
- external openvpn clients cannot connect (I have an inboud rule that uses an country alias created in pfblocker)

Everything else keeps working, internet access, outbout vpn tunnels, port forward from outside to internal servers, DNS resolver, FW rules, vlan routing, etc.

I just updated to the latest build and I hope this gets fixed in the final release.

Offline AhnHEL

  • Hero Member
  • *****
  • Posts: 636
  • Karma: +18/-0
  • It is what it is.
    • View Profile
Re: New 502 Bad Gateway
« Reply #33 on: October 10, 2017, 05:17:02 pm »
I was seeing the same thing, I upgraded to the October 9th snapshot, and did a "pkg upgrade -f" at the shell prompt and all is well again.

Statement retracted.

Took a day but yes, eventually got the bad gateway again.
« Last Edit: October 11, 2017, 03:44:29 pm by AhnHEL »
AhnHEL (Angel)
NYC

2 pfSense sites: 2.4.2 (amd64)
Dell 755 SFF E6550 @ 2.3Ghz, 4GB RAM, 100/30 Mbps, Intel X3959
Dell 7010 SFF i5-3570 @ 3.4Ghz, 8GB RAM, 940/880 Mbps, Intel X3959
OpenVPN (Road Warrior), pfBlockerNG, Gaming


Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #34 on: October 10, 2017, 05:33:56 pm »
Strange thing is that I run PFblocker on the SG-2440 boxes without issues. They have been up since I installed the 1006.2203 release .... 4 days now. Same install on the SG-8660 locks up in a couple of hours.  ???

On the SG-2440 boxes did you do a fresh install or update, and vise versa for the SG-6880?

I was seeing the same thing, I upgraded to the October 9th snapshot, and did a "pkg upgrade -f" at the shell prompt and all is well again.

I am on the 20171009.1758 and still having issues. Depending on your load will depend on how fast it takes, I have one that can run for 4 days before it locks, but I have another than locks in less than 12 hours.  I have also done the "pkg upgrade -f" and "pkg update -f" on all systems, to rule out maybe a bad file.  Still persists.

I hate to say it, but I am glad others are having issues.  I knew I wasn't crazy like what was suggested.  ::)

Offline PiBa

  • Hero Member
  • *****
  • Posts: 731
  • Karma: +123/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: New 502 Bad Gateway
« Reply #35 on: October 10, 2017, 08:02:52 pm »
Hi besides using pfBlocker, do (some of?) you guys also use xmlrpc feature?
Can you check that there are 4 active php-fpm processes 'running' ?

I found that if i sync pfSense settings several times in a period of a few seconds then sometimes the sync process bails out as soon as it needs to 'print' some output, adding ignore_user_abort and setting no time limit should allow it to always finish and remove the placed lock file..

Maybe some of you can try if this patch can solve/reduce some of these issues?: https://github.com/pfsense/pfsense/pull/3848
To apply it, either add the 2 lines manually to the changed file /usr/local/www/xmlrpc.php or apply it through patches package..

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #36 on: October 10, 2017, 10:39:59 pm »
Hi besides using pfBlocker, do (some of?) you guys also use xmlrpc feature?
Can you check that there are 4 active php-fpm processes 'running' ?

I found that if i sync pfSense settings several times in a period of a few seconds then sometimes the sync process bails out as soon as it needs to 'print' some output, adding ignore_user_abort and setting no time limit should allow it to always finish and remove the placed lock file..

Maybe some of you can try if this patch can solve/reduce some of these issues?: https://github.com/pfsense/pfsense/pull/3848
To apply it, either add the 2 lines manually to the changed file /usr/local/www/xmlrpc.php or apply it through patches package..

I dont use the xmlrpc feature on any of my boxes..  I don't know if the other guys do, they may..

Offline crisdavid

  • Jr. Member
  • **
  • Posts: 84
  • Karma: +12/-1
  • pfSense is the future of Networking.
    • View Profile
Re: New 502 Bad Gateway
« Reply #37 on: October 11, 2017, 04:52:44 am »
Having the same issue here. I've even tried using "pkg upgrade -f" in the shell but nothing. I can sign in as a user an execute commands at user level no problem but as admin I'll get stuck with the command screen unable to load as everyone else has a problem with. Shutdown won't even commence and I have to do a forced shutdown in order to restart. I'm on the latest build on my home box but I have another box with this issue running on the weekend release. Removed squid/squidguard, I'm also not using xmlrpc feature but am using pfBlockerNG.
Both of My pfSense boxes:
Dell OptiPlex 7010 SFF
OS: pfSense 2.4
CPU: I5-3570 3.4 GHz
RAM: 4GB
NIC: Intel EXPI9402PT Pro
Hard drive: 500GB

Offline seanr22a

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #38 on: October 11, 2017, 06:21:40 am »

On the SG-2440 boxes did you do a fresh install or update, and vise versa for the SG-6880?


All three came from 2.3.4-RELEASE-p1. I de-installed all packages on all three, upgraded to 2.4 and installed the packages again. I only have 502 problems with the SG-8860. Now when the 8660 is downgraded to 2.3.4-RELEASE-p1 I have problems with the XMLRPC Sync because of the mixed 2.4 and 2.3.4 setup.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +957/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: New 502 Bad Gateway
« Reply #39 on: October 11, 2017, 06:22:10 am »
Do NOT PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: New 502 Bad Gateway
« Reply #41 on: October 11, 2017, 11:00:42 am »
The bug report and the thread here are bit unclear and contradictory now.

Is this happening on current 2.4.0 snapshots or only 2.4.1?

What error messages are in the system log, if any, when it happens?

Does it only happen with pfBlocker installed, or is there some other way to trigger it? Captive Portal, perhaps? Does it require using a specific pfBlocker feature like DNSBL?

If it's the max connections error, increasing kern.ipc.soacceptqueue should fix it (somaxconn no longer exists, it has a convenience alias but using it is quirky)
« Last Edit: October 11, 2017, 11:06:07 am by jimp »
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #42 on: October 11, 2017, 11:35:59 am »
The bug report and the thread here are bit unclear and contradictory now.

Is this happening on current 2.4.0 snapshots or only 2.4.1?

What error messages are in the system log, if any, when it happens?

Does it only happen with pfBlocker installed, or is there some other way to trigger it? Captive Portal, perhaps? Does it require using a specific pfBlocker feature like DNSBL?

If it's the max connections error, increasing kern.ipc.soacceptqueue should fix it (somaxconn no longer exists, it has a convenience alias but using it is quirky)

Its happening on 2.4.0 after the Oct 3 update or around that. it happened on 2.4.1 before that.  I was on 2.4.1 and it started happening, so I reloaded to 2.4.0 and issue was resolved till after the Oct 3 update then it showed its head there as well.  Others on here are still on the 2.4.1 and it is still present as well.  It effects both 2.4.0 and 2.4.1. From what I can tell it only happens if pfBlockerNG is installed,  The 502 Bad gateway is a result of the processes not finishing.  It happens on both my systems with captive portal and without captive portal.  So CP has nothing to do with it.  As for features of pfBlocker, I have tested a bit with that, but nothing conclusive. 

I have kern.ipc.soacceptqueue set at 4096 and it still happens, Also it happens even on systems that have low connections.  I have one system I use just for home, with a super low connection rate, and it happens as well, takes longer but still will eventually get the error.

I have searched the system logs, and it shows nothing about whats causing the error itself.  It just starts filling up with the Connections errors.  But when you SSH in, it will not load the gui, you have to CTRL-Z to get a prompt.  Once you have a prompt, you can do Nano, or VI but that is about the extent of it. It will not show the directory you are in, and if you try to restart PHP it will lock up on restart and you have to CTRL-Z again.   I have found that you can surf the net just fine when its happening, however SOME pages will fail, but not all.  In general its still good to go, but it really affects the web Gui, and any CP due to it locks up the internal system.  But Lan, Wan still seem to operate just fine for the most part.

If you need any more information please let me know. I have one system that will lock up in less than 12 hours after I enable pfBlocker,  I have one that takes up to 4 days, and another between 2 to 3 days.

Thank you
BreeOge
« Last Edit: October 11, 2017, 11:54:44 am by BreeOge »

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: New 502 Bad Gateway
« Reply #43 on: October 11, 2017, 12:19:25 pm »
I need specifics. Exact error messages, a list of features enabled in pfBlocker, etc. It's nice to know that it appears isolated there, so the impact isn't too large, but there isn't enough to go by here yet.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline john_galt

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +5/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #44 on: October 11, 2017, 12:33:19 pm »
Hi,

I'm having the same problem. I'd be glad to help but not sure where to look. The only error I get is when trying to connect to the GUI. The web browser appears to time out and the message "Bad Gateway nginx". If I try to use the serial console port it is also unresponsive. My only recourse is to power cycle the device.

I have looked at the logs but to be honest I wouldn't know what to look for. Like I said I'd be glad to help if given some direction as to what to provide.


Doug
Doug