pfSense Support Subscription

Author Topic: New 502 Bad Gateway  (Read 12014 times)

0 Members and 1 Guest are viewing this topic.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +958/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: New 502 Bad Gateway
« Reply #15 on: October 04, 2017, 03:19:15 pm »
Eh well, it not just overloads nginx, it overloads the entire box apparently as mentioned earlier.

Code: [Select]
Sep 25 11:25:10 kernel sonewconn: pcb 0xfffff8004c40f690: Listen queue overflow: 193 already in queue awaiting acceptance (9 occurrences)
[/code)
Do NOT PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #16 on: October 04, 2017, 03:39:01 pm »
Still good to go since I removed PFblocker, I am thinking it may have something to do with PFblocker / Snort/ or Squidguard working together on 2.4  What was updated on 2.4.0 on the 3rd?  I looked for patch notes, but could not find it.  Any file that shows what was updated on past updates?

When it does lock up, the SSH becomes unresponsive as well, it only shows the basic login, I have to CTRL-Z to close the process.

*** Welcome to pfSense 2.4.0-RC (amd64) on fort-smith ***
^ZYou have stopped jobs.
[1] + Suspended               /etc/rc.initial
# reboot

That is the only way to get it to free up.  If you try to restart PHP it will just lock up as well.

I also updated the listen Queue overflow, and the error for overflow does not happen now till WAY later, if its been locked up for 15 min or more.  That is a symptom.  But some file that changed on the 3rd is causing issues between pfblocker, snort, or squidguard working with each other.

Still working good since I removed Pfblocker, no errors what so ever in the logs, running like a champ.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +958/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: New 502 Bad Gateway
« Reply #17 on: October 04, 2017, 03:51:50 pm »
If you read the above message, your box is completely overloaded with queued ingress connections. So yeah, in that state NOTHING works properly. Even linked you to the kernel tunable.

Also:  http://nginx.org/en/docs/freebsd_tuning.html
« Last Edit: October 04, 2017, 03:55:06 pm by doktornotor »
Do NOT PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #18 on: October 04, 2017, 04:22:53 pm »
I really appreciate the help doktornotor, and I understand where your coming from and why your giving me the solutions you are suggesting.  I guess I have not explained it well enough though.  I understand that its being queued with ingress connections, but the problem I have is why does it only happen when I updated on the 3rd to the newest 2.4.0 version and not before, system load has not changed, actually if anything system load is currently low.  Problem only showed up on 2.4.1 and on 2.4.0 after Tuesdays update.  Before that I never had an issue with Overloaded ingress connections even with pfblocker installed.  I have had the system with a heaver load than it currently has and never received these errors.  I have tried the settings you suggested mine are currently

kern.ipc.somaxconn=4096

and

backlog= 4096

and I still have the issue.

However when I removed pfblocker the issue stopped, but its never been an issue before, and like I said its actually low load at the current time.  It only raised its head after the 2.4.0 update on the 3rd or on the 2.4.1 updates.  Before then even on the 2.4.0 I had no issues even during high loads.  That is why I posted in the 2.4 Development Snapshots, because it only showed up after an update, something changed to cause the problem.  All I did was update to newest version, no load change or anything else. Update then issue showed up.  Again I do appreciate your help, but I guess I will have to wait till more are having the issue.  I know they will have issues, I have 2 boxes set up, one only has one to two users at any given time so super low load and even it started after the update.

Thank you for your assistance. Sorry for taking your time.

BreeOge
« Last Edit: October 04, 2017, 04:26:10 pm by BreeOge »

Offline seanr22a

  • Newbie
  • *
  • Posts: 20
  • Karma: +2/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #19 on: October 05, 2017, 03:07:29 pm »
I'm seeing the same thing - 502 errors. I'm running three sites all of them worked perfect until 2.4.0-RC changed from bsd 11.0 to bsd 11.1 (a few days ago)
Strange thing it's only one site that have problems. The problem site is the site with most users/traffic.

This morning I removed pfBlocker on the problem site and so far everything good. It's to much of a coincidence that the problem started when the move was made from bsd 11 to 11.1 a couple of days ago.

I had to remove squid on all three sites (XMLRPC Sync stopped working giving a constant flow of errors) and now Pfblocker is removed on one site after the introduktion of bsd 11.1.



Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #20 on: October 05, 2017, 11:02:00 pm »
Yea, it happens more on the one that has higher traffic, but if you leave it even on the low traffic ones they will eventually get it as well, just takes longer. Its like its not releasing something, and keeps building till it locks up..

I removed all my PFblockers too.. Interesting side note as well

I have 3 systems, 2 of them have Pfblocker, squid, and squidguard installed. Those are the ones getting the 502.  I have another one set up, with only Squid, and Pfblocker, and Squidguard disabled.  It has yet to get a 502 Bad gateway.  It seems to be something with the 11.1 and either Pfblocker or Squidgard or all working together, but if you  have Squid and Squidguard and no pfblocker it does not happen, same goes with Squid and Pfblocker with no Squidguard.  Its only when all 3 are present.

Seanr22a do you have those 3 installed on all your systems or a mix?
« Last Edit: October 05, 2017, 11:06:27 pm by BreeOge »

Offline seanr22a

  • Newbie
  • *
  • Posts: 20
  • Karma: +2/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #21 on: October 06, 2017, 12:24:09 am »

Seanr22a do you have those 3 installed on all your systems or a mix?

I have two SG-2440 and one SG-8860. They all have Snort, Squid and Pfblocker. The configuration is identical at these sites so I use XMLRPC Sync in all three packages.
At the two sites still running Pfblocker I have moved the IP blocklists  to Snort and only using DNSbl in the Pfblocker package. I will wait a couple of days if its still fine I will install Pfblocker using only DNSbl at the main site as well and give it a try. My problem with squid is the XMLRPC Sync but all this started to happen with bsd 11.1

To me it looks like it's running out of resources of some kind. I have a lot memory and diskspace available when it happens so its something else.

« Last Edit: October 06, 2017, 12:28:37 am by seanr22a »

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #22 on: October 06, 2017, 09:07:06 am »
Quote
I have a lot memory and diskspace available when it happens so its something else.

Same here.. 

The release notes for 11.1 are here https://www.freebsd.org/releases/11.1R/relnotes.html#kernel

Lots of changed for the kernel.  I suspect one of these are the cause, due to it crashed the kernel but I could be wrong on that.   Also I have no idea what would cause it.  just something to look at.

I know it looks like the system is being overloaded like doktornotor said, but hard to believe a system that never overloaded before the update now overloads constantly with the same load or less.  Does not make sense.

Offline luckman212

  • Hero Member
  • *****
  • Posts: 725
  • Karma: +59/-0
    • View Profile
    • @luckman212 - github
Re: New 502 Bad Gateway
« Reply #23 on: October 06, 2017, 09:11:41 am »
Hmm I'm not running into any issues on my test 2.4.1 box, but to be fair, I'm not running Snort of pfBNG. Hope this doesn't turn out to be a nasty bug in the kernel that could delay 2.4.x getting out the door. Seems like every time 2.4 is about to drop something else comes up. Been 542 days since 2.3-RELEASE :)

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #24 on: October 06, 2017, 09:23:44 am »
Hmm I'm not running into any issues on my test 2.4.1 box, but to be fair, I'm not running Snort of pfBNG. Hope this doesn't turn out to be a nasty bug in the kernel that could delay 2.4.x getting out the door. Seems like every time 2.4 is about to drop something else comes up. Been 542 days since 2.3-RELEASE :)

Yea it is for sure related to the 11.1 Kernal, PfblockerNG, or Squidguard.  Remove one or the other (pfblockerNG, or SquidGuard) and issue does not exist.

Did a test, I had one system working for over 24 hours, since I removed PfblockeNG and no 502, put it back in and in less than 2 hours, crash.

I have one system with PfblockerNG installed, and squidguard disabled, no issues at all.  Enabled Squidguard, and crash in less than 2 hours.

So the issue lies between PfblockerNG, Squidguard and the Kernal. I would suspect, but i am no expert.

Offline MaxPF

  • Full Member
  • ***
  • Posts: 259
  • Karma: +1/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #25 on: October 07, 2017, 10:24:18 am »
Started having the same issue since the 2.4RC switch from 11.0 to 11.1. I am running pfBlocker. 

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #26 on: October 07, 2017, 12:59:18 pm »
Started having the same issue since the 2.4RC switch from 11.0 to 11.1. I am running pfBlocker.

Do you have Squidguard also?

Offline MaxPF

  • Full Member
  • ***
  • Posts: 259
  • Karma: +1/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #27 on: October 07, 2017, 06:59:11 pm »
Started having the same issue since the 2.4RC switch from 11.0 to 11.1. I am running pfBlocker.

Do you have Squidguard also?

No, I don't have squidguard. It hasn't happened yet with 2.4.0.r.20171005.0827.

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #28 on: October 08, 2017, 05:20:26 pm »
My system that has a SUPER low load finally 502 Bad gateway.. So that rules out anything to do with SquidGuard, it is an issue with Pfblocker, and 11.1 kernal. 

Offline seanr22a

  • Newbie
  • *
  • Posts: 20
  • Karma: +2/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #29 on: October 10, 2017, 01:55:59 am »
My system that has a SUPER low load finally 502 Bad gateway.. So that rules out anything to do with SquidGuard, it is an issue with Pfblocker, and 11.1 kernal.

I got my 502 problems sorted out but not in the way I want ...

FW1: SG-2440 4Gb memory, 20Gb disk running 2.4.0.r.20171006.2203 - Snort, PFblocker, Squid
FW2: SG-2440 4Gb memory, 20Gb disk running 2.4.0.r.20171006.2203 - Snort, PFblocker, Squid
FW3: SG-8660 8Gb memory, 40Gb disk running 2.3.4-RELEASE-p1  - Snort, PFblocker, Squid

The above setup works perfect, no 502 errors. I want to run the SG-8660 on 2.4 as well but it simply don't work. I get 502 errors and after a while it stops to respond completely. I don't get any 502 errors on the SG-2440 running 2.4 and they have been up since October 6. Note that they run the same packages.

The SG-8660 have a 8-core CPU, more memory and disk but can't run 2.4 with these packages. This firewall have lot more users/traffic during the weeks but I did a test this weekend when the office is closed so no users/traffic - it just sits idle. It takes 3-4 hours and I get 502 error and 1-2 hours later it stops respond completely.

Now i reinstalled it with 2.3.4P1 and everything have been fine for two days :)

Only bad thing is that I use XMLRPC Sync for all packages between these firewalls (all run identical setup), one of the SG-2440 is the master and the other ones slaves. That was working with all FW on 2.4 but now there is a mix of 2.3.4 and 2.4 and XMLRPC Sync don't seem to be able to handle that.

---------------
Communications error occurred

Exception calling XMLRPC method merge_installedpackages_section #1 : Unknown method   @ 2017-10-10 08:54:31
Exception calling XMLRPC method merge_installedpackages_section #1 : Unknown method   @ 2017-10-10 08:54:32
Exception calling XMLRPC method exec_php #3 : Incorrect parameters passed to method: Signature permits 2 parameters but the request had 1   @ 2017-10-10 08:54:33
Exception calling XMLRPC method exec_php #3 : Incorrect parameters passed to method: Signature permits 2 parameters but the request had 1   @ 2017-10-10 08:54:34
----------------

[EDIT]
Forgot to write that I get the same sync error no matter if it's snort, PFblocker or squid - they all give the same error message so I don't think it have to do with the packages.
« Last Edit: October 10, 2017, 02:06:08 am by seanr22a »