pfSense Gold Subscription

Author Topic: New 502 Bad Gateway  (Read 11539 times)

0 Members and 1 Guest are viewing this topic.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +957/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: New 502 Bad Gateway
« Reply #45 on: October 11, 2017, 12:39:15 pm »
Is this happening on current 2.4.0 snapshots or only 2.4.1?

You know it'd be a whole lot easier to trace if you stopped the ridiculous "top secret" kernel commits. Annoying, stupid, plus no credits for screaming about open-source solutions and playing retarded games like this.
Do NOT PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #46 on: October 11, 2017, 12:40:04 pm »
I need specifics. Exact error messages, a list of features enabled in pfBlocker, etc. It's nice to know that it appears isolated there, so the impact isn't too large, but there isn't enough to go by here yet.

Ill get that info for you sir..

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #47 on: October 11, 2017, 01:32:21 pm »
Is this happening on current 2.4.0 snapshots or only 2.4.1?

You know it'd be a whole lot easier to trace if you stopped the ridiculous "top secret" kernel commits. Annoying, stupid, plus no credits for screaming about open-source solutions and playing retarded games like this.

Well, you know your comments about it being overloaded was annoying, stuipd, because it was the first thing I tried before I even posted about the issue.  But I didn't say anything negative to you about it, thought maybe I had missed something.  I hadn't but I didn't slam you for the suggestion, because all we want is to find the fix.  You want to say its got nothing to do with the kernel, yet it only happened when the kernel was updated.   So not to ridiculous as you say,  It may not be the Kernel itself, its probably something else, but that is as close as we have to go on since it effects multiple things.  You concentrate on one thing, and refuse to read what everyone else says its also doing.  It's probably a fix that can be done in pfBlockernNG to conform to 11.1.  We are trying to get to the solution, what are you doing?  Complaining and giving us old threads that we have already tried.

Definition of insanity = doing the same thing over and over again and expecting different results.

We appreciate you trying to help, but having the i'm better than you attitude is not helping anyone or this problem.  So in the future do not say anything is ridiculous if your not assisting in TRACING the issue.  I would be happy to do any tracing to find the issue, like me and other have stated.  Nothing in the System.log, nothing in the php log.  If you have any suggestions on other logs that could help track down the issue please post them, I will try them in a heartbeat.  But don't call anyone or anything ridiculous, that is no different than you saying you are a know it all.  All the suggestions you suggested were Kernel suggestions.  I thought you knew what you was talking about so thats why we say Kernel.

Thank you, now lets please resolve this issue without belittling people.

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #48 on: October 11, 2017, 01:50:10 pm »
I need specifics. Exact error messages, a list of features enabled in pfBlocker, etc. It's nice to know that it appears isolated there, so the impact isn't too large, but there isn't enough to go by here yet.

Ill get that info for you sir..

My config for pfBlocker

General Default settings.
DSNBL Default settings
I have 2 DSNBL Feeds, mostly for ads, and Malware
I have an IPv4 list, block Malware and torrents and sip attacks.
No Reputation
No Geoip locations selected

The only Error messages we get are 502 Bad Gateway, but we can not access SSH without closing out process with CTRL-Z, and commands such as find, ect do not work. they just hang.

Thank you

Can someone else that is also having this problem give the pfBlocker settings, maybe we can find a common element.

Offline john_galt

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +5/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #49 on: October 11, 2017, 01:56:55 pm »
I will when I get home.


Doug
Doug

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +957/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: New 502 Bad Gateway
« Reply #50 on: October 11, 2017, 01:59:07 pm »
It's probably a fix that can be done in pfBlockernNG to conform to 11.1.  We are trying to get to the solution, what are you doing?  Complaining and giving us old threads that we have already tried.

No, it cannot. pfBlockerNG is merely using the pf firewall, that's it. It's using lighttpd as a 1x1 px webserver, not even nginx. It's using Unbound as resolver to redirect the requests to that webserver. It's doing absolutely nothing that should cause any box to hang and become unresponsive.

Thank you, now lets please resolve this issue without belittling people.

Sorry to have upset you. The only cases of the 502 Bad Gateway I (and pretty much anyone else) have seen been caused by simply not enough processes to serve the nginx/php-fpm requests (and/or exhausting the connections limit altogether). Now I'll need to get a crystal ball for cases when someone goes on secret commit spree that eventually makes a giant kaboom with pretty much every core package out there.

The devs can perhaps assist with tracing, meanwhile I'm simply once again annoyed by the course taken here. We've had this some ~3 years ago when the repos have been taken offline altogether, accompanied by some giant noise about trademark violations. It's getting old. Waste of time. Annoying. Disrespectful to people who've been contributing to pfSense, even after that CLA/copyright assignment/re-licensing nonsense etc. People wanting to build this thing for development purposes get absolutely zero assistance whatsoever and are being deliberately sabotaged by the so-called build scripts.

Open-source touted all over the website, and all you get is this. Getting on a tipping point again here.
« Last Edit: October 11, 2017, 02:03:47 pm by doktornotor »
Do NOT PM for help!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: New 502 Bad Gateway
« Reply #51 on: October 11, 2017, 02:48:04 pm »
DSNBL Default settings
I have 2 DSNBL Feeds, mostly for ads, and Malware

Approximately how many clients do you have on your local network that hit DNSBL? Any idea how busy it usually is?

The only Error messages we get are 502 Bad Gateway, but we can not access SSH without closing out process with CTRL-Z, and commands such as find, ect do not work. they just hang.

Most likely you just need to start a proper shell, try running /bin/tcsh
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #52 on: October 11, 2017, 03:00:29 pm »
DSNBL Default settings
I have 2 DSNBL Feeds, mostly for ads, and Malware

Approximately how many clients do you have on your local network that hit DNSBL? Any idea how busy it usually is?

The only Error messages we get are 502 Bad Gateway, but we can not access SSH without closing out process with CTRL-Z, and commands such as find, ect do not work. they just hang.

Most likely you just need to start a proper shell, try running /bin/tcsh


I have one site that has currently has

LAN   192.168.1.175   192.168.1.199   = 5 users
LAN   192.168.1.5   192.168.1.174   = 19 users

This site will lock up in less than 12 hours.


Another site

LAN   192.168.1.41   192.168.1.50   = 3 users
LAN   192.168.1.5   192.168.1.40   = 4 users

It will take up to 24 hours.

and another


Interface   Pool Start   Pool End   # of leases in use
LAN   192.168.16.50   192.168.16.75   = 5 users

This one will take up to 24 to 48 hours. However since the last update 2.4.0.r.20171009.1758 it has been happening more frequent around every 12 hours itself.

I currently have the pfBlockerNG uninstalled on the top 2 because they are used for apartments, and didn't want to bug the tenants when it locks up the CP becomes unresponsive.  However for testing purpose I do re-install, and let it run with any changes for testing.  I can usually get a crash within 6 to 12 hours. or less sometimes.

Just a note the amount of users, has not changed from before to after this issue started.  Load has remained the same.

Thank you, I will test the /bin/tcsh and see if that gives me a good shell again.  I will test that now on the top 2, will have a result back after it crashes.

Thank you, any logs you would like to see, please let me know I will post as much as I can.

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #53 on: October 11, 2017, 03:03:58 pm »
[
The devs can perhaps assist with tracing, meanwhile I'm simply once again annoyed by the course taken here. We've had this some ~3 years ago when the repos have been taken offline altogether, accompanied by some giant noise about trademark violations. It's getting old. Waste of time. Annoying. Disrespectful to people who've been contributing to pfSense, even after that CLA/copyright assignment/re-licensing nonsense etc. People wanting to build this thing for development purposes get absolutely zero assistance whatsoever and are being deliberately sabotaged by the so-called build scripts.

Open-source touted all over the website, and all you get is this. Getting on a tipping point again here.

I never knew about that, I do not have any bad intentions for this.  Sorry if I came off that way, I was not aware.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21312
  • Karma: +1423/-26
    • View Profile
Re: New 502 Bad Gateway
« Reply #54 on: October 11, 2017, 03:06:43 pm »
If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:

Code: [Select]
/usr/bin/netstat -Ln
/usr/bin/netstat -xn
/usr/sbin/swapinfo -h
/usr/bin/top | /usr/bin/head -n7
/bin/ps uxawwd
/usr/bin/sockstat

Attach the output in a text file as it will be too large to put inline on a forum post.
« Last Edit: October 13, 2017, 12:34:08 pm by jimp »
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline john_galt

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +5/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #55 on: October 11, 2017, 03:25:40 pm »
I got home and my pfSense GUI and Serial Console was unresponsive. As stated earlier from a client point of view everything worked, wired and wireless connections in and outbound.

My setup is is almost the same as BreeOge's, with even less users. Only me with a lot of gadgets. When I left for work at 6am I had just restarted and everything worked. When I got home at 4pm it was in the reported condition.

2.4.0-RC (amd64)
built on Mon Oct 09 17:58:12 CDT 2017
FreeBSD 11.1-RELEASE-p1

What else can I provide?


Edit.... I also have openVPN with one user.


Doug
« Last Edit: October 11, 2017, 03:46:03 pm by john_galt »
Doug

Offline AhnHEL

  • Hero Member
  • *****
  • Posts: 636
  • Karma: +18/-0
  • It is what it is.
    • View Profile
Re: New 502 Bad Gateway
« Reply #56 on: October 11, 2017, 03:49:08 pm »
I have one box using the ZFS file structure, the other is using UFS, both using pfBlockerNG.  The ZFS is rock solid, and the UFS one gets the Bad Gateway after some time.  Wondering if that is a possible reason why two similar boxes with similar settings exhibit different behavior using the same snapshot and same packages.

Both running 20171009 Snapshots for 2.4.0

Just a thought
AhnHEL (Angel)
NYC

2 pfSense sites: 2.4.2 (amd64)
Dell 755 SFF E6550 @ 2.3Ghz, 4GB RAM, 100/30 Mbps, Intel X3959
Dell 7010 SFF i5-3570 @ 3.4Ghz, 8GB RAM, 940/880 Mbps, Intel X3959
OpenVPN (Road Warrior), pfBlockerNG, Gaming


Offline john_galt

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +5/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #57 on: October 11, 2017, 03:53:09 pm »
AhnHEL

I don't know but I had planned to do a fresh install of my UFS pfSense to ZFS this weekend and restore the same config.


Doug
Doug

Offline BreeOge

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +6/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #58 on: October 11, 2017, 06:42:17 pm »
I have one box using the ZFS file structure, the other is using UFS, both using pfBlockerNG.  The ZFS is rock solid, and the UFS one gets the Bad Gateway after some time.  Wondering if that is a possible reason why two similar boxes with similar settings exhibit different behavior using the same snapshot and same packages.

Both running 20171009 Snapshots for 2.4.0

Just a thought

Very interesting mine are UFS as well..

Offline john_galt

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +5/-0
    • View Profile
Re: New 502 Bad Gateway
« Reply #59 on: October 11, 2017, 06:53:00 pm »
BreeOge

I have a time frame on my pfSense. I had to reboot when I got home at 4pm. It's now 7:50pm and I am getting the 502 Bad Gateway error. I'm going to reboot and scrub the system log between those times and if someone want's to look at it fine. Regardless I'm going ahead with my plans to reinstall this weekend.


Doug
Doug