Netgate SG-1000 microFirewall

Author Topic: WebGUI and SSH not functional  (Read 303 times)

0 Members and 1 Guest are viewing this topic.

Offline McFly80

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
WebGUI and SSH not functional
« on: December 27, 2017, 11:37:50 am »
Greetings,

I had the firewall lock up a few days ago and I couldn't SSH in, couldn't get into the WebConfigurator and it was running headless, so I hard rebooted it. Seemed to come back up fine and routing, firewall and other functions worked (DHCP/DHCPv6/etc.). Afterwards though I cannot access the WebGUI or SSH, at all - just times out.

I've tired disabling pfctl, loading the debug ruleset, even loading a saved config from the backups after hooking up a monitor and keyboard. I've tried graceful reboots and a variety of things...

I'm running 2.4.2 currently. I've tried restarting WebConfigurator, restarting PHP-FHP... when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:

nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)

Any ideas? I'd prefer to not delete and redo as it's got a ton of work on the IPv6-PD and the reservations. haha

Thanks!

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2581
  • Karma: +207/-9
    • View Profile
Re: WebGUI and SSH not functional
« Reply #1 on: December 27, 2017, 11:47:12 am »
Hi,
... when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:
nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)
Any ideas?
Well : what about this one : no more memory !?
Can you develop that one ? Like how much installed ? What other memory eaters (also called packages), etc.

Btw : No GUI (we know why) and no SSH (the ssh also abandoned, check log for reason) so you are using the console access. Run "top" to see more info.

Offline McFly80

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: WebGUI and SSH not functional
« Reply #2 on: December 27, 2017, 01:49:35 pm »
I totally agree it's "possible" but it's got 8GB of memory on a Core i5... I've got 5450MB free currently on top.

Even for CPU use, when I run top, it's the highest user of resources. Swap has 16GB and 16GB free. :)
« Last Edit: December 27, 2017, 02:01:29 pm by McFly80 »

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2581
  • Karma: +207/-9
    • View Profile
Re: WebGUI and SSH not functional
« Reply #3 on: December 27, 2017, 02:47:34 pm »
Great, no apparent memory issue.

Next focus :
Who are the zombies on your system - I've none. Dead nginx instances ?
Easy to check also : stop all processes that are activated by packages. Also : you have only access to the console so it's more a manual operation to "stop" package so the won't restart on reboot. Visit  /usr/local/etc/rc.d and do some clean up (copy the related script files out of the way, for example, move them to /root/) and restart - see if the GUI comes up now.

My "top" command :
Quote
Mem: 34M Active, 280M Inact, 340M Wired, 184M Buf, 1288M Free
(I have pfSense running on 4Gb)
Your "Buf" size is 3 times smaller then mine ? (I don't know what "Buf" really is, except that it is reserved work kernel space for communication)

Btw : this is what I should do with my setup - not some sort of "you should do this and all will be fine".

Offline McFly80

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: WebGUI and SSH not functional
« Reply #4 on: December 27, 2017, 03:34:38 pm »
Hi there,

Checked it out - they're all bandwidthd processes that are zombies... 8 of 'em.

I'll have to check the startup and logging, kinda slow.

Thanks!

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2581
  • Karma: +207/-9
    • View Profile
Re: WebGUI and SSH not functional
« Reply #5 on: December 28, 2017, 08:24:57 am »
.... all bandwidthd processes ...
I'll bet that bandwidthd as allocating all internal "Buf" (limited !) memory. With a final result that the GUI web server (nginx) would (re) start anymore.

edit : have a look at this https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
« Last Edit: December 28, 2017, 08:28:30 am by Gertjan »

Offline McFly80

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: WebGUI and SSH not functional
« Reply #6 on: January 02, 2018, 02:11:31 am »
Well, I've tried these buffer changes... no luck.

Why can't I ssh in either? Why would a lockup break SSH also? It's like it reordered the firewall rules and locked me out.

I'm trying to manually remove rules on this now... but I'll tell you - I didn't go in and heavily edit things on the install - so if running a few downloads of Linux ISO torrents does this - I'm at a loss how people don't see some crazy issues in production environments. lol

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2581
  • Karma: +207/-9
    • View Profile
Re: WebGUI and SSH not functional
« Reply #7 on: January 02, 2018, 04:37:55 am »
What about a simple console access - and goto default.
I'll bet all will be fine and up afterwards.

Then add your settings, rules, etc, and make a pause between each step - test.

As soon as something breaks you will know precisely what not to do, and you have the console to go one step back.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 15736
  • Karma: +1469/-210
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: WebGUI and SSH not functional
« Reply #8 on: January 02, 2018, 04:53:45 am »
"It's like it reordered the firewall rules and locked me out."

There is a specific rule to prevent that - the antilockout rule that allow the port the gui listens on and the ssh port.. Did you disable this rule?

Did you create a rule in floating that happens before interface rules that overrode the antilockout rule?  Are you coming from a different network and not the lan network that doesn't have the antilock out rule?
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.3-RELEASE (work)
1x SG-3100 2.4.3-RELEASE (work)
1x SG-4860 2.4.3-RELEASE (home)