SG-1000 CPU usage and webgui hanging

[lordbob75]

I've had my SG-1000 for a few months now, and I really like it, though I've been noticing some issues.

My CPU usage seems to always be at 100%, and any reductions in usage last seconds at best.  This occurs whether there is mere kilobytes of data, or I max out my connection.

I have no real idea if this is related, but the webgui is extremely slow.  Sometimes it can take literally minutes to load a page.  I realize the hardware isn't super powerful, so perhaps that's all it is, but it seems like it's too slow.

Also possibly related, the webgui will hang and then return a page unavailable error through the browser (as if my network was down).  This mostly only happens when I max out my download (60Mbps), which should be just over half of what the sg-1000 is supposed to handle.  Now, my connection seems to stay up while this is happening as Netflix usually continues to work, but sometimes sites won't load.  I know this can happen when the connection is overloaded, so I turned off the downloads, but the webgui doesn't want to come back and the connection issues to websites continue until I reboot the sg-1000.  I am not able to SSH into it either, it simply never connects.  It doesn't even give me a connection error.

I'm not sure this is the sg-1000 causing all of this, but it seems related.  Any ideas?  Any assistance is appreciated.

[jimp]

Connect to the console and watch the output of "top -aSH" at a shell prompt while you use the device. That will be more accurate than watching from the GUI, which consumes a lot of CPU itself.

What packages or services do you have enabled on the firewall? Any traffic shaping or limiters?

[rnatalli]

I've been testing one of these and it's a sensitive little sucker.  I'm running an OpenVPN client, PFBlocker NG, and Watchdog and it runs okay for SOHO.  One of the tricks is making sure none of your packages are updating too frequently and/or at peak times.  For example, when I setup PFBlocker originally, it was set to update its lists every couple hours and that would hang the interface, although the network traffic was flowing for the most part.  Set this for 3am and this problem has disappeared.

[chrcoluk]

I discovered recently certain tasks will cause the GUI to hang.

e.g. if my WAN goes down, the process of attempting to reconnect can cause the GUI to hang, maybe this is one of the reasons for the planned changed after 2.4.

If the GUI does get stuck you can force a restart of php-fpm from the shell menu, but that can break tasks currently been run by any scripts.

[lordbob75]

Connect to the console and watch the output of "top -aSH" at a shell prompt while you use the device. That will be more accurate than watching from the GUI, which consumes a lot of CPU itself.

What packages or services do you have enabled on the firewall? Any traffic shaping or limiters?

I've got:
BandwidthD (removing this though)
Net-SNMP (will remove in favour of telegraf)
openvpn-client-export
PFBlockerNG
Service_WatchDog
Softflowed (will remove in favour of telegraf)

PFBlockerNG just does DNS adblocking, no shaping.  I never setup any limiting or shaping.

I'll have to try console if this happens again.

I've been testing one of these and it's a sensitive little sucker.  I'm running an OpenVPN client, PFBlocker NG, and Watchdog and it runs okay for SOHO.  One of the tricks is making sure none of your packages are updating too frequently and/or at peak times.  For example, when I setup PFBlocker originally, it was set to update its lists every couple hours and that would hang the interface, although the network traffic was flowing for the most part.  Set this for 3am and this problem has disappeared.

My PFBlockerNG lists update one list per day late at night.  They were not running at the time this occured.

I discovered recently certain tasks will cause the GUI to hang.

e.g. if my WAN goes down, the process of attempting to reconnect can cause the GUI to hang, maybe this is one of the reasons for the planned changed after 2.4.

If the GUI does get stuck you can force a restart of php-fpm from the shell menu, but that can break tasks currently been run by any scripts.

Makes sense.  I figured there was a good chance it was just overloaded.  I cannot get to the shell from SSH though, will have to try on console next time.

[jimp]

"PFBlockerNG just does DNS adblocking"

That feature can use quite a lot of memory, and the SG-1000 does not have a lot of memory to start with. When a FreeBSD system runs out of RAM one of the symptoms is that processes for the GUI/SSH/Console may stop responding until the RAM demand goes down. Eventually the process that requested the RAM and failed will stop and it will come back. If you see a serial console message about being out of swap space, then you need to cut down the number of things running or options configured since you're probably running it out of RAM.

[lordbob75]

"PFBlockerNG just does DNS adblocking"

That feature can use quite a lot of memory, and the SG-1000 does not have a lot of memory to start with. When a FreeBSD system runs out of RAM one of the symptoms is that processes for the GUI/SSH/Console may stop responding until the RAM demand goes down. Eventually the process that requested the RAM and failed will stop and it will come back. If you see a serial console message about being out of swap space, then you need to cut down the number of things running or options configured since you're probably running it out of RAM.

Interesting.  That's unfortunate, but makes sense.  I assume there is no way to add memory to it?  I'll have to check on the memory usage and see if it's possible to reduce.  I know some of my lists are probably unnecessary so I can probably trim it down some.

Thanks for the assistance!

[mezzoman]

I have this same issue, even with no packages.  (I had PFBLocker and OpenVPN installed but I removed them just to see if it would have a positive effect).  CPU at 100% continuously in the webGUI.  I did connect with a USB console cable and checked the processes with top -aSH.  netstat was at times 1200%+ of CPU.  It was immediately niced, but over the course of ~60 seconds it popped to the top (punn intended) 10 or so times ranging from 500% of CPU to 1200%.  Any insight is appreciated.  This is with no extra packages and even the internal DNS resolver/forwarder turned off.  (It was unbearable with it on ;-)

Thanks!

[Grimson]

https://forum.pfsense.org/index.php?topic=139255.0

[mezzoman]

Thanks!