Netgate SG-1000 microFirewall

Author Topic: RRD-Data doesn't survive a restart / shutdown  (Read 2505 times)

0 Members and 1 Guest are viewing this topic.

Offline darkcrucible

  • Jr. Member
  • **
  • Posts: 86
  • Karma: +5/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #15 on: June 11, 2014, 01:48:21 pm »
I only skimmed this thread but this issue happens nearly every time I reboot/poweroff my alix board. If you hook up a serial cable during the bootup, it'll say something about RRD graphs...(killed). If that happens, then it's all gone. The LEDs at this point will likely be in the knight rider mode.

The way I avoid it is to disable all OpenVPN instances prior to a reboot and not install any packages at all, especially the OpenVPN client export. This allows enough free memory during startup to avoid the RRD from getting killed.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21747
  • Karma: +1503/-26
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #16 on: June 11, 2014, 03:20:13 pm »
How many interfaces do you have? It's easy to run the ALIX out of RAM, especially if you have more than a handful of interfaces, which can also fill up the tiny RAM disk on ALIX.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 12275
  • Karma: +494/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #17 on: June 11, 2014, 03:38:01 pm »
The fact that Flo is running Snort pretty much ensures low RAM.  ;)

Steve

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 392
  • Karma: +31/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #18 on: June 13, 2014, 04:54:14 am »
Actually I'm not running Snort! (I would have been interested but I discovered before installation that this would require more potent hardware.) I have pfBlocker installed (but not activated) an I use freeradius2. I can uninstall pfBlocker.

Regarding the number of interfaces: Currently I have 7 interfaces, of which I could delete 1 or 2. (One interface is not used anymore. A MODEM interface on the WAN side collects incoming traffic that is generated by the modem and is therefor not VLAN tagged by my provider. I want to block this traffic in the firewall silently. Apart from that this MODEM interface is not really necessary.)

Thank's to darkcrucible for his information! Until now I had no clue as to where and when this problem actually occurs. This seems clear now: Lack of memory kills the process which is responsible for restoring the backuped data during boot.

I will now reduce the number of interfaces and remove pfBlocker. As I do not use OpenVPN there is nothing to gain here. I hope this will help otherwise I will probably have to live with the problem.

Btw.:

Quote
The LEDs at this point will likely be in the knight rider mode.

I noticed that "knight rider mode" but I don't know when this occurs and what it should tell me. What is this all about?

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 12275
  • Karma: +494/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #19 on: June 13, 2014, 07:49:45 am »
Actually I'm not running Snort!
Ah, my bad. I attributed Chemlud's comment about Snort to you.  :-[

My understanding is that when the RRD data is backed up all the files are gathered in one place before being compressed into the tgz file you see in /conf. At some point in that process all the files are held in RAM with only 256MB to play with it's easy to run out.
If you've upgraded from an earlier version you may find that you have some RRD files for interfaces which have changed names that are no longer in use but are still taking up space/being backed up. For example I have a file for 'WAN-quality' and 'GW_WAN-quality' but they are no longer updated the actual file is now 'WAN_PPPOE-quality'. I also have files for gateways I used to have, on my modem interface, which you may have. You can access to modem without having a gateway on the interface.

I did once suggest the possibility of moving either /var or /tmp onto a USB drive to free up RAM in low RAM systems. As JimP said at the time it would never be supported and could cause problems if/when the drive eventually failed. Maybe something to consider if all else fails.

Steve
« Last Edit: June 13, 2014, 07:54:37 am by stephenw10 »

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 392
  • Karma: +31/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #20 on: June 13, 2014, 11:17:05 am »
RRD data about deprecated interfaces is not a problem as far as disk space is concerned. My CF card has plenty of space.

Are you sure that all files must be held in memory simultaneously when the tgz file is created? Aren't there pipes for handling such stuff more memory efficiently? I will check now from time to time whether the backups are successful. According to darkcrucible's comment the problem is that the data is destroyed at boot time.

Using a usb drive sounds dangerous to me. This would be ok only if the system can boot even if the usb drive fails. I have no clue as to how I would configure this anyway.

About the modem interface: This is somewhat complicated. I have two interfaces WAN and WAN_IPTV which have VLAN tagged traffic from my provider. My modem assigns itself an RFC 1918 IP address and unfortunately generates plenty of UPNP service discovery requests. I want to block this silently because otherwise this spams my firewall log. But I want to keep the logging of the default block rule. These packets arrive neither on the WAN nor the WAN_IPTV interface. So any firewall rules there do not work. This does not even work with floating firewall rules. Nonetheless in the firewall log the block get attributed to the WAN interface's default block rule. So to be able to write a firewall rule for the modem at all I have to create an interface for it first. ::)

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 12275
  • Karma: +494/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #21 on: June 13, 2014, 04:50:36 pm »
Yes, it's not an issue with CF card space nor, normally, with space in /tmp or /var which are the same size across all Nano installs. That said I had to increase the size of /var on my home box to 80MB to avoid errors a few versions back. Can't remember the details now.  :-\ I have 512MB in the box, and could easily add more, so it's much less of an issue.
Edit: Yes here we go, anything in this thread look familiar:
https://forum.pfsense.org/index.php?topic=66588.0
I was running out of space in /var every time the RRD backup ran at midnight.

No, I'm not sure of anything really.  ;) I haven't investigated the code closely but I seem to remember something reported by one of the devs to that effect. Maybe it's more of an issue when the compressed data is extracted.

Like I said moving data onto a USB stick would be a last ditch solution.

Ah, I see what you're saying about the modem interface. If it doesn't have a gateway on it that there won't be an RRD quality file created for it though.

Steve
« Last Edit: June 13, 2014, 04:59:31 pm by stephenw10 »