pfSense Support Subscription

Author Topic: RRD-Data doesn't survive a restart / shutdown  (Read 2401 times)

0 Members and 1 Guest are viewing this topic.

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
RRD-Data doesn't survive a restart / shutdown
« on: June 07, 2014, 03:05:29 pm »
When I do a shutdown of my pfSense (2.1.3 on an Alix 2d13) and restart the box again my RRD graphs only show data starting from the last shutdown. I would really like this data to be kept over reboots.

In System / Advanced / Miscellaneous I have configured the setting "Periodic RRD Backup" to 24 hours. I expected that this should do the trick.

I assume I'm missing something obvious, some other setting I should know?

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #1 on: June 09, 2014, 01:49:59 pm »
Anybody?

Is anyone here experiencing the same problem at all or does this work for everyone else?

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11959
  • Karma: +469/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #2 on: June 09, 2014, 04:07:11 pm »
It should survive across a reboot.
Which install type are you running?
How are you rebooting the machine?

Steve

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #3 on: June 09, 2014, 11:42:55 pm »
Thank you Steve for helping!

I run an Alix 2d13 from a flash card (4GB). I did a shutdown these days (because I had to replace something with the power supply). If packages should be relevant: I have freeradius2 and pfBlocker (however disabled).

I used Diagnostics / Halt System to shutdown. I disconnected the box from the power supply after I got no more replies to pinging the box and waiting some more.

Even when an unexpected power outage should occur I would have expected that the RRD data would at least be kept until the latest backup, so at a maximum 24 hours of data should get lost.

I had this problem already before only now I looked exactly what happened.

I'm not sure whether this makes a difference ( I think it shouldn't):

  • Before shutdown I duplicated slices. (The correct way round. I can be sure because I made some changes to the configuration before.) I did not switch boot slices.
  • Before shutdown I also backuped the configuration excluding the RRD data.


chemlud

  • Guest
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #4 on: June 10, 2014, 01:39:11 am »
Don't know if it matters:

I have Diagnostics -> nanoBSD

"Keep media mounted read/write at all times. " checked...

(as I run Snort on the CF card, too...)

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #5 on: June 10, 2014, 04:04:09 am »
At the moment it says "Read/Write (reference count 4)". I don't know how and when this changes and how to find these references. I would have assumed that the RRD data backup would switch to read/write on its own.

But who knows, something is odd anyway: I have checked "Keep media mounted read/write at all times." now. If that makes a difference I should know, when I do a reboot after 24 hours. I will let you know.

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #6 on: June 10, 2014, 04:14:19 am »
I have checked the logs for obvious problems with RRD but didn't find any direct hint. However I had a thought: From time to time I have log entries like this:

Code: [Select]
kernel: pid 73592 (php), uid 0, was killed: out of swap space
My memory utilization is at nearly 80%. Any process needing large amounts of memory could possibly fail.

Perhaps this happens to a dedicated backup script regularly? Is there a dedicated backup script? (I found this running: /var/db/rrd/updaterrd.sh.) If so how do I find out which this is and when it will be triggered?

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11959
  • Karma: +469/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #7 on: June 10, 2014, 05:11:52 am »
It should backup the RRD data both periodically and when you shutdown correctly, as you are doing. The backup and the config file are help on the /conf/ slice which is independent of the two boot slices in Nano. I suspect that you are running out of RAM when it tries to back up. Check the date on file /conf/rrd.tgz and /conf/backup/rrd.tgz.

Steve

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #8 on: June 10, 2014, 10:39:15 am »
Now that's interesting. There is a file /conf/rrd.tgz with a size of approx. 2MBytes. Timestamp is today 0:00.

(Unfortunately I destroyed it now: The "Load" button on the Edit File Dialog does NOT load the file to my local computer ... shame on me. ::).) Btw. How do I transfer a file from the pfSense to my machine? scp didn't work?

But: no damage done. I can view the current graphs with data back until last reboot now. This data will probably be contained in the next backup. If not that's no problem either.

Is the current data persisted somewhere also or is this kept in memory only?

Result: The rrd backup facility seems to work. I can verify this when I get the next backup (tomorrow) and manage to download and examine the file.

This means there probably is a problem with using the backuped data. How can this happen?

Offline KOM

  • Hero Member
  • *****
  • Posts: 5497
  • Karma: +680/-23
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #9 on: June 10, 2014, 12:23:24 pm »
You can use the File Manager package.  Once installed, look for it under Diagnostics - File Manager.

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11959
  • Karma: +469/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #10 on: June 10, 2014, 02:33:47 pm »
From Windows WinSCP works though it maynot be using SCP because from Linux use sftp://root@yourpfsensebox.

Steve

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #11 on: June 10, 2014, 06:53:55 pm »
Oh well, sftp works if using root (not admin ...)

The backup files seem to contain data back about 3 days, not longer.

That means: There is a working backup procedure. These possibilities come to mind now: On a reboot the backup might get destroyed. Or it may be overwritten with the next backup after a reboot. Or when the data gets restored from the backup the first time the data cannot be read and is dumped.


Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11959
  • Karma: +469/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #12 on: June 11, 2014, 06:08:19 am »
If there's a fault with the file the system logs should be full of errors at boot time. If you have it set to backup periodically (at 24hr periods) the files should be at most 1 day old I would have thought.

Steve

Offline -flo-

  • Sr. Member
  • ****
  • Posts: 374
  • Karma: +29/-0
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #13 on: June 11, 2014, 11:10:06 am »
I did a reboot just a few minutes ago to observe the log messages during boot. After the reboot the RRD graphs are as complete as they were before. No data got lost. Needless to say there were no apparent errors in the log.

That's fine but I know there was a problem last time I did this. Now I don't even know how to analyze this. :-(

Thank you for helping so far. I give up on this now.

Offline stephenw10

  • Administrator
  • Hero Member
  • *****
  • Posts: 11959
  • Karma: +469/-15
    • View Profile
Re: RRD-Data doesn't survive a restart / shutdown
« Reply #14 on: June 11, 2014, 11:24:33 am »
Without actually having logs for when this happened it's hard to say but I would still guess your Alix ran out of ram.
Do you have the RRD memory usage graph for when it happened?

Steve