pfSense Gold Subscription

Author Topic: Found a small error "Remote_Config_Backup" in Wiki.  (Read 516 times)

0 Members and 1 Guest are viewing this topic.

Offline mrzaz

  • Full Member
  • ***
  • Posts: 163
  • Karma: +4/-0
  • Beta Errors Eradicate Roguecode or BEER for short.
    • View Profile
Found a small error "Remote_Config_Backup" in Wiki.
« on: May 06, 2017, 12:18:21 pm »
Hello Jim,

I think I found a small error in the "Remote_Config_Backup" in Wiki. Or at least differences depending on what platform it is executed on.
https://doc.pfsense.org/index.php/Remote_Config_Backup

I have found that the script for taking backup and more specific the "donotbackuprrd=no" stopped working between 2017-02-01 -> 2017-03-01 timeframe
(which is the monthly backup schedule) where it stopped backup the XML with full RRD.

I started to check the script and made a change from "donotbackuprrd=no" to "donotbackuprrd=0" and then it started working again.
Please update the WIKI page with this.  Possible a note that on some platforms "=0" is needed instead of "=no".

All my backups taken with donotbackuprrd=no does NOT contain RRD data but when I changed to "donotbackuprrd=0" it started working directly.
Script running on Synology DS713+ with DSM 6.1-15047 Update 1

Code: [Select]
root@DiskStation3TB:/volume1/web_backend/tools# /usr/bin/wget --version
GNU Wget 1.15 built on linux-gnu.

+digest +https +ipv6 -iri +large-file -nls +ntlm +opie +ssl/openssl


My working script now contains:

Code: [Select]
#!/bin/ash
BACKUPDIR="/volume1/BACKUPNEW/pfsensebak/backup/daily"
USERNAME="<removed>"
PASSWORD="<removed>"
PORT="80"
SITES="x.x.x.x"
ZIP="/usr/bin/zip"
FIND="/usr/bin/find"
RMFILE="/bin/rm"
WGET="/usr/bin/wget"
BACKUPDAYS="30"
RMFILE="/bin/rm"
cd /volume1/web_backend/tools
for site in $SITES

do
       $WGET -qO- --keep-session-cookies --save-cookies /tmp/cookies.txt --no-check-certificate --timeout=10 http://$site:$PORT/diag_backup.php | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/csrf.txt
       $WGET -qO- --keep-session-cookies --load-cookies /tmp/cookies.txt --save-cookies /tmp/cookies.txt --no-check-certificate --post-data "login=Login&usernamefld=$USERNAME&passwordfld=$PASSWORD&__csrf_magic=$(cat /tmp/csrf.txt)" --timeout=10 http://$site:$PORT/diag_backup.php  | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/csrf2.txt
       if [ -e /tmp/csrf2.txt ]; then
       DATETMP="`date +%Y%m%d-%H%M%S`"
       FILENAME1="$BACKUPDIR/config-$site-$DATETMP.xml"
       FILENAME2="$BACKUPDIR/config-$site-$DATETMP-withrrd.xml"
                  $WGET --keep-session-cookies --load-cookies /tmp/cookies.txt --no-check-certificate --post-data "download=download&donotbackuprrd=1&__csrf_magic=$(head -n 1 /tmp/csrf2.txt)" --timeout=10 http://$site:$PORT/diag_backup.php -O $FILENAME1 > /dev/null 2>&1
                  $WGET --keep-session-cookies --load-cookies /tmp/cookies.txt --no-check-certificate --post-data "download=download&donotbackuprrd=0&__csrf_magic=$(head -n 1 /tmp/csrf2.txt)" --timeout=120 http://$site:$PORT/diag_backup.php -O $FILENAME2 > /dev/null 2>&1
                  rm -f /tmp/cookies.txt
                  rm -f /tmp/csrf.txt
                  rm -f /tmp/csrf2.txt
       $ZIP -q -9 -j $FILENAME1.zip $FILENAME1
       $ZIP -q -9 -j $FILENAME2.zip $FILENAME2
       $RMFILE $FILENAME1
       $RMFILE $FILENAME2
else
       echo "Failed to retrieve backup from $site"
fi
                                                                                                   
done
                                                                                                             
$FIND $BACKUPDIR -type f -name "*.xml.gz" -mtime +$BACKUPDAYS -exec rm {} \;
$FIND $BACKUPDIR -type f -name "*.xml.zip" -mtime +$BACKUPDAYS -exec rm {} \;



I also have a modified version as well (it requires rar executable as well):

Code: [Select]
#!/bin/ash
BACKUPDIR="/volume1/BACKUPNEW/pfsensebak/hansbuhlin/daily"
USERNAME="<removed>"
PASSWORD="<removed>"
PORT="443"
SITES="x.x.x.x"
RAR="/volume1/web_backend/tools/rar"
FIND="/usr/bin/find"
RMFILE="/bin/rm"
WGET="/usr/bin/wget"
BACKUPDAYS="30"
RMFILE="/bin/rm"
cd /volume1/web_backend/tools
/volume1/web_backend/tools/rar
for site in $SITES

do
       $WGET -qO- --keep-session-cookies --save-cookies /tmp/cookies.txt --no-check-certificate --timeout=10 https://$site:$PORT/diag_backup.php | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/csrf.txt
       $WGET -qO- --keep-session-cookies --load-cookies /tmp/cookies.txt --save-cookies /tmp/cookies.txt --no-check-certificate --post-data "login=Login&usernamefld=$USERNAME&passwordfld=$PASSWORD&__csrf_magic=$(cat /tmp/csrf.txt)" --timeout=10 https://$site:$PORT/diag_backup.php  | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > /tmp/csrf2.txt
       if [ -e /tmp/csrf2.txt ]; then
       DATETMP="`date +%Y%m%d-%H%M%S`"
       FILENAME1="$BACKUPDIR/config-$site-$DATETMP-rarpasswordencrypted.xml"
       FILENAME2="$BACKUPDIR/config-$site-$DATETMP-withrrd-rarpasswordencrypted.xml"
                  $WGET --keep-session-cookies --load-cookies /tmp/cookies.txt --no-check-certificate --post-data "download=download&donotbackuprrd=1&__csrf_magic=$(head -n 1 /tmp/csrf2.txt)" --timeout=10 https://$site:$PORT/diag_backup.php -O $FILENAME1 > /dev/null 2>&1
                  $WGET --keep-session-cookies --load-cookies /tmp/cookies.txt --no-check-certificate --post-data "download=download&donotbackuprrd=0&__csrf_magic=$(head -n 1 /tmp/csrf2.txt)" --timeout=120 https://$site:$PORT/diag_backup.php -O $FILENAME2 > /dev/null 2>&1
                  rm -f /tmp/cookies.txt
                  rm -f /tmp/csrf.txt
                  rm -f /tmp/csrf2.txt
       $RAR a -ep -m5 -hp<replace with own password> $FILENAME1.rar $FILENAME1
       $RAR a -ep -m5 -hp<replace with own password> $FILENAME2.rar $FILENAME2
       $RMFILE $FILENAME1
       $RMFILE $FILENAME2
else
       echo "Failed to retrieve backup from $site"
fi
                                                                                                   
done
                                                                                                             
$FIND $BACKUPDIR -type f -name "*.xml.rar" -mtime +$BACKUPDAYS -exec rm {} \;

//Dan Lundqvist

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21387
  • Karma: +1432/-26
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #1 on: May 08, 2017, 12:31:11 pm »
On what version? 2.3.x? 2.4 snapshots?

Nothing changed on that page during the relevant time period that would alter the behavior of the script that I see.

It's always checked if donotbackuprrd is set vs unset, not the value. Using 0 would cause it to evaluate as if it were unset. Just omit the parameter entirely, rather than giving it any value.

Code: [Select]
if ($_POST['backuparea'] !== "rrddata" && !$_POST['donotbackuprrd']) {
That line has not changed since May of 2012.

Code: [Select]
$_POST['donotbackuprrd'] = "no";
var_dump(!$_POST['donotbackuprrd']);

Outputs "bool(false)"

Code: [Select]
$_POST['donotbackuprrd'] = 0;
var_dump(!$_POST['donotbackuprrd']);

Outputs "bool(true)"

The page is fine, it does not offer any guidance for backing up RRD. That could be added, but nothing changed in the behavior stated on the page.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline mrzaz

  • Full Member
  • ***
  • Posts: 163
  • Karma: +4/-0
  • Beta Errors Eradicate Roguecode or BEER for short.
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #2 on: May 08, 2017, 01:47:42 pm »
Hi Jim,

It is really weird.   The problem was seen on 2.3.3_1 at the latest and as soon as I changed the script from "no" to "0" it started working and downloading RRD as it should.
"yes" and "no" defined it always exclude RRD.

I just tried again on 2.3.4 but same issue...   with yes/no it downloads both XML config without RRD and if i change script back again to 1/0 it starts working, then one contains the RRD data.
I just made a tcpdump of working and non-working session if you want?  I can send it to you offlist as it contains real config data.

//Dan Lundqvist

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21387
  • Karma: +1432/-26
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #3 on: May 08, 2017, 01:51:23 pm »
The code has not changed in years. It behaves as I described 100% of the time. yes or no does not matter, the value itself is never checked, only the variable is tested, and the way it is tested.

If the variable is defined to a non-zero value, it will not backup RRD. If it is undefined or set to 0, it will backup RRD.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline mrzaz

  • Full Member
  • ***
  • Posts: 163
  • Karma: +4/-0
  • Beta Errors Eradicate Roguecode or BEER for short.
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #4 on: May 08, 2017, 01:55:15 pm »
I can not explain it other than it did stop working at a certain date.   If that was due to upgrade of the NAS or something else I am not sure
but something happened.

BUT, if this is a local issue only at my installation, then we leave it at that. But bare in mind if you hear anyone else with same issue.

//Danne

Offline Higgiefresh

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #5 on: July 28, 2017, 12:39:47 pm »
Thanks so much for bringing this up and posting it.  Setting this up today for the first time (on 2.3.4-RELEASE-p1), I had the same hard time trying to figure out why deleting the variable or setting the value to yes/no wouldn't work.  After some Googling I found this post, set it to 0 and now I'm getting my Config+RRD data in full.

Offline mrzaz

  • Full Member
  • ***
  • Posts: 163
  • Karma: +4/-0
  • Beta Errors Eradicate Roguecode or BEER for short.
    • View Profile
Re: Found a small error "Remote_Config_Backup" in Wiki.
« Reply #6 on: July 29, 2017, 09:28:03 am »
Thanks so much for bringing this up and posting it.  Setting this up today for the first time (on 2.3.4-RELEASE-p1), I had the same hard time trying to figure out why deleting the variable or setting the value to yes/no wouldn't work.  After some Googling I found this post, set it to 0 and now I'm getting my Config+RRD data in full.

This means that I am not alone with this issue and also that the solution I found works for more than one. 
Jim, did you see this?

Best regards
Dan Lundqvist
Stockholm, Sweden