next »

[johnpoz]

I don't show any entries in bogonsv6 table under diag?

I have tried toggling the setting, just they are not listed?

gitsync as of this morning, and reboot after gitsync

2.1-DEVELOPMENT (i386)
built on Fri Oct 21 12:51:56 EDT 2011
FreeBSD 8.1-RELEASE-p6

[databeestje]

I will need to verify. I do see we ship with a empty table but the file it fetches should hold some 50k networks.

[Cino]

i manually adding it a while ago and it hit a pf-filter limit. Can't remember what it was but its the same limit that IPBlocker and pfBlocker has when large list.

[databeestje]

It could be that the import is going wrong. Is there contents i. /Etc/bogonsv6 ?

[johnpoz]

-rw-r--r--   1 root  wheel          0 Mar  6  2011 bogonsv6

File is empty.

[databeestje]

Arr. That is suspect. Since my install does have a filled file you need to check if your dns works.

There is a script in /etc rc.update_bogons.sh that normally runs. Does that work?

[johnpoz]

I can assure you my dns works

ok I ran that script by hand, it first picked a random number of 53000 something -- that is seconds? So it was going to sleep for 14 hours?  WTF

Nov 24 13:03:23    root: rc.update_bogons.sh is sleeping for 53124

So I killed it and ran it by hand with 10 seconds  And it clearly updated the files.

[2.1-DEVELOPMENT][root@pfsense.local.lan]/etc(18): ls -la bogons*
-rw-r--r--  1 root  wheel     132 Nov 24 13:10 bogons
-rw-r--r--  1 root  wheel  761384 Nov 24 13:10 bogonsv6

Nov 24 13:10:11    root: rc.update_bogons.sh is ending the update cycle.
Nov 24 13:10:11    root: Bogons files downloaded: 49475 addresses added.
Nov 24 13:10:10    root: Bogons file downloaded: no changes.
Nov 24 13:10:08    root: rc.update_bogons.sh is beginning the update cycle.
Nov 24 13:10:08    root: rc.update_bogons.sh is starting up.

So something wrong with cron?  Why has that script not kicked off?  After I kicked it off by hand, the tables loads bogonsv6 -- and yeah its huge and does take a while to load the table

[johnpoz]

ok did an update last night to
2.1-DEVELOPMENT (i386)
built on Fri Nov 25 17:45:38 EST 2011
FreeBSD 8.1-RELEASE-p6

and now bogonsv6 is empty again?
No entries exist in this table.

looked at /etc
-rw-r--r--  1 root  wheel  132 Apr 28  2011 bogons
-rw-r--r--  1 root  wheel    0 Mar  6  2011 bogonsv6

So is there something wrong in the cron that should update these?


So I manually ran script rc.update_bogons.sh 10
Nov 30 14:50:29    admin: Bogons files downloaded: 49621 addresses added.

[2.1-DEVELOPMENT][admin@pfsense.local.lan]/etc(19): ls -la bo*
-rw-r--r--  1 root  wheel     132 Nov 30 14:50 bogons
-rw-r--r--  1 root  wheel  763666 Nov 30 14:50 bogonsv6


So what is suppose to update these, do you have to manually run the script - should something kick it off after an update?  On some sort of schedule?

[iamzam]

I am running the IPv6 2.1-DEVELOPMENT (amd64) branch, and I see tons of these in the system log:

Jan  3 09:03:07 pfs root: rc.update_bogons.sh is beginning the update cycle.
Jan  3 09:03:09 pfs root: Could not download http://files.pfsense.org/mirrors/bogon-bn-nonagg.txt.md5 (md5 mismatch)
Jan  3 09:03:09 pfs root: rc.update_bogons.sh is starting up.
Jan  3 09:03:09 pfs root: rc.update_bogons.sh is sleeping for 35036
Jan  3 09:03:09 pfs root: Could not download http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5 (md5 mismatch)
Jan  3 09:03:09 pfs root: rc.update_bogons.sh is ending the update cycle.
Jan  3 09:03:09 pfs root: rc.update_bogons.sh is starting up.
Jan  3 09:03:09 pfs root: rc.update_bogons.sh is sleeping for 37978
Jan  3 09:03:52 pfs root: rc.update_bogons.sh is beginning the update cycle.



I logged in via ssh and there are 64 "sh /etc/rc.update_bogons.sh" processes running.

I could kill all the processes and manually download the file and put it in place but if the md5 sums don't match there is possibly a bigger problem than that?

[jimp]

Looks fine from here:

Code:

[jimp@files mirrors]$ md5 bogon-bn-nonagg.txt
MD5 (bogon-bn-nonagg.txt) = 82bf1e457c3cbb64021067e2a2230051
[jimp@files mirrors]$ cat bogon-bn-nonagg.txt.md5
MD5 (bogon-bn-nonagg.txt) = 82bf1e457c3cbb64021067e2a2230051
[jimp@files mirrors]$ md5 fullbogons-ipv6.txt
MD5 (fullbogons-ipv6.txt) = 0c56fda817b9933477c576a36da0e5b6
[jimp@files mirrors]$ cat fullbogons-ipv6.txt.md5
MD5 (fullbogons-ipv6.txt) = 0c56fda817b9933477c576a36da0e5b6

[iamzam]

I'm not sure what was happening with the md5 sums not matching.  I killed all the sleeping processes (and all the sleep processes too) and i still couldn't run it manually.  I finally downloaded the http://files.pfsense.org/mirrors/fullbogons-ipv6.txt file (and removed the first and last comment lines) and put it into place at /etc/bogonsv6 but the web interface still showed the bogonsv6 table empty. 

I then went into the IPv4 bogons table and clicked the Download the latest bogon data button.  After that the bogonsv6 table was populated.

I do notice one thing with the rc.update_bogons.sh script:

in /etc/rc.update_bogons.sh there is a problem with the last part:

BOGON_MD5=`/usr/bin/fetch -q -o - "http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5" | awk '{ print $4 }'`
ON_DISK_MD5=`md5 /tmp/bogonsv6 | awk '{ print $4 }'`
if [ "$BOGON_MD5" = "$ON_DISK_MD5" ]; then
   egrep -v "^#" /tmp/bogonsv6 > /etc/bogonsv6
   /etc/rc.conf_mount_ro
   RESULT=`/sbin/pfctl -t bogonsv6 -T replace -f /etc/bogonsv6 2>&1`
   rm /tmp/bogons
   echo "Bogons files downloaded:  $RESULT" | logger
else
   echo "Could not download http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5 (md5 mismatch)" | logger
   # Relaunch and sleep
   sh /etc/rc.update_bogons.sh &    
fi


It writes the downloaded file (minus the '^#'  lines) to /tmp/bogonsv6 but after replacing /etc/bogonsv6 it deletes /tmp/bogons instead of /tmp/bogonsv6

It is possible, i suppose, that if the script was run multiple times simultaneously the IPv6 part would delete the /tmp/bogons file while the other script was attempting to use it to update /etc/bogons

[jimp]

I fixed that line in the script just now.

Not sure how you could have had so many of them sitting there unfinished though, that is a bit perplexing.

[iamzam]

Thanks jimp for fixing that!

It is strange since my crontab has the rc.update_bogons.sh running once a day,

...although each time the rc.update_bogons.sh script is run, it has the initial sleep plus each section has an additional relaunch and sleep in it for a total of 4 relaunch and 5 sleeps if it has major problems...
Such as if the WAN interface is down, or some other problem such as md5 (weird i know).

maybe an exit 1 should be called after the first relaunch so that it doesn't relaunch up to 4 times/script and start a relaunch cascade!