Netgate SG-1000 microFirewall

Author Topic: dnsbl_error.log growth rate /size  (Read 370 times)

0 Members and 1 Guest are viewing this topic.

Offline planktonclapped

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
dnsbl_error.log growth rate /size
« on: November 09, 2017, 03:24:31 am »
Hi,

Came down this morning to find my 2.4 installation was wedged, after a reboot I saw that the root filesystem was full, the culprit seemed to be a 14G dnsbl_error.log. 

I've got the logging option in the DNSBL tab set to disabled but I'm getting, roughly, 200 entries per second written to the dnsbl_error.log - all similar to:
Code: [Select]
2017-11-09 09:18:37: (configfile-glue.c.694) === start of condition block ===
2017-11-09 09:18:37: (configfile-glue.c.350) go parent global/SERVERsocket==0.0.0.0:8443
2017-11-09 09:18:37: (configfile-glue.c.622) 2 (cached) result: true
2017-11-09 09:18:37: (configfile-glue.c.557) HTTP["host"] ( device-metrics-us.amazon.com ) compare to  .*
2017-11-09 09:18:37: (configfile-glue.c.615) 3 (uncached) result: true

Can someone point me in the right direction to understand what the apparent error is that is being logged so frequently or, failing that, control the log files growth.

Offline nickd

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: dnsbl_error.log growth rate /size
« Reply #1 on: January 20, 2018, 10:42:02 am »
Also having this exact same issue, the file filled up all 6GB available on my filesystem. Did you ever figure out what was causing it or how to fix it? Interesting that all of my log was all filled with entries related to "device-metrics-us.amazon.com" as well. Sample of beginning of my log file:

Code: [Select]
2017-11-02 23:01:44: (log.c.217) server started
2017-11-02 23:07:46: (configfi202018-01-06 20:33:39: (configfile-glue.c.677) === start of condition block ===
2018-01-06 20:33:39: (configfile-glue.c.385) 3 global/HTTPhost=~.* not available yet
2018-01-06 20:33:39: (configfile-glue.c.589) 1 (uncached) result: unset
2018-01-06 20:33:39: (configfile-glue.c.677) === start of condition block ===
2018-01-06 20:33:39: (configfile-glue.c.531) SERVER["socket"] ( 0.0.0.0:8443 ) compare to  0.0.0.0:8443
2018-01-06 20:33:39: (configfile-glue.c.589) 2 (uncached) result: true
2018-01-06 20:33:39: (configfile-glue.c.677) === start of condition block ===
2018-01-06 20:33:39: (configfile-glue.c.342) go parent global/SERVERsocket==0.0.0.0:8443
2018-01-06 20:33:39: (configfile-glue.c.596) 2 (cached) res2018-01-07 17:11:33: (2018-01-07 23:33:32: (configfile-glue.c.677) === start of condition block ===
2018-01-07 23:32018-01-08 01:14:13: (configfile-glue.c.677) === start of condition block ===
2018-01-08 01:14:13: (configfile-glue.c.531) HTTP["host"] ( device-metrics-us.amazon.com ) compare to  .*
2018-01-08 01:14:13: (configfile-glue.c.589) 1 (uncached) result: true
2018-01-08 01:14:13: (configfile-glue.c.677) === start of condition block ===
2018-01-08 01:14:13: (configfile-glue.c.596) 2 (cached) result: true
2018-01-08 01:14:13: (configfile-glue.c.677) === start of condition block ===
2018-01-08 01:14:13: (configfile-glue.c.342) go parent global/SERVERsocket==0.0.0.0:8443
2018-01-08 01:14:13: (configfile-glue.c.596) 2 (cached) result: true
2018-01-08 01:14:13: (configfile-glue.c.531) HTTP["host"] ( device-metrics-us.amazon.com ) compare to  .*
2018-01-08 01:14:13: (configfile-glue.c.589) 3 (uncached) result: true
2018-01-08 03:05:18: (configfile-glue.c.677) === start of condition block ===
2018-01-08 03:05:18: (configfile-glue.c.385) 3 global/HTTPhost=~.* not available yet
2018-01-08 03:05:18: (configfile-glue.c.589) 1 (uncached) result: unset
2018-01-08 03:05:18: (configfile-glue.c.677) === start of condition block ===
2018-01-08 03:05:18: (configfile-glue.c.531) SERVER["socket"] ( 0.0.0.0:8443 ) compare to  0.0.0.0:8443
2018-01-08 03:05:18: (configfile-glue.c.589) 2 (uncached) result: true
2018-01-08 03:05:18: (configfile-glue.c.677) === start of condition block ===
2018-01-08 03:05:18: (configfile-glue.c.342) go parent global/SERVERsocket==0.0.0.0:8443
2018-01-08 03:05:18: (configfile-glue.c.596) 2 (cached) result: true
2018-01-08 03:05:18: (configfile-glue.c.385) 3 global/SERVERsocket==0.0.0.0:8443/HTTPhost=~.* not available yet
2018-01-08 03:05:18: (configfile-glue.c.589) 3 (uncached) result: unset
2018-01-08 03:05:19: (configfile-glue.c.677) === start of condition block ===
2018-01-08 03:05:19: (configfile-glue.c.531) HTTP["host"] ( device-metrics-us.amazon.com ) compare to  .*
« Last Edit: January 20, 2018, 10:50:44 am by nickd »

Offline RonpfS

  • Hero Member
  • *****
  • Posts: 717
  • Karma: +96/-2
    • View Profile
Re: dnsbl_error.log growth rate /size
« Reply #2 on: January 20, 2018, 11:39:18 am »
In  Firewall / pfBlockerNG / Log Browser tab you can delete the file to free disk space. You could also download it to a local drive if you want to keep it.

In Firewall / pfBlockerNG / General tab, there is a setting for log file size.
2.3.5-RELEASE-p1 (amd64)
Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
pfBlockerNG 2.1.2_2/Dev, suricata 4.0.3_1

Offline ghorsepower

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: dnsbl_error.log growth rate /size
« Reply #3 on: January 22, 2018, 08:17:48 am »
I am having identical problems. It seems like the dnsbl_error.log keeps trying to stay under the file size limit imposed for minutes to hours then something fails and the log file stops clearing just keeps growing and growing until you run out of disk space.

If you just delete the log file with rm -rf from shell access it still grows and eats disk space because the file is locked apparently. So i have been rebooting every day to clear it and regain diskspace so my firewall wont crash.

I figured out today that if you go to Firewall/pfBlockerNG/DNSBL and uncheck "Enable DNSBL", click save. Delete /var/log/pfblockerng/dnsbl_error.log, it actually deletes and regains lost disk space, avoiding a reboot. Then go back to settings and re-enable DNSBL it at least saves a reboot daily? Hope they fix this bug soon because its getting tedious to clean this by hand each day.

The culprit seems to be "HTTP["host"] ( device-metrics-us.amazon.com ) compare to  .*"

I am not sure what is causing this, I do have several amazon devices, kindles, fire-tv's etc... They are apparently causing DNSBL to vomit at the rate of 200 lines per second like the gentleman said above? Eventually overloading the clearing function of log management filling the disk.

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2608
  • Karma: +822/-5
    • View Profile
    • Click for Support
Re: dnsbl_error.log growth rate /size
« Reply #4 on: January 23, 2018, 05:35:54 pm »
The next release will have a new function to process this log... Just got bogged down since getting back from the holidays... So still working on a couple loose ends...  Thanks!
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline BBcan177

  • Moderator
  • Hero Member
  • *****
  • Posts: 2608
  • Karma: +822/-5
    • View Profile
    • Click for Support
Re: dnsbl_error.log growth rate /size
« Reply #5 on: January 23, 2018, 05:39:56 pm »
You could add these domains to Unbound as a Host override and set them to resolve to 0.0.0.0

Which will bypass DNSBL completely...
"Experience is something you don't get until just after you need it."

 | http://pfblockerng.com | Twitter @BBcan177  | #pfBlockerNG |

Offline ghorsepower

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: dnsbl_error.log growth rate /size
« Reply #6 on: January 24, 2018, 10:46:36 am »
The next release will have a new function to process this log... Just got bogged down since getting back from the holidays... So still working on a couple loose ends...  Thanks!

Awesome, thanks for the work and looking forward to the next release.

Offline Valeriy

  • Jr. Member
  • **
  • Posts: 52
  • Karma: +7/-0
    • View Profile
Re: dnsbl_error.log growth rate /size
« Reply #7 on: February 19, 2018, 07:14:06 pm »
I can confirm that issue exist, 43GB log file after 8 hours ))