Netgate SG-1000 microFirewall

Author Topic: Unbound stopped and won't start  (Read 1646 times)

0 Members and 1 Guest are viewing this topic.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Unbound stopped and won't start
« on: October 10, 2015, 10:47:31 am »
Well - unbound quit today.  Pfsense is fine.  Nothing changed on the network. 
It has been unchanged for months and today, the service just crashed and no matter what it won't start and stay started.

Switched back to DNS forwarder.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +896/-7
    • View Profile
    • Chris Buechler
Re: Unbound stopped and won't start
« Reply #1 on: October 15, 2015, 06:41:48 pm »
Split this to its own topic since it had nothing to do with the thread it was posted in.

What comes up in your resolver log when it tries to start?

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #2 on: October 16, 2015, 07:27:26 am »
OK - I reverted the setup....   Here is what it says:

Oct 16 08:24:01    unbound: [85316:0] fatal error: failed to setup modules
Oct 16 08:24:01    unbound: [85316:0] error: module init for module validator failed
Oct 16 08:24:01    unbound: [85316:0] error: validator: could not apply configuration settings.
Oct 16 08:24:01    unbound: [85316:0] error: validator: error in trustanchors config
Oct 16 08:24:01    unbound: [85316:0] error: error reading auto-trust-anchor-file: /var/unbound/root.key
Oct 16 08:24:01    unbound: [85316:0] error: failed to read /root.key
Oct 16 08:24:01    unbound: [85316:0] error: failed to load trust anchor from /root.key at line 1, skipping
Oct 16 08:24:01    
Oct 16 08:24:01    unbound: [85316:0] notice: init module 0: validator
Oct 16 08:23:54    unbound: [58658:0] fatal error: failed to setup modules
Oct 16 08:23:54    unbound: [58658:0] error: module init for module validator failed
Oct 16 08:23:54    unbound: [58658:0] error: validator: could not apply configuration settings.
Oct 16 08:23:54    unbound: [58658:0] error: validator: error in trustanchors config
Oct 16 08:23:54    unbound: [58658:0] error: error reading auto-trust-anchor-file: /var/unbound/root.key
Oct 16 08:23:54    unbound: [58658:0] error: failed to read /root.key
Oct 16 08:23:54    unbound: [58658:0] error: failed to load trust anchor from /root.key at line 1, skipping
Oct 16 08:23:54    
Oct 16 08:23:54    unbound: [58658:0] notice: init module 0: validator

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #3 on: October 16, 2015, 08:06:33 am »
errrrr....   Found the answer here.

https://forum.pfsense.org/index.php?topic=87357.0

However, the idea that anything was corrupted by an upgrade seems unlikely since I didn't do any upgrades recently.

It simply broke without having been touched.  No Idea why.

I was able to fix it but still would feel better if I knew why it broke to begin with.
« Last Edit: October 16, 2015, 08:13:09 am by kejianshi »

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +896/-7
    • View Profile
    • Chris Buechler
Re: Unbound stopped and won't start
« Reply #4 on: October 21, 2015, 12:40:58 am »
You make note of the contents of root.key before deleting it?

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #5 on: October 21, 2015, 05:04:00 am »
I'm sorry - I didn't.

The nearest reason I can guess this may have happened is a write to the file interrupted by power flicker/outage.

The ups is currently needing a battery swap.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +896/-7
    • View Profile
    • Chris Buechler
Re: Unbound stopped and won't start
« Reply #6 on: October 21, 2015, 03:31:42 pm »
Was hoping to get a lead on the root cause there. Seems it's happened to roughly a half dozen people, but none have reported what the contents of root.key were before deleting it.

If you happen to see it again (seems unlikely), or anyone else that happens upon this thread in the future seeing it, please note the contents of the file. Diag>Command, run:
Code: [Select]
cat /var/unbound/root.key
Or download /var/unbound/root.key from same page or via scp. The contents should be text, so cat should suffice.

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +962/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Unbound stopped and won't start
« Reply #7 on: October 21, 2015, 03:40:27 pm »
but none have reported what the contents of root.key were before deleting it.

You mean like this one? https://forum.pfsense.org/index.php?topic=87357.msg479617#msg479617 - there's some inetd nonsense in there.
Do NOT PM for help!

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #8 on: October 21, 2015, 03:50:33 pm »
CMB - I have a few of these running here and there so if it happens again I will take a look at whats inside the file before I send it to bit heaven.

doktornotor - Yep - Thats exactly the errors it was throwing out.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +896/-7
    • View Profile
    • Chris Buechler
Re: Unbound stopped and won't start
« Reply #9 on: October 21, 2015, 08:26:54 pm »
Thanks, I looked through all those threads and missed the contents in that one. Looks like it's ending up with contents of other files in /var/ which would indicate the file wasn't fsynced by unbound after being written out. unbound-anchor also fails to create/update root.key if has invalid contents.

Should be fixed.
https://redmine.pfsense.org/issues/5334

Also reporting upstream to unbound as it should be doing that fsync and doesn't appear to be.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #10 on: October 21, 2015, 11:32:50 pm »
That is cool.  I like fixed things (-:

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +896/-7
    • View Profile
    • Chris Buechler
Re: Unbound stopped and won't start
« Reply #11 on: October 23, 2015, 04:12:25 pm »
Unbound fixed the missing fsync for a future release.
https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=712

The fsync I added should fix it in the mean time.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4995
  • Karma: +199/-43
  • Debugging...
    • View Profile
Re: Unbound stopped and won't start
« Reply #12 on: October 23, 2015, 05:26:32 pm »
Even better...

So I guess my bad batteries were less of a curse than I thought.

Now I need to replace them...   From 8k miles away...

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +962/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Unbound stopped and won't start
« Reply #13 on: October 24, 2015, 02:01:17 am »
Should be fixed.
https://redmine.pfsense.org/issues/5334

Cannot reproduce the original issue (ZFS on the test rigs doesn't seem to suffer from any of similar "features") but intentionally screwing the anchors file gets recovered just fine now...
Do NOT PM for help!