pfSense Forum

pfSense English Support => General Questions => Topic started by: Trel on January 26, 2015, 10:07:48 am

Title: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 26, 2015, 10:07:48 am
This has now happened three times

The symptoms I can see are

1. HTTP Webpages load blank
2. HTTPs webpages give a security error
3. Accessing pages by IP works
4. Any IP based connection works
5. Tracert appears valid

When this happens, if
1. I release and renew the IP for the WAN it works again
2. If I reboot PFSense, it works again

Additionally, while this is occuring
1. I CAN access the firewall's GUI internally (correct behavior)
1. I CAN access the firewall's GUI externally (correct behavior)

This started happening since I upgraded to 2.2 on Saturday morning.
No new rules have been created, no new firewall logs are showing up when this happens, nothing not usual in any of the log tabs.

This is a physical box with PFSense installed directly.
I just rebooted it remotely, to get everything back up and working.

Does anyone know what's going on here, or where I can look for more info?

EDIT: I should add, that the packages I have installed are
1. arping
2. Cron
3. File Manager
4. Notes
5. OpenVPN Client Export Utility
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 26, 2015, 04:37:24 pm
What specifically is the cert error you get on HTTPS sites in that circumstance? Short of some hacking to try to transparently proxy HTTPS, which wouldn't happen with that list of packages you have there, there wouldn't be anything within the firewall itself that'd cause a cert error.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 26, 2015, 06:18:08 pm
What specifically is the cert error you get on HTTPS sites in that circumstance? Short of some hacking to try to transparently proxy HTTPS, which wouldn't happen with that list of packages you have there, there wouldn't be anything within the firewall itself that'd cause a cert error.

I can get that information if this happens again.
I had only had the blank page one when I was present, but I was told about a site giving the invalid cert, so I told them to try another site to see if it was their machine's clock being wrong or similar, and that's how I found out the blank page thing was happening again.

At that point I remotely rebooted and it hasn't happened again yet.

If I had to guess, considering the webpages didn't report an error, just loaded blank, I'd guess that it was the same for the cert and as such, they were invalid.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 26, 2015, 08:22:11 pm
It's happening again.  I can't get the cert error to occur on my machine, but I am getting the blank pages.

Is there any debug steps I can take before I reboot?

EDIT: I can reproduce the cert issue, any site affected shows up as a self signed cert from and to "lolcat"
EDIT2: It suddenly resolved itself and the pages began loading again and certs were valid.

I think I'm likely going to reinstall 2.1.5 if I can't find any reason for this.  It coincides with the upgrade and the cert part of it, especially with an actual name showing up there unless that's some default name in Firefox that I don't know about when it can't load a cert, has me seriously worried.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 26, 2015, 11:47:59 pm
Packet capture of the problem traffic would be telling.

The whole getting a self-signed certificate with "lolcat" is a serious cause for concern, nothing on your firewall would be doing that, that suggests some kind of malware somewhere. Potentially on a client machine that's running an ARP poisoning tool and hijacking your connections on occasion.

Check your system log for any indications of "xx:xx:xx:xx:xx:xx is using my IP 192.168.1.1" (replace 1.1 with your LAN IP), that's one place to see if you have something trying to ARP poison.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 27, 2015, 10:06:46 am
Packet capture of the problem traffic would be telling.

The whole getting a self-signed certificate with "lolcat" is a serious cause for concern, nothing on your firewall would be doing that, that suggests some kind of malware somewhere. Potentially on a client machine that's running an ARP poisoning tool and hijacking your connections on occasion.

Check your system log for any indications of "xx:xx:xx:xx:xx:xx is using my IP 192.168.1.1" (replace 1.1 with your LAN IP), that's one place to see if you have something trying to ARP poison.

I'll try that when next it happens.
I don't see anything in the system log that has any mention of the IP.

Someone in IRC suggested that when it happens next, I run 'openssl s_client -showcerts -connect site:443' from an SSH to the firewall to verify it's not something upstream.
Considering that I don't necessarily have to reboot PFSense to resolve it, and releasing/renewing the WAN IP fixes it as well has me worried on that.

If it is something upstream, I'm extra worried because my connection is ISP--Modem--PFSense--Switch(s)--Computers
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 27, 2015, 01:46:15 pm
Running openssl from the firewall itself is a good idea, that'll at least bisect the issue. It seems unlikely it'd be on your ISP's side at least if it's the same ISP you're using to hit the forum, Comcast. It's possible, and given a change of WAN IP fixes it, that makes it seem more likely it's WAN-side, as nothing LAN-side would be impacted by that (unless it's just a coincidence). Some inept small ISP with a bunch of customers on the same broadcast domain and inadequate protection against things like ARP poisoning from customer to customer, I could see as being potentially more likely.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 27, 2015, 02:39:59 pm
Running openssl from the firewall itself is a good idea, that'll at least bisect the issue. It seems unlikely it'd be on your ISP's side at least if it's the same ISP you're using to hit the forum, Comcast. It's possible, and given a change of WAN IP fixes it, that makes it seem more likely it's WAN-side, as nothing LAN-side would be impacted by that (unless it's just a coincidence). Some inept small ISP with a bunch of customers on the same broadcast domain and inadequate protection against things like ARP poisoning from customer to customer, I could see as being potentially more likely.

I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.
I can't speculate further until I can run the openssl command from pfsense when it happens again.

And yes, currently Comcast, is the ISP in question.

I have one other theory as to the lolcat cert in that it's placeholder text in Firefox in the event that a cert loads completely blank, which would make sense as non SSL pages load blank while this is happening.  I would need to look at (or ask someone familiar with) Firefox's sourcecode to know for sure.

But that will be answered when I run openssl from pfsense at least if I get something other than the cert I got when I ran it baseline for comparison.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: virgiliomi on January 27, 2015, 04:21:16 pm
I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.
...
And yes, currently Comcast, is the ISP in question.
Comcast's DHCP leases are for a few days, which is why you don't get a new address with a release/renew. From looking at the DHCP client leases file on my box, it looks like they're about 4 days (renew time is half of the lease, and there's 2 days from renew to expire). IPv6 prefix leases are 7 days, from what I was told by a Comcast network engineer in another forum.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 27, 2015, 05:35:59 pm
I know. I meant the issue wasn't stopping from my WAN IP changing when I release/renew.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: saywhat on January 28, 2015, 07:35:37 am
We have had the exact same issue here in UK. We use BT as the ISP/

Same lolcat 3rd party self signed cert appearing for many sites, all DNS being redirected to 195.22.26.248 which shows as being a malicious IP in Portugal, used for lots of spammy domains.

Interestingly, we had Google DNS set on pfsense. When I changed this to OpenDNS the problem immediately went away, pings began to return correct IPs again etc.

I know Google DNS was hijacked before, so that is a possibility, but I would have thought an attack such as that would have hit the news on twitter by now.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 08:33:28 am
We have had the exact same issue here in UK. We use BT as the ISP/

Same lolcat 3rd party self signed cert appearing for many sites, all DNS being redirected to 195.22.26.248 which shows as being a malicious IP in Portugal, used for lots of spammy domains.

Interestingly, we had Google DNS set on pfsense. When I changed this to OpenDNS the problem immediately went away, pings began to return correct IPs again etc.

I know Google DNS was hijacked before, so that is a possibility, but I would have thought an attack such as that would have hit the news on twitter by now.

Now THAT actually gives me something to go on.

My DNS servers are
4.2.2.2
8.8.8.8
4.2.2.1
8.8.4.4

The 8.8.8.8 and 8.8.4.4 being Google DNS.
Are those the ones you have configured?

EDIT: and that could also explain how releasing and renewing the WAN connection fixes it.  If it loses connection with 4.2.2.2 and goes to Google's 8.8.8.8 and that's the problem, releasing/renewing might re-establish contact with 4.2.2.2.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: saywhat on January 28, 2015, 09:03:55 am
We were using 8.8.8.8 and 8.8.4.4

Changed them over to opendns and machines responded almost immediately

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 09:07:30 am
We were using 8.8.8.8 and 8.8.4.4

Changed them over to opendns and machines responded almost immediately

That looks like it has a good chance at being the cause then.
I'm going to remove those from my list and just keep the Level3 ones (4.2.2.1 and 4.2.2.2) and see if it ever happens again.

That might also explain why it didn't happen until right after the 2.2 upgrade if dnsmasq had a higher tolerance before falling over to the secondary DNS server than unbound does.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Pakken on January 28, 2015, 04:19:46 pm
God, and I thought I was the only one having this problem since I came up reading this thread.

Any news about that? Same invalid cert, same google dns.
Spent the last night trying to figure out what the he** could have happened.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 04:30:03 pm
God, and I thought I was the only one having this problem since I came up reading this thread.

Any news about that? Same invalid cert, same google dns.
Spent the last night trying to figure out what the he** could have happened.

Other than us three, I haven't found anyone who reported it anywhere but here.

But it's way too coincidental that three people got the same symptoms and had the same dns.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 04:41:51 pm
Me also - Thats main reason I turned off forwarder and turned on unbound on one of my systems.
The kids were reporting same exact issues as you...

Unbound with DNSSEC is technically slower than a forwarder but it seems faster in actual use and the kids report its solid.
I'm also using it over the VPN for my private use.

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on January 28, 2015, 04:43:57 pm
NSA testing some new (broken) toys? :D
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 04:46:02 pm
I will just say I like unbound and leave it at that...    (-;

Unbound + VPN = my tinfoil hat
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 05:56:47 pm
I just had this happen with level3 DNS (4.2.2.1 and 4.2.2.2) as the DNS servers.  I removed them leaving ONLY OpenDNS and it immediately started resolving correctly again.

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 06:04:31 pm
A lack of resolution could simply be a network error.  I was really only seeing issue with HTTPS sites.
Cert errors just smell like MITM to me. 
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 06:06:38 pm
A lack of resolution could simply be a network error.  I was really only seeing issue with HTTPS sites.
Cert errors just smell like MITM to me.

It's not a lack of resolution.  It IS resolving to a different IP.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 06:17:14 pm
I'm certain no one would use DNS resolution to effect a MITM attack.   (You are just paranoid)™
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 28, 2015, 06:33:19 pm
I'm certain no one would use DNS resolution to effect a MITM attack. 

That's actually pretty common, there's a variety of malware that will do just that to individual PCs, and sometimes to exploit routers and change their DNS servers so it impacts all LAN hosts. A variety of consumer-grade routers have been susceptible to such attacks.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 28, 2015, 06:35:21 pm
I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.

After the further details later in the thread, I think why that has an impact is because it's triggering a DNS cache flush in the DNS forwarder, so the poisoned replies are no longer there.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 06:48:57 pm
haha - Yeah.  I know.  My sarcasm wasn't obvious enough?  I'll try harder.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 28, 2015, 07:00:57 pm
haha - Yeah.  I know.  My sarcasm wasn't obvious enough?  I'll try harder.

Oh, the sarcasm font on here must be broken, sorry. :)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 07:08:19 pm
I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.

After the further details later in the thread, I think why that has an impact is because it's triggering a DNS cache flush in the DNS forwarder, so the poisoned replies are no longer there.

I actually just asked about this here: https://forum.pfsense.org/index.php?topic=87743.0

Is that a possible scenario, because if so I have a good idea of what might be doing it then.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 07:15:57 pm
My problem was originating outside the house between the ONT and the FIOS and or google DNS servers...
Its nothing inside the network that was causing it, but hopefully its mitigated now.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 07:18:02 pm
This problem is originating outside the house between the ONT and the FIOS and or google DNS servers...
Its nothing inside the network that was causing it, but hopefully its mitigated now.

If I can verify that pfsense itself is seeing the incorrect IPs for DNS lookups, there's definitely nothing internal that could be causing that at all?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 07:22:50 pm
There are just too many ways to mess with DNS especially if you can't trust the network between your machine and the servers. In the end, at best you can really only make sure that the guys playing games with your network aren't common criminals because you don't own the root servers.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 07:25:22 pm
There are just too many ways to mess with DNS especially if you can't trust the network between your machine and the servers. In the end, at best you can really only make sure that the guys playing games with your network aren't common criminals because you don't own the root servers.

But I'm asking about this specifically, I need to know if what's been happening could be due to an infected machine elsewhere on my network, or if it's definitely happening due to something from WAN and beyond.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 07:27:43 pm
I don't think one machine on the network could be the problem (unless that machine is pfsense its self), at least in my case, or going to unbound+DNSSEC would have made no difference.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on January 28, 2015, 07:39:30 pm
If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 07:43:16 pm
I don't think one machine on the network could be the problem (unless that machine is pfsense its self), at least in my case, or going to unbound+DNSSEC would have made no difference.

I just double checked and it looks like I never enabled DNSSEC when I changed to unbound.  Does that change your answer at all?
(needless to say, it's getting turned on now)

If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.

North East USA.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 07:43:30 pm
How about this - I will show you where the pfsense is sitting.  Hows that?  Check PM
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on January 28, 2015, 07:45:57 pm
If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.

North East USA.

Seems to be geographically diverse. At least two in the US in this thread, one in the UK.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 07:48:42 pm
The pfsense in question is in Maryland, for me.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 07:56:09 pm
By the way, I can confirm for sure that pfsense was seeing the bad DNS.

I have an alias for facebook.com
I saw this in the resolver log
Code: [Select]
filterdns: adding entry 195.22.26.248 to table Social_Test on host facebook.com
195.22.26.248 is the bad IP.  PFSense itself saw that when it did its update on the alias resolution.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 08:09:44 pm
https://www.virustotal.com/en/ip-address/195.22.26.248/information/

https://www.robtex.net/en/advisory/ip/195/22/26/248/

Seems like there is an associated IP block thats pretty much into everything bad.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on January 28, 2015, 08:12:08 pm
https://www.virustotal.com/en/ip-address/195.22.26.248/information/

https://www.robtex.net/en/advisory/ip/195/22/26/248/

Seems like there is an associated IP block thats pretty much into everything bad.

I don't doubt that, but that doesn't answer the question as to how when using google dns and level3 dns with unbound, that legitimate sites started resolving to this IP range, unless I'm missing something.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on January 28, 2015, 08:18:13 pm
No idea
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: agreenfield1 on February 05, 2015, 07:42:58 pm
Same thing happened to me this morning: https certs signed by lolcat, all dns inquiries not handled by pfsense directly give 195.22.26.248, and using the Google DNS and Level 3 dns servers.  I was able to resolve the issue for the time being by checking the 'Allow DNS server list to be overridden by DHCP/PPP on WAN' box, which presumably switched pfsense from using the compromised/poisoned DNS server to my ISPs DNS server.

I originally thought this issue was unrelated to pfsense, and posted the issue here:https://forum.pfsense.org/index.php?topic=88238.0 (https://forum.pfsense.org/index.php?topic=88238.0).  But after seeing this thread, it seems like pfsense 2.2 / DNS Resolver / Unbound may be a factor?

Configuration: PFSense 2.2, DNS Resolver, GoogleDNS and Level3 as primary and secondary DNS servers respectively.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 05, 2015, 08:40:03 pm
Nope - Because the same thing was happening to me using dnsmasq...

Actually switching to unbound + DNSSEC cured it.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 05, 2015, 09:40:57 pm
Quote
I originally thought this issue was unrelated to pfsense, and posted the issue here:https://forum.pfsense.org/index.php?topic=88238.0.  But after seeing this thread, it seems like pfsense 2.2 / DNS Resolver / Unbound may be a factor?

This has nothing to do with pfSense.  It has to do with you relying solely on google/level3 for all your DNS and someone is playing with it.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 05, 2015, 09:51:05 pm
Yep...   Now who could do that on a broad basis?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 05, 2015, 09:54:36 pm
It's intriguing.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 06, 2015, 08:39:24 am
Actually once I switched fully to unbound + DNSSEC only, I had a new issue.  At the same times, unbound would stop working.  The service would be running, but it wouldn't resolve anything until I restarted the service.

I finally found a common thread for that happening.  It almost always directly followed someone doing a lookup of

Code: [Select]
api-nyc01.exip.org
or
Code: [Select]
ns3.csof.net
The IP for those are in the 195.22.x range that was mentioned earlier.

Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

If someone is willing to look at that, because of how it lines up, it looks like trying to access/doing a lookup on those domains will either cause the blank pages and lolcat certs, or will cause unbound to stop resolving until the service is restarted. 

It's too coincidental to ignore in this case.

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 06, 2015, 08:44:48 am
It almost always directly followed someone doing a lookup of

Code: [Select]
api-nyc01.exip.org
or
Code: [Select]
ns3.csof.net
The IP for those are in the 195.22.x range that was mentioned earlier.
Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

Tried both, unbound still working. :) Apparently no NSA love here.  :'( ;D
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 06, 2015, 08:53:38 am
Now, I also had block rules in place for that range of IP.
I wonder if that could interact in some way.

Additionally, if you have Snort/Suricata installed, do you now have alerts mentioning the Anubis DNS Sinkhole?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 06, 2015, 08:59:20 am
if you have Snort/Suricata installed

Noooooooooooooo!!! (http://theairtacticalassaultgroup.com/forum/images/smilies/bolt.gif)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 06, 2015, 11:24:25 am
!!!
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: heper on February 07, 2015, 11:41:11 am
(unbound has been enabled for more then a month without issues ... until an hour ago)
i've suddenly been experiencing the blank pages + dns redirects to buydomains.com for lots of valid domains.

i tried to fix it by enabling dnssec ... didn't help

for now i've enabled "forwarding mode" on unbound ... this seems to fix the issue.

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 07, 2015, 12:15:23 pm
Unbound in resolver mode?  That makes no sense.  The deal is is that makes it pretty much impossible to affect everything.  They have to target specific name servers for specific domains (or .com, or . (root) etc..

What DNS servers are you handing out to your clients?  Running unbound means nothing if your clients are going to 8.8.8.8 / 8.8.4.4 for DNS.

Want an easy way to find out?  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged....
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 07, 2015, 12:56:05 pm
After you properly set up DNS and DNSSEC, you still have to clear DNS cache on each client and also have to make sure your clients are not infected with something or running some stupid browser add-on that hijacks things. 
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 2chemlud on February 07, 2015, 01:15:48 pm
  ....  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged....

Port 53 is not allowed at my network for more than a year. Doing fine with the DNS servers in the General setup and keep awful devices such as Buffallo Linkstations etc from phoning home...
Title: same thing just happened here (pfsense 2.2 + dns resolver active)
Post by: swix on February 09, 2015, 05:53:52 am
When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

Code: [Select]
swix@pc:~> host google.ch
google.ch has address 195.22.26.248                                                                                                                                                                   
google.ch mail is handled by 10 mx1.csof.net.                                                                                                                                                         
google.ch mail is handled by 10 mx2.csof.net.                         
                                                                                                                               
swix@pc:~> host aaaaaafadkfjdu93jifa.ch
aaaaaafadkfjdu93jifa.ch has address 195.22.26.248                                                                                                                                                 
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx1.csof.net.                                                                                                                                       
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx2.csof.net.

Unbound server is set as local resolver for a small LAN, with no forwarding to remote resolvers, so everything should be resolved locally.

Still investigating about how this could happen, and I will update this thread as soon as I find anything.
Kind regards.

PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Pakken on February 09, 2015, 06:32:10 am
So far happened only one time for me.
After enabling dnssec and disabling all the forwards to public dns servers it seems to be fixed.
In addition, I've created a floating rule to block every local subnet to that 195.22.0.0 range.

Will keep you updated.
To be honest the strange thing is that in a couple of years of pfsense pre-2.2 and dnsmasq this never happened.
The problem appeared straight after upgrading to 2.2 and dnsresolver even tho, once again, only happened one time so far to me.

Best regards
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 09, 2015, 07:00:27 am
When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
https://www.virustotal.com/en/ip-address/195.22.26.248/information/

If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.

PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.

Disabling DNSSEC most certainly does NOT help anything. Very broken idea.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 07:38:57 am
I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
https://www.virustotal.com/en/ip-address/195.22.26.248/information/
If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.
ng DNSSEC most certainly does NOT help anything. Very broken idea.

Thanks for the suggestion, yes, I will try to have a look on this, but the network device (VDSL Bridge Zyxel P-870M) is in bridge mode, so I have no way to connect directly to it (or only via a serial console, with a cable to be found yet).   Newest Firmware = 2009.

It just happened again a few minutes ago (3rd time today).

I was also trying to see if the root-servers file was tempered anyhow, but /etc/unbound/root.hints does not exist at all on the pfsense router.

Log extract when problem is happening, with many requests to "ns*.csof.net" servers where it shouldn't be the case  :

Code: [Select]
Feb  9 12:55:25 pf unbound: [39509:0] info: reply from <4.85.in-addr.arpa.> 195.186.196.180#53
Feb  9 12:55:25 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN       
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.csof.net. AAAA IN 
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns4.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <ubuntu.com.> 195.22.26.248#53          ########## wrong !
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER

TBC.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 09:22:12 am
I just want to clear up a few things

When any DNS Server can be used (not just unbound) and DNS Sec is set to off
-A DNS lookup from any computer to one of the domains cause EVERY subsequent lookup to resolve to "195.22.26.248"
(persists until unbound service is restarted)

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)

Thanks to a packet capture I was able to find which domains were being looked up, I then overrode the hosts and set the IP to 0.0.0.0 so they resolve but obviously can't get out

When only unbound can be used and DNS Sec is set to ON, port 53 is blocked except to pfsense, AND the hosts are overrode so Unbound doesn't make any query on those domains outward either
-All problems seem to cease


(Also, due to the packet capture, I can say the original request is coming from an unrooted Android device requesting port 80 on a few of those sites, based on the URL, it's requesting an API for determining the outward IP)

EDIT: here's the overrides I did to flat out prevent those names from being resolved.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 10:34:39 am
Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.    And last time, it stopped by itself after about 5 minutes.

When "broken", all webrequests are redirected to http://xsso.www.example.org (with the original domain name instead of example.org).
Code: [Select]
GET /domain/www.example.org HTTP/1.1
Host: sso.mlwr.io
User-Agent: Mozilla/5.0 (X11; Linux x86_64) (...)
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de,en-US;q=0.8,en;q=0.6
Cookie: anbsso=a5f4221ae2729d945150c83748e2ea12 (...)

Response: HTTP/1.1 302 Moved Temporarily
Server: nginx-perl/1.2.9.7
Date: Mon, 09 Feb 2015 14:29:31 GMT
Content-Type: text/html
Transfer-Encoding: chunked Connection: keep-alive
Set-Cookie: btst=b1b2035cffe818d92d7f6604a1318beb|myip|...
Location: http://xsso.www.example.org/a5f4221ae2729d945150c83748e2ea12

Just increased the logfiles size to try to see more next time it happens and added monitoring to get an alert directly.



And another view with wget, with and without https, with the same "lolcat" I already saw in another thread:

Code: [Select]
                                                                                                   
om@ompc:~> wget http://www.example.org
--2015-02-09 13:55:37--  http://www.example.org                                                                                                                                                     
Resolving www.example.org (www.example.org)... 195.22.26.248                                                                                                                                           
Connecting to www.example.org (www.example.org)|195.22.26.248|:80... connected.                                                                                                                       
HTTP request sent, awaiting response... 302 Moved Temporarily                                                                                                                                     
Location: http://sso.mlwr.io/domain/www.example.org [following]                                                                                                                                     
--2015-02-09 13:55:39--  http://sso.mlwr.io/domain/www.example.org
Resolving sso.mlwr.io (sso.mlwr.io)... 195.22.26.248
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Cookie coming from sso.mlwr.io attempted to set domain to example.org
Length: unspecified [text/html]
Saving to: ‘index.html.3’


om@ompc:~> wget https://www.example.org
--2015-02-09 13:55:43--  https://www.example.org/
Resolving www.example.org (www.example.org)... 195.22.26.248
Connecting to www.example.org (www.example.org)|195.22.26.248|:443... connected.
ERROR: cannot verify www.example.org's certificate, issued by ‘/CN=lolcat’:
  Self-signed certificate encountered.
    ERROR: certificate common name ‘lolcat’ doesn't match requested host name ‘www.example.org’.
To connect to www.example.org insecurely, use `--no-check-certificate'.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 10:39:47 am
You said you enabled DNSSEC, but question, what do you have in

System -> General -> DNS servers?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 10:41:55 am
PS:  installed packages on this router: arpwatch, bandwithd, cron, darkstat, mailreport, nrpe, rrd summary, openvpn client export.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 10:48:26 am
You said you enabled DNSSEC, but question, what do you have in
System -> General -> DNS servers?

Completely empty, so shoud unbound start with root servers directly.  I was wondering where was the hints file for unbound, but it seems to be directly in the binary file (strings unbound) :

Code: [Select]
A.ROOT-SERVERS.NET.
198.41.0.4
B.ROOT-SERVERS.NET.
192.228.79.201
C.ROOT-SERVERS.NET.
192.33.4.12
D.ROOT-SERVERS.NET.
199.7.91.13
E.ROOT-SERVERS.NET.
192.203.230.10
F.ROOT-SERVERS.NET.
192.5.5.241
G.ROOT-SERVERS.NET.
192.112.36.4
H.ROOT-SERVERS.NET.
128.63.2.53
I.ROOT-SERVERS.NET.
192.36.148.17
J.ROOT-SERVERS.NET.
192.58.128.30
K.ROOT-SERVERS.NET.
193.0.14.129
L.ROOT-SERVERS.NET.
199.7.83.42
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 10:50:50 am
Ok.  So what are the DNS servers configured on the client?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 10:56:23 am
Ok.  So what are the DNS servers configured on the client?
     

Set via DHCP, simply the router's ip address:
Code: [Select]
   option domain-name-servers 192.168.1.100;
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 11:25:54 am
And you've actually verified that's the case on the client?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 09, 2015, 11:44:18 am
Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.   

It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

- Block/redirect all DNS queries on LAN to pfSense
- Find and reimage infected crap.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 11:51:03 am
- Find and reimage infected crap.

I agree, but I would like to point out the way it affects pfsense/unbound is not a good thing at all.

A lookup on a completely isolated network segment made unbound start giving bad resolutions to ALL network segments when other DNS servers were permitted, and when they were blocked, unbound simply stopped replying.

That's not really the best outcome for a single computer looking up a bad domain.



Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 11:55:10 am
And you've actually verified that's the case on the client?

Yes, + also did tests with dig @192.168.1.100 and directly via the pfsense shell. "poisoned" ip in every case (after a few seconds after the beginning of a new occurence of the issue).


It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

Yep, I supposed that too, but sometimes there are collateral effets to such settings.


- Block/redirect all DNS queries on LAN to pfSense
- Find and reimage infected crap.

It will continue tomorrow, now it is calm again, as everybody left the office :)    But even if it is related to one malicious host on the LAN, it shouldn't be able to break the unbound resolver so easily...

Thanks again for all your feedbacks and until tomorrow!
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 12:14:10 pm
If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

You need to be looking at these queries from the root back and see where things go wrong.

Very intriguing.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 09, 2015, 12:22:04 pm
If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 12:27:43 pm
Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

I would put this on LAN:

pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log

Put that above your normal pass rule.  If everything is as you say, it should log nothing.

On pfSense 2.2 you should be able to set the dest to ! This Firewall (self).
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 09, 2015, 12:34:42 pm
Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

Pretty sure I could get these guys (https://www.turris.cz/en/) involved in investigating the issue here (they've also written the Knot DNS server (https://www.knot-dns.cz/) so I'm rather convinced they are familiar with DNS  :P) -- however that'd require either remote access or at least uncensored traffic captures. Not example.com -- totally useless.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 12:38:32 pm
If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

Using Comcast with a modem only (not a gateway in bridged mode).  Here's the block rule.
With these settings, if I try to look up the domain I get this scenario

Quote
When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)


I understand that an infected machine should not be on the network, but if a mere typical DNS lookup can cause this much havoc, then something is really wrong.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 09, 2015, 12:44:44 pm
Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.
Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

It wasn't obfuscated, it really looked like that... (also with other domains, juste replace example.com by anything)


I would put this on LAN:
pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log
Put that above your normal pass rule.  If everything is as you say, it should log nothing.

Ok, thanks, will setup this.


Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

I would be really happy to know the cause, it is really strange that Trel is having a similar problem with the very same target IP "195.22.26.248", especially from different countries/ISP's.   The only recent change to our infrastructure was upgrading to pfSense 2.2 at the beginning of January, otherwise nothing special.  But I'll setup some network monitoring tools later this week.

Best regards
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 12:46:19 pm
When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)

A search of redmine does not show that as an open issue.  Have you reported it?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 2chemlud on February 09, 2015, 12:51:42 pm

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)[


I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 12:53:30 pm
I'm not in a position to test this at the moment.  Tonight.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 12:55:59 pm

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)


I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

When you say interruptions, could those have been unbound not responding?
Someone did mention that one of the times I was unable to restart the service manually (as I was not available) it began working again after 45-50 minutes.

Either way though, as soon as I overrode the DNS for those sites, it's never happened again.

I'm not in a position to test this at the moment.  Tonight.

If you're going to test, try accessing and resolving

Code: [Select]
api-nyc01.exip.organd
Code: [Select]
ns3.csof.net
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 01:04:42 pm
Quote
A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything

So I can test it properly, to what domains is this referring?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 01:14:25 pm
If you're going to test, try accessing and resolving

Code: [Select]
api-nyc01.exip.organd
Code: [Select]
ns3.csof.net

What about it?

dig . ns

Pick a root server at random:

dig @e.root-servers.net ns3.csof.net

Get a list of gtld servers.  Pick one at random:

dig @e.gtld-servers.net ns3.csof.net

Pertinent info:

Code: [Select]
;; AUTHORITY SECTION:
csof.net. 172800 IN NS ns61.domaincontrol.com.
csof.net. 172800 IN NS ns62.domaincontrol.com.

;; ADDITIONAL SECTION:
ns61.domaincontrol.com. 172800 IN A 216.69.185.32
ns62.domaincontrol.com. 172800 IN A 208.109.255.32

Pick one of those:

dig @216.69.185.32 ns3.csof.net
Code: [Select]
;; QUESTION SECTION:
;ns3.csof.net. IN A

;; ANSWER SECTION:
ns3.csof.net. 600 IN A *** 195.22.26.199 ***

Their name servers either want that name to resolve to 195.22.26.199 or are giving bogus information or are otherwise hacked.  What, exactly, would you expect unbound to do to fix that?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 01:47:05 pm
Those domains aren't the problem.

The problem is what happens AFTER looking them up.

I can see there's some failure to communicate here.

Problem 1:
Without DNSSEC on and with other DNS servers allowed, a few minutes after looking up those domains, looking up google.com will return something in the 195.22.x range (persisting until I restarted unbound, or possibly 45 minutes to an hour)

At this point I switched to DNSSEC and blocked all outgoing DNS except to the firewall.

Problem 2:
With DNSSEC enabled and only unbound able to resolve, a few minutes after looking up those domains, lookup up google.com will return nothing the result will be blank as if the domain didn't exist (persisting until I restarted unbound, or possibly 45 minutes to an hour)

At this point, I put DNS overrides in for those domains setting them to 0.0.0.0 so they would not be able to be looked up at all.

At this point, the symptoms have stopped.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 01:51:23 pm
Ok I just let unbound look them up.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 02:28:34 pm
Start listening to these guys.  There's something strange and it looks like it's in unbound.

Code: [Select]
$ dig @192.168.223.1 www.google.com

; <<>> DiG 9.8.3-P1 <<>> @192.168.223.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 53699
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com. IN A

;; Query time: 81 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 12:23:46 2015
;; MSG SIZE  rcvd: 32

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 02:30:36 pm
After bouncing unbound:

Code: [Select]
$ dig @192.168.223.1 www.google.com

; <<>> DiG 9.8.3-P1 <<>> @192.168.223.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54480
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com. IN A

;; ANSWER SECTION:
www.google.com. 300 IN A 216.58.216.36

;; Query time: 1333 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 12:26:36 2015
;; MSG SIZE  rcvd: 48

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 2chemlud on February 09, 2015, 03:37:06 pm

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)


I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

When you say interruptions, could those have been unbound not responding?
Someone did mention that one of the times I was unable to restart the service manually (as I was not available) it began working again after 45-50 minutes.

....

My thread is this here:

https://forum.pfsense.org/index.php?topic=88272

:-)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 07:34:03 pm
Start listening to these guys.  There's something strange and it looks like it's in unbound.

I'm guessing it happened a few moments after doing a lookup on one of the domains I mentioned?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 07:51:18 pm
Yup.

Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames.  I'm doing it one more time at unbound log level 5.

ETA: Ironically, I had to kill unbound to post this because:

$ dig forum.pfsense.org

; <<>> DiG 9.8.3-P1 <<>> forum.pfsense.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30471
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;forum.pfsense.org.      IN   A

;; Query time: 1781 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 17:46:41 2015
;; MSG SIZE  rcvd: 35
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 09, 2015, 08:00:39 pm
Have you guys considered setting the advanced settings that prevent excessive replies and purge things when excessive replies and poisoning is possibly happening?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 08:05:48 pm
https://forum.pfsense.org/index.php?topic=88466.msg488411#msg488411
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: agreenfield1 on February 09, 2015, 08:11:01 pm
Yup.

Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames.  I'm doing it one more time at unbound log level 5.



This sounds like the manifestation of the issue with DNSSEC enabled.  At some point, you may want to try it again with DNSSEC disabled; you should then see all domains being resolved to a hostile IP, bad certs for https, etc. like reported in the beginning of this thread.  Not sure if this would help with the diagnostics.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 08:25:33 pm
Yup.

Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames.  I'm doing it one more time at unbound log level 5.



This sounds like the manifestation of the issue with DNSSEC enabled.  At some point, you may want to try it again with DNSSEC disabled; you should then see all domains being resolved to a hostile IP, bad certs for https, etc. like reported in the beginning of this thread.  Not sure if this would help with the diagnostics.

Yes, DNSSEC was the change I made that made it go from the bad domain resolutions to the failure to resolve at all.

I've had both issues and DNSSEC was the difference between which one I got.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 08:58:51 pm
Uncheck DNSSEC, Save, Apply....  There is most certainly something here....

Code: [Select]
$ dig ns3.csof.net

; <<>> DiG 9.8.3-P1 <<>> ns3.csof.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1411
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 3

;; QUESTION SECTION:
;ns3.csof.net. IN A

;; ANSWER SECTION:
ns3.csof.net. 600 IN A 195.22.26.199

;; AUTHORITY SECTION:
csof.net. 1792 IN NS ns62.domaincontrol.com.
csof.net. 1792 IN NS ns61.domaincontrol.com.
csof.net. 1792 IN NS ns4.csof.net.
csof.net. 1792 IN NS ns2.csof.net.
csof.net. 1792 IN NS ns1.csof.net.

;; ADDITIONAL SECTION:
ns4.csof.net. 1792 IN A 54.72.8.183
ns2.csof.net. 1793 IN A 212.6.183.201
ns1.csof.net. 1792 IN A 54.77.72.254

;; Query time: 29 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 18:48:25 2015
;; MSG SIZE  rcvd: 203

$ dig api-nyc01.exip.org

; <<>> DiG 9.8.3-P1 <<>> api-nyc01.exip.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14360
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;api-nyc01.exip.org. IN A

;; ANSWER SECTION:
api-nyc01.exip.org. 10 IN A 195.22.26.248

;; Query time: 206 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 18:48:35 2015
;; MSG SIZE  rcvd: 52

$ dig www.pfsense.org

; <<>> DiG 9.8.3-P1 <<>> www.pfsense.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51593
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;www.pfsense.org. IN A

;; ANSWER SECTION:
www.pfsense.org. 10 IN A 195.22.26.248

;; AUTHORITY SECTION:
org. 172779 IN NS ns1.csof.net.
org. 172779 IN NS ns2.csof.net.
org. 172779 IN NS ns3.csof.net.
org. 172779 IN NS ns4.csof.net.

;; Query time: 159 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 18:48:39 2015
;; MSG SIZE  rcvd: 129
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 09:04:07 pm
looks like unbound is allowing itself to get polluted by this nonsense:

Code: [Select]
gridbug:etc cjl$ dig @ns1.csof.net. com. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. com. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31834
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;com. IN NS

;; ANSWER SECTION:
com. 172800 IN NS ns1.csof.net.
com. 172800 IN NS ns2.csof.net.
com. 172800 IN NS ns3.csof.net.
com. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 163 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:58:46 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. . ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. . ns
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
gridbug:etc cjl$ dig @ns1.csof.net. net. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. net. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23620
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;net. IN NS

;; ANSWER SECTION:
net. 172800 IN NS ns1.csof.net.
net. 172800 IN NS ns2.csof.net.
net. 172800 IN NS ns3.csof.net.
net. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 161 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:19 2015
;; MSG SIZE  rcvd: 162

gridbug:etc cjl$ dig @ns1.csof.net. com. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. com. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57675
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;com. IN NS

;; ANSWER SECTION:
com. 172800 IN NS ns1.csof.net.
com. 172800 IN NS ns2.csof.net.
com. 172800 IN NS ns3.csof.net.
com. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 164 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:23 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. org. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. org. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20295
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;org. IN NS

;; ANSWER SECTION:
org. 172800 IN NS ns1.csof.net.
org. 172800 IN NS ns2.csof.net.
org. 172800 IN NS ns3.csof.net.
org. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 162 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:27 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. edu. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. edu. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28770
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;edu. IN NS

;; ANSWER SECTION:
edu. 172800 IN NS ns1.csof.net.
edu. 172800 IN NS ns2.csof.net.
edu. 172800 IN NS ns3.csof.net.
edu. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 161 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:31 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. gov. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. gov. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46880
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;gov. IN NS

;; ANSWER SECTION:
gov. 172800 IN NS ns1.csof.net.
gov. 172800 IN NS ns2.csof.net.
gov. 172800 IN NS ns3.csof.net.
gov. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 162 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:34 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. mil. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. mil. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21791
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;mil. IN NS

;; ANSWER SECTION:
mil. 172800 IN NS ns1.csof.net.
mil. 172800 IN NS ns2.csof.net.
mil. 172800 IN NS ns3.csof.net.
mil. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 160 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:40 2015
;; MSG SIZE  rcvd: 165

gridbug:etc cjl$ dig @ns1.csof.net. us. ns

; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. us. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5930
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;us. IN NS

;; ANSWER SECTION:
us. 172800 IN NS ns1.csof.net.
us. 172800 IN NS ns2.csof.net.
us. 172800 IN NS ns3.csof.net.
us. 172800 IN NS ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net. 100 IN A 54.77.72.254
ns2.csof.net. 100 IN A 212.6.183.201
ns3.csof.net. 100 IN A 195.22.26.199
ns4.csof.net. 100 IN A 54.72.8.183

;; Query time: 161 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Mon Feb  9 18:59:43 2015
;; MSG SIZE  rcvd: 164
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 09:14:27 pm
Back to dnsmasq for me until someone says something.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 09, 2015, 09:35:19 pm
I don't know which I should be.

Upset that unbound is doing this
Relieved that everything I was seeing can be reproduced
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Derelict on February 09, 2015, 09:50:22 pm
You should be thankful you found something that made it easily-reproducible.  No need to be mad.  Things happen - Look at BIND's history.  And you can always just run dnsmasq in the meantime even though that's the exact opposite advice you got at first.  :/
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: agreenfield1 on February 09, 2015, 10:23:49 pm
I'm really out of my league trying to diagnose this, but here is the output of 'unbound-control -c /var/unbound/unbound.conf lookup slashdot.org', when the issue is occuring with DNSSEC enabled:

Code: [Select]
unbound-control -c /var/unbound/unbound.conf lookup slashdot.org
The following name servers are used for lookup of slashdot.org.
;rrset 85785 4 0 7 0
org. 172185 IN NS ns1.csof.net.
org. 172185 IN NS ns2.csof.net.
org. 172185 IN NS ns3.csof.net.
org. 172185 IN NS ns4.csof.net.
;rrset 83879 2 1 11 4
org. 83879 IN DS 21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org. 83879 IN DS 21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0D90F01BA
org. 83879 IN RRSIG DS 8 1 86400 20150219170000 20150209160000 16665 . RxYpI0BzYpGTE/PjRQdR4SZaxlvXCja3SJyx10JagTfz20gnltl4ar94GOwp8bA/ktY/7JxMoJvzCTAtcsGaTGRv04yDHr7WaydMxZuPCP9YT9Ixc+fX9IAZlSfwLCkBQgiC0mVeRiq+LmbIJhI2grJbTtvy96O9mipAqkFR42g= ;{id = 16665}
;rrset 1185 1 0 3 0
ns4.csof.net. 1185 IN A 54.72.8.183
;rrset 197 1 0 8 0
ns3.csof.net. 197 IN A 195.22.26.199
;rrset 1185 1 0 8 0
ns2.csof.net. 1185 IN A 212.6.183.201
;rrset 1185 1 0 8 0
ns1.csof.net. 1185 IN A 54.77.72.254
Delegation with 4 names, of which 4 can be examined to query further addresses.
It provides 4 IP addresses.
54.77.72.254    NoDNSSEC rto 344 msec, ttl 286, ping 84 var 65 rtt 344, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
212.6.183.201    NoDNSSEC rto 365 msec, ttl 286, ping 69 var 74 rtt 365, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
195.22.26.199    rto 96256 msec, ttl 286, ping 0 var 94 rtt 376, tA 3, tAAAA 3, tother 3, EDNS 0 assumed.
54.72.8.183      NoDNSSEC rto 315 msec, ttl 286, ping 99 var 54 rtt 315, tA 0, tAAAA 0, tother 0, EDNS 0 probed.

And then after restarting unbound:

Code: [Select]
unbound-control -c /var/unbound/unbound.conf lookup slashdot.org
The following name servers are used for lookup of slashdot.org.
;rrset 86374 6 0 2 0
org. 86374 IN NS a0.org.afilias-nst.info.
org. 86374 IN NS a2.org.afilias-nst.info.
org. 86374 IN NS b0.org.afilias-nst.org.
org. 86374 IN NS b2.org.afilias-nst.org.
org. 86374 IN NS c0.org.afilias-nst.info.
org. 86374 IN NS d0.org.afilias-nst.org.
;rrset 86374 2 1 2 0
org. 86374 IN DS 21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org. 86374 IN DS 21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0D90F01BA
org. 86374 IN RRSIG DS 8 1 86400 20150219170000 20150209160000 16665 . RxYpI0BzYpGTE/PjRQdR4SZaxlvXCja3SJyx10JagTfz20gnltl4ar94GOwp8bA/ktY/7JxMoJvzCTAtcsGaTGRv04yDHr7WaydMxZuPCP9YT9Ixc+fX9IAZlSfwLCkBQgiC0mVeRiq+LmbIJhI2grJbTtvy96O9mipAqkFR42g= ;{id = 16665}
;rrset 86374 1 0 1 0
d0.org.afilias-nst.org. 172774 IN A 199.19.57.1
;rrset 86374 1 0 1 0
d0.org.afilias-nst.org. 172774 IN AAAA 2001:500:f::1
;rrset 86374 1 0 1 0
c0.org.afilias-nst.info. 172774 IN A 199.19.53.1
;rrset 86374 1 0 1 0
c0.org.afilias-nst.info. 172774 IN AAAA 2001:500:b::1
;rrset 86374 1 0 1 0
b2.org.afilias-nst.org. 172774 IN A 199.249.120.1
;rrset 86374 1 0 1 0
b2.org.afilias-nst.org. 172774 IN AAAA 2001:500:48::1
;rrset 86374 1 0 1 0
b0.org.afilias-nst.org. 172774 IN A 199.19.54.1
;rrset 86374 1 0 1 0
b0.org.afilias-nst.org. 172774 IN AAAA 2001:500:c::1
;rrset 86374 1 0 1 0
a2.org.afilias-nst.info. 172774 IN A 199.249.112.1
;rrset 86374 1 0 1 0
a2.org.afilias-nst.info. 172774 IN AAAA 2001:500:40::1
;rrset 86374 1 0 1 0
a0.org.afilias-nst.info. 172774 IN A 199.19.56.1
;rrset 86374 1 0 1 0
a0.org.afilias-nst.info. 172774 IN AAAA 2001:500:e::1
Delegation with 6 names, of which 0 can be examined to query further addresses.
It provides 12 IP addresses.
2001:500:e::1    not in infra cache.
199.19.56.1      not in infra cache.
2001:500:40::1  not in infra cache.
199.249.112.1    not in infra cache.
2001:500:c::1    not in infra cache.
199.19.54.1      not in infra cache.
2001:500:48::1  not in infra cache.
199.249.120.1    rto 356 msec, ttl 874, ping 8 var 87 rtt 356, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
2001:500:b::1    not in infra cache.
199.19.53.1      rto 482 msec, ttl 874, ping 22 var 115 rtt 482, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
2001:500:f::1    not in infra cache.
199.19.57.1      not in infra cache.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 09, 2015, 10:48:30 pm
I'll ask again...

In Services: DNS Resolver: Advanced

Have you tried checking:

Harden Glue

Harden DNSSEC data

Unwanted Reply Threshold (10 million)

etc...   ???
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on February 09, 2015, 11:00:48 pm
Have you tried checking:

Harden Glue

Harden DNSSEC data

These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 09, 2015, 11:06:13 pm
Yeah - I went on a voyage of discovery to find a combo that worked...
Less a matter of being smart.  More a matter of random experimentation.

Even reading and searching online yielded little results.

But yeah - Fixes alot.

I'm glad you guys are making it defaults.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 10, 2015, 02:44:44 am
Couple more of the hardened stuff here: https://forum.pfsense.org/index.php?topic=88466.msg488511#msg488511

- harden-referral-path is hardcoded to no in unbound.inc ATM  :(
- harden-below-nxdomain can be set via the advanced config

(created a feature request (https://redmine.pfsense.org/issues/4399) for the two above)

- there's also this use-caps-for-id thing (patch to expose in GUI in 4205 (https://redmine.pfsense.org/issues/4205) or stick use-caps-for-id: yes to advanced config.

Another thing that makes me wonder... dnsmasq AFAICT is not compiled with DNSSEC support at all on pfSense even though it's apparently supported (http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html) now. Hmmm. And of course, no matter what you are completely screwed with unsigned domains, and the DNS protocol is a piece of insecure crap.

A couple of suggestions for your own servers:
- sign your domains (ahem... why's pfsense.org not signed?!)
- if your registrar does not allow you to do so, switch registrar
- if the TLD you are using is not signed, choose a different one
- use DANE/TLSA (http://tools.ietf.org/html/rfc6698) for your critical servers at least.

Some useful tools:
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 10, 2015, 04:04:57 am
Update for our situation (only dns active, harden glue settings not (yet) updated).
The situation happened again a few minutes ago, and here it is how it looked in the resolver.log (first time "csof" is visible today)  :

Code: [Select]
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for 239.75.246.46.in-addr.arpa. PTR IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <75.246.46.in-addr.arpa.> 91.213.246.6#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.transitionalprotocol.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving f.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving g.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving k.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving l.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.transitionalprotocol.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving h.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving i.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving m.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving d.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: response for a.ns.portlane.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <portlane.net.> 80.67.0.6#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for b.ns.portlane.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <portlane.net.> 80.67.0.6#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for 125.10.113.188.in-addr.arpa. PTR IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <10.113.188.in-addr.arpa.> 80.240.240.2#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving 113.188.in-addr.arpa. DS IN
Feb 10 10:03:18 pf unbound: [24807:1] notice: sendto failed: Operation not permitted
Feb 10 10:03:18 pf unbound: [24807:1] notice: remote address is 2001:500:13::c7d4:35 port 53
Feb 10 10:03:18 pf unbound: [24807:1] info: error sending query to auth server 2001:500:13::c7d4:35 port 53
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.52.178.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.csof.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.csof.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving e.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving d.gtld-servers.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.52.178.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:0] info: response for 125.74.1.116.in-addr.arpa. PTR IN
Feb 10 10:03:18 pf unbound: [24807:0] info: reply from <1.116.in-addr.arpa.> 202.103.224.69#53
Feb 10 10:03:18 pf unbound: [24807:0] info: query response was NXDOMAIN ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.26.92.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.> 212.6.183.201#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.31.80.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.csof.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.43.172.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.> 212.6.183.201#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for 113.188.in-addr.arpa. DS IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <188.in-addr.arpa.> 199.212.0.53#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved not a delegation point
Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved no DS.
Feb 10 10:03:18 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. A IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.> 54.77.72.254#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.csof.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.> 192.26.92.30#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:18 pf unbound: [24807:1] info: response for 239.75.246.46.in-addr.arpa. PTR IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <239.75.246.46.in-addr.arpa.> 195.22.26.248#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:18 pf unbound: [24807:1] info: resolving 246.46.in-addr.arpa. DS IN
Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved not a delegation point
Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved no DS.
Feb 10 10:03:18 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE
Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.csof.net. AAAA IN
Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <csof.net.> 208.109.255.32#53
Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:18 pf unbound: [24807:0] info: response for 166.31.215.188.in-addr.arpa. PTR IN
Feb 10 10:03:18 pf unbound: [24807:0] info: reply from <31.215.188.in-addr.arpa.> 89.39.166.2#53
Feb 10 10:03:18 pf unbound: [24807:0] info: query response was THROWAWAY
Feb 10 10:03:19 pf unbound: [24807:1] info: response for ns1.csof.net. AAAA IN
Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <csof.net.> 208.109.255.32#53
Feb 10 10:03:19 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:19 pf unbound: [24807:1] info: resolving 205.112.194.173.in-addr.arpa. PTR IN
Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns2.google.com. AAAA IN
Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns4.google.com. AAAA IN
Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns1.google.com. AAAA IN
Feb 10 10:03:19 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. AAAA IN
Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.> 212.6.183.201#53
Feb 10 10:03:19 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:19 pf unbound: [24807:1] info: response for 205.112.194.173.in-addr.arpa. PTR IN
Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <194.173.in-addr.arpa.> 216.239.36.10#53
Feb 10 10:03:19 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving www.insign.ch. A IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving dns2.insign.ch. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving dns1.insign.ch. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: response for www.insign.ch. A IN
Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <insign.ch.> 46.175.9.120#53
Feb 10 10:03:20 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:20 pf unbound: [24807:1] info: NSEC3s for the referral proved no DS.
Feb 10 10:03:20 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving 64.94.172.95.in-addr.arpa. PTR IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-b.pnap.net. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-c.pnap.net. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-a.pnap.net. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving 185.112.194.173.in-addr.arpa. PTR IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns3.google.com. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns4.google.com. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns1.google.com. AAAA IN
Feb 10 10:03:20 pf unbound: [24807:1] info: response for 185.112.194.173.in-addr.arpa. PTR IN
Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <194.173.in-addr.arpa.> 216.239.32.10#53
Feb 10 10:03:20 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:20 pf unbound: [24807:1] info: response for 64.94.172.95.in-addr.arpa. PTR IN
Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <94.172.95.in-addr.arpa.> 64.95.61.4#53
Feb 10 10:03:41 pf unbound: [24807:0] info: query response was REFERRAL
Feb 10 10:03:41 pf unbound: [24807:0] info: resolving dns4.bigrock.in. AAAA IN
Feb 10 10:03:41 pf unbound: [24807:0] info: resolving dns2.bigrock.in. AAAA IN
Feb 10 10:03:41 pf unbound: [24807:0] info: resolving asia1.akam.net. AAAA IN
Feb 10 10:03:41 pf unbound: [24807:0] notice: sendto failed: Operation not permitted
Feb 10 10:03:41 pf unbound: [24807:0] notice: remote address is 2600:1401:2::43 port 53
Feb 10 10:03:41 pf unbound: [24807:0] info: error sending query to auth server 2600:1401:2::43 port 53
Feb 10 10:03:41 pf unbound: [24807:1] info: response for ns0.mirasystem.net. A IN
Feb 10 10:03:41 pf unbound: [24807:1] info: reply from <net.> 54.72.8.183#53
Feb 10 10:03:41 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:42 pf unbound: [24807:0] info: response for asia1.akam.net. AAAA IN
Feb 10 10:03:42 pf unbound: [24807:0] info: reply from <akam.net.> 184.85.248.67#53
Feb 10 10:03:42 pf unbound: [24807:0] info: query response was nodata ANSWER
Feb 10 10:03:42 pf unbound: [24807:0] info: response for dns2.bigrock.in. AAAA IN
Feb 10 10:03:42 pf unbound: [24807:0] info: reply from <bigrock.in.> 2.22.230.64#53
Feb 10 10:03:42 pf unbound: [24807:0] info: query response was nodata ANSWER
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.epn.ru. A IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <EPN.RU.> 195.22.26.248#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns0.epn.ru. AAAA IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <epn.ru.> 195.22.26.248#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns0.epn.ru. A IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <EPN.RU.> 195.22.26.248#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was ANSWER
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.epn.ru. AAAA IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <EPN.RU.> 195.22.26.248#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was nodata ANSWER
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.hn.cnc.cn. AAAA IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.> 210.52.207.2#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.hn.cnc.cn. A IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.> 210.52.207.2#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns2.hn.cnc.cn. A IN
Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.> 210.52.207.2#53
Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL
Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns2.hn.cnc.cn. AAAA IN

At the end, all requests get resolved as "195.22.26.248".
Trigger request couldn't yet be found (the log rule on port 53 for requests not going over the router returned nothing at this moment).

So when it starts, every single request then go over one of these ns1.csof.net name server, and get answered as if csof.net was a root server, but with wrong data.  For example:

Normal case when querying the pfsense resolver (when everything is ok) :

Code: [Select]
om@ompc:~> dig -t ns @192.168.1.100 ch.

; <<>> DiG 9.9.5-rpz2+rl.14038.05-P1 <<>> -t ns @192.168.1.100 ch.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62805
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 14

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ch.                            IN      NS

;; ANSWER SECTION:
ch.                     85292   IN      NS      c.nic.ch.
ch.                     85292   IN      NS      b.nic.ch.
ch.                     85292   IN      NS      e.nic.ch.
ch.                     85292   IN      NS      d.nic.ch.
ch.                     85292   IN      NS      a.nic.ch.
ch.                     85292   IN      NS      h.nic.ch.
ch.                     85292   IN      NS      f.nic.ch.

;; ADDITIONAL SECTION:
a.nic.ch.               171692  IN      A       130.59.1.80
a.nic.ch.               171692  IN      AAAA    2001:620::4
b.nic.ch.               171692  IN      A       130.59.211.10
b.nic.ch.               171692  IN      AAAA    2001:620::5
c.nic.ch.               171692  IN      A       147.28.0.39
c.nic.ch.               171692  IN      AAAA    2001:418:1::39
d.nic.ch.               171692  IN      A       200.160.0.5
d.nic.ch.               171692  IN      AAAA    2001:12ff:0:a20::5
e.nic.ch.               171692  IN      A       194.0.17.1
e.nic.ch.               171692  IN      AAAA    2001:678:3::1
f.nic.ch.               171692  IN      A       194.146.106.10
f.nic.ch.               171692  IN      AAAA    2001:67c:1010:2::53
h.nic.ch.               171692  IN      A       194.42.48.120

;; Query time: 0 msec
;; SERVER: 192.168.1.100#53(192.168.1.100)
;; WHEN: Tue Feb 10 10:55:52 CET 2015
;; MSG SIZE  rcvd: 427


When querying this ns1.csof.net server :

Code: [Select]
om@ompc:~> dig -t ns @ns1.csof.net ch.

; <<>> DiG 9.9.5-rpz2+rl.14038.05-P1 <<>> -t ns @ns1.csof.net ch.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45232
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;ch.                            IN      NS

;; ANSWER SECTION:
ch.                     172800  IN      NS      ns1.csof.net.
ch.                     172800  IN      NS      ns2.csof.net.
ch.                     172800  IN      NS      ns3.csof.net.
ch.                     172800  IN      NS      ns4.csof.net.

;; ADDITIONAL SECTION:
ns1.csof.net.           100     IN      A       54.77.72.254
ns2.csof.net.           100     IN      A       212.6.183.201
ns3.csof.net.           100     IN      A       195.22.26.199
ns4.csof.net.           100     IN      A       54.72.8.183

;; Query time: 54 msec
;; SERVER: 54.77.72.254#53(54.77.72.254)
;; WHEN: Tue Feb 10 10:45:26 CET 2015
;; MSG SIZE  rcvd: 164


I now increased the verbosity of unbound with "unbound-control -c /var/unbound/unbound.conf verbosity 4" and waiting for another hit...   If you have other suggestions in the mean time, please feel free.

Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 10, 2015, 04:12:53 am
Does that mean you have already made all the suggested changes?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 10, 2015, 04:19:13 am
Does that mean you have already made all the suggested changes?

Apparently not.

Quote
Update for our situation (only dns active, harden glue settings not (yet) updated).

People, by all means feel free to experiment if its your network. However, mind that if there are any unsuspecting users behind your router using your resolver, this is not the best idea around. Noone likes getting their boxes infected via malicious pages served via the subverted DNS, getting their banking info compromised etc.!
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 10, 2015, 04:25:26 am
Does that mean you have already made all the suggested changes?

(if the question is for me) : no, as we would first would like to find out what/who is triggering this phenomena.  But we'll change the setup then (today before the evening) and report again.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 10, 2015, 04:29:30 am
Yeah - Please do find out.  Its a mystery for me.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 10, 2015, 07:00:24 am
Activating following settings (as suggested previously in this thread) seems to solve the issue for now, at least for the last 3 hours :

Code: [Select]
harden-glue: yes
harden-dnssec-stripped: yes

In the mean time I have a 500 MB resolver.log in "verbose=5" mode to analyze.   


Title: [SOLVED] Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 10, 2015, 09:17:42 am
With the great support of Wouter Wijngaards from the Unbound-Dev-Team this issue can be considered as closed, at least on our side.
Activating "harden-glue: yes" and "harden-dnssec-stripped: yes" was the solution, or a patch can be applied to fix this situation without enabling these options.

Quote from: Wouter Wijngaards
Hi Martin, Olivier,

I have found the issue, it is in unbound.  You have turned off
harden-glue in config and this disabled the poison checks (that is its
purpose) and caused the poisoning.  The domain is not an attack domain
I think but a domain-reseller.

Turn on harden-glue: yes in unbound.conf

Also this patch may solve the problem (and you can continue to use
harden-glue: no).  Note that harden-glue no allows people (not this
specific attack) to poison nameserver glue records and then poison
your cache as well, also with the patch.


Index: iterator/iter_scrub.c
===================================================================
--- iterator/iter_scrub.c       (revision 3329)
+++ iterator/iter_scrub.c       (working copy)
@@ -680,7 +680,9 @@
                                 * (we dont want its glue that was approved
                                 * during the normalize action) */
                                del_addi = 1;
-                       } else if(!env->cfg->harden_glue) {
+                       } else if(!env->cfg->harden_glue && (
+                               rrset->type == LDNS_RR_TYPE_A ||
+                               rrset->type == LDNS_RR_TYPE_AAAA)) {
                                /* store in cache! Since it is relevant
                                 * (from normalize) it will be picked up
                                 * from the cache to be used later */


Why was harden-glue turned off?  And perhaps I should change
documentation or implementation of this misfeature?

Best regards,
   Wouter


More information from the Unbound-Users mailing list : https://unbound.nlnetlabs.nl/pipermail/unbound-users/2015-February/003768.html

Suggestion for pfSense 2.2.1 : apply this patch to unbound and/or activate the harden-options by default.  I'll check later if there is anything in redmine about this issue or will create a new one accordingly.

Thanks again for your fast and helpful answers on this forum & I hope it will remain stable this way !  Now I can get back to my IPSEC issues :-)
Kind regards.

PS: about the trigger :

Code: [Select]

> (...) could you see which host/device caused this issue ?

Yes, the offending query was for ns2.transitionalprotocol.net A.
The logs are missing lines (the logger seems to skip lines when
presented at a high rate).  That query was made to resolve another
query for 105.73.246.46.in-addr.arpa. PTR.
And this was requested by
Feb 10 11:48:15 pf unbound: [68609:0] debug: udp request from ip4
192.168.1.150 port 62403 (len 16)

The query for that reverse address seems innocent.
Title: Re: [SOLVED] Re: Periodic since 2.2 pages load blank, certs invalid
Post by: dgcom on February 10, 2015, 09:57:05 am
Quote from: Wouter Wijngaards
Why was harden-glue turned off?

Indeed, this is a good question, keeping in mind that unbound is enabled by default in all new 2.2 installations, I wonder how many unsuspecting people are being affected by this.
I feel that it warrants an official announcement by pfSense team with specific suggestions on how to remedy the issue...
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: Trel on February 10, 2015, 10:14:04 am
I'd say applying that patch AND enabling harden glue by default would probably be the safest option considering the havoc this was able to cause.

In the mean time, even with harden glue on, I'm going to keep my 0.0.0.0 overrides ;)
Title: Re: [SOLVED] Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 10, 2015, 11:08:29 am
Suggestion for pfSense 2.2.1 : apply this patch to unbound and/or activate the harden-options by default.  I'll check later if there is anything in redmine about this issue or will create a new one accordingly.

Voilà : https://redmine.pfsense.org/issues/4402
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 10, 2015, 11:19:22 am
harden-glue is already on by default in RELENG_2_2:

https://redmine.pfsense.org/projects/pfsense/repository/revisions/ef120e878558f84ed14369a484c0938ccd1b6db5

(The iter_scrub.c patch still useful though...)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 0x10C on February 10, 2015, 01:21:35 pm
I just started getting this problem today for the first time since I installed the 2.2-RELEASE.

I was also using the 8.8.8.8 and 8.8.4.4 DNS from Google. I've changed to using OpenDNS now which seems to have stopped the problem?

The thread is really long I read the first few pages, should I activate the Harden Glue feature? - Just a worried noobie.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 10, 2015, 01:24:35 pm
Harden Glue - yes, that definitely should be enabled. Otherwise, OpenDNS does not support DNSSEC at all, so all those other DNSSEC-related hardening features are kinda useless till you switch back to Google, or just disable the forwarding altogether.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 0x10C on February 10, 2015, 01:35:20 pm
Ok so I've turned Hardened Glue on, what else should I enable?
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: heper on February 10, 2015, 01:38:31 pm
Have you tried checking:

Harden Glue

Harden DNSSEC data

These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 0x10C on February 10, 2015, 01:40:46 pm
Have you tried checking:

Harden Glue

Harden DNSSEC data

These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.

Okay I've enabled both of those. They will work okay with OpenDNS? - Thanks :)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 10, 2015, 01:42:34 pm
They will work okay with OpenDNS? - Thanks :)

Goto (https://forum.pfsense.org/index.php?topic=87491.msg488877#msg488877)...
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: 0x10C on February 10, 2015, 01:53:27 pm
They will work okay with OpenDNS? - Thanks :)

Goto (https://forum.pfsense.org/index.php?topic=87491.msg488877#msg488877)...

I read that but I just wanted to confirm because I wasn't sure if you mistyped what you said due to the way it was worded.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: cmb on February 11, 2015, 02:19:07 am
Okay I've enabled both of those. They will work okay with OpenDNS?

Yes
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: kejianshi on February 11, 2015, 02:28:16 am
Just to be clear, with harden glue I see no issue, but I'm not sure that opendns supports DNSSEC, so if forwarding from opendns with DNSSEC enabled, might get "nothing".

I'm not running opendns here, but seems like it didn't support DNSSEC in the past.  Not sure about now.

(For me at least, trying to use DNSSEC with non-DNSSEC capable servers results in DNS failure to resolve)
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: doktornotor on February 11, 2015, 05:13:50 am
Obviously, as already stated multiple times here, hardening features that make use of DNSSEC will do absolutely nothing useful when you are forwarding DNS queries to OpenDNS or any other open DNS server that does not support DNSSEC at all. If you want DNSSEC, stop using OpenDNS.
Title: Re: Periodic since 2.2 pages load blank, certs invalid
Post by: swix on February 12, 2015, 09:29:56 am
Final followup here for me (I hope): Unbound 1.5.2rc1 has just been released, http://www.unbound.net/pipermail/unbound-users/2015-February/003774.html

Interesting part of the release notes in our case:

Quote
This release fixes a DNSSEC validation issue when an upstream server
with different trust anchors introduces unsigned records in messages.
 Harden-glue when turned off allows potentially poisonous records in
the cache in the hopes of that enabling DNS resolution for 'impossible
to resolve' domains, it is fixed to have 'less cache poisoning',
quotes added because it is by definition not secure to turn off
harden-glue.  New features are that "inform" can be used to see which
IPs lookup a domain, and unbound-control can use named unix pipes.

According to Chris in Redmine, this should be fixed in 2.2.1.