pfSense Support Subscription

Author Topic: Periodic since 2.2 pages load blank, certs invalid  (Read 14881 times)

0 Members and 1 Guest are viewing this topic.

Online kejianshi

  • Hero Member
  • *****
  • Posts: 4934
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #45 on: February 05, 2015, 09:51:05 pm »
Yep...   Now who could do that on a broad basis?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9100
  • Karma: +1037/-307
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #46 on: February 05, 2015, 09:54:36 pm »
It's intriguing.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Trel

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +11/-1
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #47 on: February 06, 2015, 08:39:24 am »
Actually once I switched fully to unbound + DNSSEC only, I had a new issue.  At the same times, unbound would stop working.  The service would be running, but it wouldn't resolve anything until I restarted the service.

I finally found a common thread for that happening.  It almost always directly followed someone doing a lookup of

Code: [Select]
api-nyc01.exip.org
or
Code: [Select]
ns3.csof.net
The IP for those are in the 195.22.x range that was mentioned earlier.

Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

If someone is willing to look at that, because of how it lines up, it looks like trying to access/doing a lookup on those domains will either cause the blank pages and lolcat certs, or will cause unbound to stop resolving until the service is restarted. 

It's too coincidental to ignore in this case.


Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +958/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #48 on: February 06, 2015, 08:44:48 am »
It almost always directly followed someone doing a lookup of

Code: [Select]
api-nyc01.exip.org
or
Code: [Select]
ns3.csof.net
The IP for those are in the 195.22.x range that was mentioned earlier.
Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

Tried both, unbound still working. :) Apparently no NSA love here.  :'( ;D
Do NOT PM for help!

Offline Trel

  • Sr. Member
  • ****
  • Posts: 368
  • Karma: +11/-1
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #49 on: February 06, 2015, 08:53:38 am »
Now, I also had block rules in place for that range of IP.
I wonder if that could interact in some way.

Additionally, if you have Snort/Suricata installed, do you now have alerts mentioning the Anubis DNS Sinkhole?

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +958/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #50 on: February 06, 2015, 08:59:20 am »
if you have Snort/Suricata installed

Noooooooooooooo!!!
Do NOT PM for help!

Online kejianshi

  • Hero Member
  • *****
  • Posts: 4934
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #51 on: February 06, 2015, 11:24:25 am »
!!!

Offline heper

  • Hero Member
  • *****
  • Posts: 2688
  • Karma: +253/-11
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #52 on: February 07, 2015, 11:41:11 am »
(unbound has been enabled for more then a month without issues ... until an hour ago)
i've suddenly been experiencing the blank pages + dns redirects to buydomains.com for lots of valid domains.

i tried to fix it by enabling dnssec ... didn't help

for now i've enabled "forwarding mode" on unbound ... this seems to fix the issue.


Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9100
  • Karma: +1037/-307
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #53 on: February 07, 2015, 12:15:23 pm »
Unbound in resolver mode?  That makes no sense.  The deal is is that makes it pretty much impossible to affect everything.  They have to target specific name servers for specific domains (or .com, or . (root) etc..

What DNS servers are you handing out to your clients?  Running unbound means nothing if your clients are going to 8.8.8.8 / 8.8.4.4 for DNS.

Want an easy way to find out?  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged....
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Online kejianshi

  • Hero Member
  • *****
  • Posts: 4934
  • Karma: +196/-40
  • Debugging...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #54 on: February 07, 2015, 12:56:05 pm »
After you properly set up DNS and DNSSEC, you still have to clear DNS cache on each client and also have to make sure your clients are not infected with something or running some stupid browser add-on that hijacks things. 

Offline 2chemlud

  • Sr. Member
  • ****
  • Posts: 385
  • Karma: +20/-5
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #55 on: February 07, 2015, 01:15:48 pm »
  ....  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged....

Port 53 is not allowed at my network for more than a year. Doing fine with the DNS servers in the General setup and keep awful devices such as Buffallo Linkstations etc from phoning home...
https://www.youtube.com/watch?v=TlBIa8z_Mts

I have been permanently banned from posting/PM on this forum. Sorry, no replies... ;-)

Offline swix

  • Jr. Member
  • **
  • Posts: 26
  • Karma: +1/-0
    • View Profile
same thing just happened here (pfsense 2.2 + dns resolver active)
« Reply #56 on: February 09, 2015, 05:53:52 am »
When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

Code: [Select]
swix@pc:~> host google.ch
google.ch has address 195.22.26.248                                                                                                                                                                   
google.ch mail is handled by 10 mx1.csof.net.                                                                                                                                                         
google.ch mail is handled by 10 mx2.csof.net.                         
                                                                                                                               
swix@pc:~> host aaaaaafadkfjdu93jifa.ch
aaaaaafadkfjdu93jifa.ch has address 195.22.26.248                                                                                                                                                 
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx1.csof.net.                                                                                                                                       
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx2.csof.net.

Unbound server is set as local resolver for a small LAN, with no forwarding to remote resolvers, so everything should be resolved locally.

Still investigating about how this could happen, and I will update this thread as soon as I find anything.
Kind regards.

PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.

Offline Pakken

  • Jr. Member
  • **
  • Posts: 45
  • Karma: +0/-0
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #57 on: February 09, 2015, 06:32:10 am »
So far happened only one time for me.
After enabling dnssec and disabling all the forwards to public dns servers it seems to be fixed.
In addition, I've created a floating rule to block every local subnet to that 195.22.0.0 range.

Will keep you updated.
To be honest the strange thing is that in a couple of years of pfsense pre-2.2 and dnsmasq this never happened.
The problem appeared straight after upgrading to 2.2 and dnsresolver even tho, once again, only happened one time so far to me.

Best regards

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +958/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #58 on: February 09, 2015, 07:00:27 am »
When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
https://www.virustotal.com/en/ip-address/195.22.26.248/information/

If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.

PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.

Disabling DNSSEC most certainly does NOT help anything. Very broken idea.
Do NOT PM for help!

Offline swix

  • Jr. Member
  • **
  • Posts: 26
  • Karma: +1/-0
    • View Profile
Re: Periodic since 2.2 pages load blank, certs invalid
« Reply #59 on: February 09, 2015, 07:38:57 am »
I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
https://www.virustotal.com/en/ip-address/195.22.26.248/information/
If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.
ng DNSSEC most certainly does NOT help anything. Very broken idea.

Thanks for the suggestion, yes, I will try to have a look on this, but the network device (VDSL Bridge Zyxel P-870M) is in bridge mode, so I have no way to connect directly to it (or only via a serial console, with a cable to be found yet).   Newest Firmware = 2009.

It just happened again a few minutes ago (3rd time today).

I was also trying to see if the root-servers file was tempered anyhow, but /etc/unbound/root.hints does not exist at all on the pfsense router.

Log extract when problem is happening, with many requests to "ns*.csof.net" servers where it shouldn't be the case  :

Code: [Select]
Feb  9 12:55:25 pf unbound: [39509:0] info: reply from <4.85.in-addr.arpa.> 195.186.196.180#53
Feb  9 12:55:25 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN       
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.csof.net. AAAA IN 
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns4.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.> 54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <ubuntu.com.> 195.22.26.248#53          ########## wrong !
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER

TBC.