Netgate SG-1000 microFirewall

Author Topic: DNS over TLS forwarding howto  (Read 1420 times)

0 Members and 1 Guest are viewing this topic.

Offline tman222

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +6/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #30 on: November 04, 2017, 08:43:35 pm »
"forwarding (to e.g. OpenDNS or Google instead of going to the root DNS servers)?"

Thought you said you read the RFC??  When you use the forwarder you do not talk to roots, you would have to send the forwarder the FULL thing your looking for, not just the pieces of the fqdn you need to find the authoritative server.  So you could ask it for the record..

So in this scenario instead of asking roots hey whats the NS for .com in www.domain.com - its just asks hey whats the NS for .com
Then asks hey NS for .com whats the NS for domain.com, vs asking for NS of www.domain.com

How would that work with a forwarder?

You do not need to edit the conf file directly.. Just add it to the custom options box.

here... This simple.. see attached.

I ask the resolver hey www.testthisdomain.com and it just ask the NS for .com for testthisdomain.com vs the www.testthisdomain.com see attached sniff pic.

> dig -x 192.31.80.30 +short
d.gtld-servers.net.

;; QUESTION SECTION:
;com.                           IN      NS

;; ANSWER SECTION:
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.

Thanks John, that makes a lot more sense to me now.  I actually went ahead and tried this out today with qname-minimisation: yes enabled, and forwarding disabled and so far I haven't had any issues with DNS lookups.  I won't enable strict mode just yet as I saw that you guys ran into some issues with it.

I do have more general question through regarding the Unbound resolver though:  In the past I've had the resolver enabled with forwarding and used Google's public DNS servers as the upstream DNS servers to forward queries to in order to try to improve performance.  Today I disabled forwarding to try out qname-minimisation for better privacy and to be honest I've haven't really found browsing to be any slower (despite the additional lookups that may be occurring).  I'm curious if everyone else thinks that the added privacy is worth a bit of a performance hit in DNS lookup speed or whether one would be better of just using a the resolver together with forwarding to a fast public DNS  (e.g. Google).  What does everyone think?

Thanks in advance.

Offline kejianshi

  • Hero Member
  • *****
  • Posts: 4948
  • Karma: +195/-40
  • Debugging...
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #31 on: November 04, 2017, 09:06:46 pm »
Hmmmm...

Assuming you are using a caching resolver like unbound and its getting its DNS with a bit of added encryption overhead for end users it would not be much of a performance hit at all because only the first query is via TLS.  Future queries will pull from the cache as long as they remain valid.  I'd want the root servers to be what I was connecting to though.

Now here is the issue I see with the encryption.  The server load.  Its going to be just crazy with TLS added.  I mean, I have DNS over TLS right now since it all comes from my pfsense in the USA over vpn.  This is done for many reasons, one of which is privacy. 

It will be more meaningful if TLS is rolled into unbound and bind and all the goodness starts with the root servers.  Technology advances may make this feasible and may have already.  Not sure..  I'm 100% sure the feature was not included due to hardware load issues and bandwidth considerations as well as latency.  Not because the people who designed bind and unbound don't know how to write the code. 
« Last Edit: November 04, 2017, 09:12:04 pm by kejianshi »

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14426
  • Karma: +1336/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #32 on: November 05, 2017, 01:37:27 am »
"I've haven't really found browsing to be any slower (despite the additional lookups that may be occurring)."

Not sure where that little bit of FUD started that resolving is going to be slower.. In any sense that would matter.. Unless your on a some really high latency connection. The only time that there would be any sort of delay would be a cold lookup having to walk all the way down. 

So you asking google for something.. What is that 30ms..  What does it take to fully resolve something?  Lets call it 300ms worst case -- lets just say..  But that is only on the COLD lookup.. Your talking less then 3 tenths of a second..

After that then your 1ms since the next lookups for www.domain.tld will come from cache.

Now keep in mind that was FULL cold lookup.. Had to talk to roots.. But you have looked up the NS for the tld in question, you no longer have to go ask roots every time. you look up newdomain.tld.. It will go straight their for the NS of newdomain.tld

Keep in mind that talking to the authoritative servers means you will always get the FULL TTL... not whatever google had left on its timer.. So Maybe you got 10 seconds left, and then have to do a query again in 10 seconds.  While your resolver got the full 1 hour ttl, etc. So if you got two queries in 1 hour the forwarder would have to get asked twice while you resolver would have only had to do the 1 lookup.

Once something is cached it all becomes moot if your resolving or forwarding.  Since its cached.. So why would you care about a few ms difference when looking up something cold.. Your telling me you can tell the difference in your page load be it it took 30 ms or 300ms..  Really?

Keep in mind that 300ms is most likely really high exaggeration... But you could always do some testing if you curious... Just clear your unbound cache and then do a simple dig.. How long does it take to resolve..

Unless your isp is only allowing you to talk to its ns, or your on really high latency internet - say sat or something.  Resolving is going to be better all the way around, privacy, security and performance is a non issue to be honest.. Since once something has been looked up.. Your client caches it at the os, the browser caches it, your local dns caches it.. Your not cold resolving every single time all the way down from roots, etc.

Shoot resolving can even be faster.. Once you know the NS for domainX, looking up any records in domainX now just mean talking directly to that NS - which could even be closer to you in RTT or faster to respond than google..  Here just did a query direct to google for forum.pfsense.org took 64ms.  While query direct to the NS for pfsense.org only took 52 ms.. And got back more data..  Why anyone would forward - especially if privacy is a concern.. And just giving someone all your info vs spreading it around to just the authoritative servers your wanting to go too and the roots.. That are global controlled setup.. You really think they are tracking billy's IP is going to sites xyz and selling that info?

edit:
Resolver also has a setting or prefetch.. Which can remove the full Colds or even the talking to the authoritative NS delay out... Since the resolver will resolve www.domain.tld  if someone asks for it when there is 10% or less left on the TTL.. So lets say the TTL is 24 hours.. If someone asks for that record and there is less than 2.4 hours left on the TTL then the resolver will refresh that info by resolving it again and now the TTL is back to 24 hours.  In the back ground, so next query comes by for that record.  Its there waiting in the 1ms response cache.  No need to even go ask google for it at 30ms, etc.
« Last Edit: November 05, 2017, 01:44:17 am by johnpoz »
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE (home)

Offline V3lcr0

  • Full Member
  • ***
  • Posts: 183
  • Karma: +7/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #33 on: November 05, 2017, 08:09:13 am »
I have tried using forwarding(to OpenDNS) and using unbound with no forwarding...I have found negligible difference in speed(didn't actually measure the speed, I just didn't notice while surfing), however after a reboot it can be very slow using unbound(John not sure that is what you mean by "Cold" lookup)...it then speeds up.

I did play with enable "qname-minimisation-strict" to see how it worked...I didn't do extensive testing but noticed it broke Skype while using the IOS app...
« Last Edit: November 05, 2017, 08:21:23 am by V3lcr0 »

Offline tman222

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +6/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #34 on: November 05, 2017, 11:17:50 am »
"I've haven't really found browsing to be any slower (despite the additional lookups that may be occurring)."

Not sure where that little bit of FUD started that resolving is going to be slower.. In any sense that would matter.. Unless your on a some really high latency connection. The only time that there would be any sort of delay would be a cold lookup having to walk all the way down. 

So you asking google for something.. What is that 30ms..  What does it take to fully resolve something?  Lets call it 300ms worst case -- lets just say..  But that is only on the COLD lookup.. Your talking less then 3 tenths of a second..

After that then your 1ms since the next lookups for www.domain.tld will come from cache.

Now keep in mind that was FULL cold lookup.. Had to talk to roots.. But you have looked up the NS for the tld in question, you no longer have to go ask roots every time. you look up newdomain.tld.. It will go straight their for the NS of newdomain.tld

Keep in mind that talking to the authoritative servers means you will always get the FULL TTL... not whatever google had left on its timer.. So Maybe you got 10 seconds left, and then have to do a query again in 10 seconds.  While your resolver got the full 1 hour ttl, etc. So if you got two queries in 1 hour the forwarder would have to get asked twice while you resolver would have only had to do the 1 lookup.

Once something is cached it all becomes moot if your resolving or forwarding.  Since its cached.. So why would you care about a few ms difference when looking up something cold.. Your telling me you can tell the difference in your page load be it it took 30 ms or 300ms..  Really?

Keep in mind that 300ms is most likely really high exaggeration... But you could always do some testing if you curious... Just clear your unbound cache and then do a simple dig.. How long does it take to resolve..

Unless your isp is only allowing you to talk to its ns, or your on really high latency internet - say sat or something.  Resolving is going to be better all the way around, privacy, security and performance is a non issue to be honest.. Since once something has been looked up.. Your client caches it at the os, the browser caches it, your local dns caches it.. Your not cold resolving every single time all the way down from roots, etc.

Shoot resolving can even be faster.. Once you know the NS for domainX, looking up any records in domainX now just mean talking directly to that NS - which could even be closer to you in RTT or faster to respond than google..  Here just did a query direct to google for forum.pfsense.org took 64ms.  While query direct to the NS for pfsense.org only took 52 ms.. And got back more data..  Why anyone would forward - especially if privacy is a concern.. And just giving someone all your info vs spreading it around to just the authoritative servers your wanting to go too and the roots.. That are global controlled setup.. You really think they are tracking billy's IP is going to sites xyz and selling that info?

edit:
Resolver also has a setting or prefetch.. Which can remove the full Colds or even the talking to the authoritative NS delay out... Since the resolver will resolve www.domain.tld  if someone asks for it when there is 10% or less left on the TTL.. So lets say the TTL is 24 hours.. If someone asks for that record and there is less than 2.4 hours left on the TTL then the resolver will refresh that info by resolving it again and now the TTL is back to 24 hours.  In the back ground, so next query comes by for that record.  Its there waiting in the 1ms response cache.  No need to even go ask google for it at 30ms, etc.

Thanks John for taking the time to craft this very helpful response.  After reading what you wrote and doing a bit more research on my own (looking through past threads on Unbound and performance) I understand the tradeoff's far better now.  This thread was also very helpful:

https://forum.pfsense.org/index.php?topic=112160.0

I think a lot of the performance concerns can be traced back to the fact that a good number of domains today have pretty short TTL's (e.g. like www.cnn.com in the thread linked above).  So on a small network (e.g. a home network), the TTL will expire after a few minutes necessitating another lookup the next time someone makes a DNS request for that same website.  That lookup may/may not be faster than going against e.g. Google's DNS servers which likely already have the DNS record in question cached.  I think that's probably where the main privacy/performance tradeoff comes in.  In my opinion, adding up to a few tenths of a second of additional latency is worth it instead of sending all queries to one set of DNS servers on the internet.  If speed is of upmost concern then using then using Unbound with forwarding enabled (to a fast DNS server with a large cache) is probably an option to consider.

Now it's possible to tweak the TTL values on Unbound to improve performance.  Frankly, I wouldn't really want to touch the minimum TTL value (default is 0) since you run the risk of then having expired/bad records in the cache.  However, what do you think about the max TTL value?  I see that it is set to 86400 (1 day) by default.  By running dig to a few sites on the net I saw that the expiry on records can be higher.  Would it be bad idea to increase this value to two days (172800) instead to make sure that cache records don't expire prematurely?

--------------------

Anyway, I do not want to take this thread too far off-topic.  I found some additional information regarding using Unbound over TLS:

https://calomel.org/unbound_dns.html

In general, I would agree with what kejianshi posted.  Seems to me the additional overhead would add enough load on servers to slow down what was intended to be fast and lightweight protocol.  That being said, hardware and connections speeds have continued to improve, so would be cool if this could be implemented at some point.  For now, I think using the Unbound resolver with forwarding disabled (and the qname-minimisation option discussed in this thread enabled) is an easy to improve privacy by 1. limiting the amount of information that is shared during the recursive DNS lookup and 2) spreading the DNS request across multiple servers from multiple orgs vs. just one.

Thanks again to all for this great discussion.

Offline PertFlavus

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #35 on: November 06, 2017, 01:53:28 am »
Unbound in particular makes for a pretty poor client for dns over tls at the moment. If you look at the implementation progress, it doesn't support reusing sessions or out of order queries. As a server these are no issue but as a client it will mean creating and tearing down a connection with each query. This matters because unbound is effectively the dns client in this example.

Freebsd seems really good at keeping up with unbound releases, so hopefully in the not too distant future this won't be the case.

Offline tman222

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +6/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #36 on: November 07, 2017, 08:52:48 pm »
I'm curious if anyone had an opinion on whether it might make sense to increase the max TTL value in the resolver from 1 to 2 days to potentially improve cache performance?  Or is best to keep it at 1 day or lower?  Thanks again.

Offline PertFlavus

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #37 on: November 08, 2017, 09:22:47 am »
I'm curious if anyone had an opinion on whether it might make sense to increase the max TTL value in the resolver from 1 to 2 days to potentially improve cache performance?  Or is best to keep it at 1 day or lower?  Thanks again.

I expect not much to change with that. You should just enable prefetch support. When the 1 day period expires it'll check for an update to refresh the cache prior to you requesting it.

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 387
  • Karma: +20/-50
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #38 on: November 22, 2017, 02:14:17 pm »
guys what really helps in losing google dns caching hit rates, is the new unbound feature called serve-expired.  It is now in the pfsense GUI as one of the advanced options, so you need only tick the box.

What it does is when a dns record is cached, even when TTL hits 0 (expired) a new lookup from the lan will serve the cached record but at the same time the cache is updated so the next lookup after is more up to date.

So you can enable the privacy stuff, use your own forwarder or do direct lookups from pfsense unbound, and this option mitigates most cold cache issues.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline tman222

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +6/-0
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #39 on: November 23, 2017, 11:07:55 am »
guys what really helps in losing google dns caching hit rates, is the new unbound feature called serve-expired.  It is now in the pfsense GUI as one of the advanced options, so you need only tick the box.

What it does is when a dns record is cached, even when TTL hits 0 (expired) a new lookup from the lan will serve the cached record but at the same time the cache is updated so the next lookup after is more up to date.

So you can enable the privacy stuff, use your own forwarder or do direct lookups from pfsense unbound, and this option mitigates most cold cache issues.

So I saw this option a couple weeks ago and started wondering if there would be any adverse impact to using it.  I suppose there is the chance that the first lookup returns an incorrect result and an outdated page (or no page at all).  A quick refresh/reload on the browser should fix that though (since the cache has refreshed in the background).  Are there any security considerations one should be aware of when enabling this feature?  I actually decided to try it (i.e. enable serve-expired) out and so far everything is working fine.  I can see this option as potentially being beneficial on smaller networks with fewer clients.

This leads me to a more general question:  Why do a lot of major sites have short TTL's these days?  Is this being done for load balancing reasons?

Thanks in advance.

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 387
  • Karma: +20/-50
    • View Profile
Re: DNS over TLS forwarding howto
« Reply #40 on: November 23, 2017, 01:47:28 pm »
There is a chance of course you end up going to an invalid ip, but in my experience the chance of that happening is extremely tiny.  The providers that set silly low TTL that last just a few seconds change so they can redirect quickly in the event of an outage and for load balancing purposes, I cannot remember this causing me a problem in the several weeks I have been using it.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK