pfSense Forum

pfSense English Support => Installation and Upgrades => Topic started by: JKnott on October 30, 2017, 05:45:12 am

Title: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 05:45:12 am
Yesterday, after I updated to 2.4.1, I noticed web sites took a lot longer to load.  I ran speedtest.net and got only about 14 Mb down, when I normally get mid 70s.  Upload was unaffected at the normal about 11 Mb.  I rebooted both pfSense and cable modem and now speedtest download is normal, but the web sites are still slow to load.  For example, when I reload the page for this site in the Chrome browser, it normally happens so fast I have to watch closely to verify it actually reloaded.  Now it takes a few seconds.  Firefox is sluggish too.

Has anyone else noticed this?
Title: Re: Poor performance with 2.4.1
Post by: NogBadTheBad on October 30, 2017, 07:18:15 am
Nope still getting my regular speeds.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 07:50:38 am
Nope still getting my regular speeds.

After rebooting, I'm now getting normal bandwidth from speedtest, but just connecting is taking much longer.  Even getting new messages headers with IMAP seems to be taking longer.  I'm thinking perhaps a DNS issue.  I'm using the resolver.
Title: Re: Poor performance with 2.4.1
Post by: johnpoz on October 30, 2017, 12:57:48 pm
If I had to guess prob something with your boxes trying to use ULA addresses.. ;)

If you believe its dns related, and your running the resolver.. Then why don't you troubleshoot simple process of resolving something.  Once something is cached then resolving drops off the table as a problem.. 

What does simple query from your client look like when you try and resolve something.. Use your fav tool, dig, nslookup, host, etc.  While a dns problem might cause you not to be able to resolve a specific host, or possible delay in the lookup... Once you talked to server for your IMAP.. it would have nothing to do with downloading the message headers.. DNS would no longer be in the loop after you looked up the imap server via its name, etc.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 01:06:47 pm
At the moment, I'm using the Google DNS, instead of pfSense on this computer.  It appears to work better.  I'll try your suggestions later.  However, this happened immediately after I updated to 2.4.1 yesterday.  Also, I'm aware of the cache effect.


Title: Re: Poor performance with 2.4.1
Post by: haleakalas on October 30, 2017, 01:58:43 pm
Yesterday, after I updated to 2.4.1, I noticed web sites took a lot longer to load.  I ran speedtest.net and got only about 14 Mb down, when I normally get mid 70s.  Upload was unaffected at the normal about 11 Mb.  I rebooted both pfSense and cable modem and now speedtest download is normal, but the web sites are still slow to load.  For example, when I reload the page for this site in the Chrome browser, it normally happens so fast I have to watch closely to verify it actually reloaded.  Now it takes a few seconds.  Firefox is sluggish too.

Has anyone else noticed this?

@JKnott : We do maintain a large number of pfSense boxes for our SOHO users at large, all built on relatively modest hardware, some of them as old as 10, some brand new, mostly Intel but also some AMD cpus.

Our internal stats show that pfsense 241 is significantly more sensitive to hardware component mix than 2.3.4-p1 was.
We can't figure out if the main cause is simply FreeBSD 11.1 or there are other reasons.

On practically all our hardware (which was tentatively upgraded) the upgrade process has failed at the first try!
One a very few, a second or even a third attempt allowed to get the upgrade to go through to full completion, but always with some dysfunctional package or setting somewhere.

So as a matter of procedure we decided to backup 2.3.4-p1's configuration (for each piece of hardware individually), then run a clean 2.4.0 or 2.4.1 install and then restore the old config. That's how we got to get most of our hardware up and running.

But then we started to observe performance issues or freezing or disappearance of some hardware.
For instance in many cases the USB-GPS dongles (which we use as a time source for NTP) would first work fine but an hour or two later would simply disappear.

We see lots of WAN connection issues where the internet connection suddenly dies out for 5-10 seconds and comes back. As we heavily run 2-way video and lots of voip that kind of disturbance becomes visible by the users immediately.

The worst part is that on a large number of hardware we observe slow but constant performance degradation. Initially just a speed issue, with some GUI sluggishness and gradual freezing of the whole box. In some cases this happens 2-3 days later.

Many of our SOHO users decided to switch back to 2.3.4-p1.

We are a Linux shop but not really FreeBSD specialists, so we have just started to dig into the root cause analysis with some help from BSD folks.

But all in all we are concerned about the evolution of this platform, but unfortunately it is not as if there are tons of alternatives that suit our budgetary constraints.

There was a time the pfsense routers of ours ran 70 to 90 days in a row untouched, undisturbed, at peak performance. (Version 2.2.2 gave us that kind of reliable performance)
Since the 2.3.x generation it's hard to see a machine running more than 2 weeks without having to be rebooted for one reason or another.

Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 02:17:56 pm
Quote
@JKnott : We do maintain a large number of pfSense boxes for our SOHO users at large, all built on relatively modest hardware, some of them as old as 10, some brand new, mostly Intel but also some AMD cpus.

My system is built on an refurb HP computer with an AMD CPU.  There was no problem upgrading, but the performance hit was immediately noticeable.  I had not seen a performance change with any other update in the 1.5 years I've been running pfSense.  I'm also a lot stronger on Linux than FreeBSD.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 02:21:41 pm
If you think it's DNS, dig/drill are your friends.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 02:46:10 pm
I just verified it's the pfSense DNS.  I set my computer's DNS back to pfSense and the first time I reloaded the forum index page, it took several seconds.  Subsequent reloads were quick.  I also tried the Google news page.  The first time is took about 18 seconds, the next 2.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 02:50:35 pm
dig/drill
Title: Re: Poor performance with 2.4.1
Post by: haleakalas on October 30, 2017, 02:56:40 pm
@JKnott : What is your RTT and RTTsd values under WAN Gateway? Have you seen any significant change from version 234 to 241?
If you have a spare disk with your 234 backup copy and you can swap between 234 and 241 you can quickly get to the bottom of the speed issue.
Title: Re: Poor performance with 2.4.1
Post by: hda on October 30, 2017, 03:07:21 pm
... I'm thinking perhaps a DNS issue.  I'm using the resolver.
Do you also use in "General DNS Resolver Options" Network Interfaces :: "All" and Outgoing Network Interfaces :: "All" ?

I myself see better performance if using Network Interfaces :: "All" (or any iface selections) and Outgoing Network Interfaces :: "WAN"

But then... the DNS Resolver Log records like mad with the address of my WAN Link-Local IPv6 like:
Quote
Oct 30 17:30:29    unbound    45462:3    error: can't bind socket: Can't assign requested address for fe80::20d:b9ff:fe40:79b8
Oct 30 17:30:29    unbound    45462:3    error: can't bind socket: Can't assign requested address for fe80::20d:b9ff:fe40:79b8
....
Why ? I did not select it...  Is this error an unwanted feature ?
And why does the logging keep quiet when selecting "All & All".
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:21:38 pm
I just ran dig.

When I don't specify server:
dig cnn.com

; <<>> DiG 9.9.9-P1 <<>> cnn.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59675
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                59      IN      A       151.101.129.67
cnn.com.                59      IN      A       151.101.193.67
cnn.com.                59      IN      A       151.101.1.67
cnn.com.                59      IN      A       151.101.65.67

;; Query time: 410 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Mon Oct 30 16:12:31 EDT 2017
;; MSG SIZE  rcvd: 100

The server the response comes from is the 2nd in resolv.conf.  PfSense is the first.

When I specify that same DNS server:

dig cnn.com

; <<>> DiG 9.9.9-P1 <<>> cnn.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59675
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                59      IN      A       151.101.129.67
cnn.com.                59      IN      A       151.101.193.67
cnn.com.                59      IN      A       151.101.1.67
cnn.com.                59      IN      A       151.101.65.67

;; Query time: 410 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Mon Oct 30 16:12:31 EDT 2017
;; MSG SIZE  rcvd: 100


Now when I specify the pfSense firewall:

dig @<address removed> cnn.com

; <<>> DiG 9.9.9-P1 <<>> @<address removed> cnn.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


Looks to me like my pfSense DNS resolver is not working at all for servers on the Internet.  It does appear to work for local hosts.  The delay when I first try to access a site would be caused by the failure and then trying the 2nd DNS listed in resolv.conf.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 03:24:32 pm
;; connection timed out; no servers could be reached

Not responding at all. check the config on whatever <address removed> is. Make sure you can reach that. Make sure that query is not blocked by firewall rules, etc etc etc
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:27:38 pm
;; connection timed out; no servers could be reached

Not responding at all. check the config on whatever <address removed> is. Make sure you can reach that. Make sure that query is not blocked by firewall rules, etc etc etc

That <address removed> is the public address for the LAN side of my firewall.  Since I can get to the Internet through pfSense, I can certainly reach it, access the configuration etc..
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:29:02 pm
Quote
Do you also use in "General DNS Resolver Options" Network Interfaces :: "All" and Outgoing Network Interfaces :: "All" ?

I have WAN selected for outgoing and everything but WAN for the LAN side.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:30:58 pm
@JKnott : What is your RTT and RTTsd values under WAN Gateway? Have you seen any significant change from version 234 to 241?
If you have a spare disk with your 234 backup copy and you can swap between 234 and 241 you can quickly get to the bottom of the speed issue.

I have never checked RTT etc., so I don't know what they were before.  However, as I mentioned in another note, pfSense is flat out failing to resolve external addresses, but appears to be OK for local.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 03:43:14 pm
Quote
I have WAN selected for outgoing and everything but WAN for the LAN side.

Just select All and All and try again. It sounds like you are not actually listening on the address you are specifying.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:47:07 pm
The service status shows DNS Resolver stopped and I can't start it.

The log has several lines of "Oct 30 16:18:37   unbound   95941:0   error: can't bind socket: Can't assign requested address for fe80::214:d1ff:fe2b:edea".  That's the link local address for my WAN port.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 03:49:11 pm
Quote
I have WAN selected for outgoing and everything but WAN for the LAN side.

Just select All and All and try again. It sounds like you are not actually listening on the address you are specifying.

That seems to have it working.  Why would this change between versions?
Title: Re: Poor performance with 2.4.1
Post by: hda on October 30, 2017, 04:56:51 pm
I have WAN selected for outgoing and everything but WAN for the LAN side.

Finally I found the Resolver corresponding settings which work perfect, fast and no errors in Log.

For me I have set with GUI:
Network Interfaces: LAN, OPT1, OPT2, Localhost
Outgoing Network Interfaces: Localhost

In unbound.conf that is correctly found as:
Quote
# Interface IP(s) to bind to
interface: 192.168.1.1
interface: 2001:****:####:1::1
interface: 10.8.4.1
interface: 192.168.22.1
interface: 2001:****:####:3::1
interface: 127.0.0.1
interface: ::1

# Outgoing interfaces to be used
outgoing-interface: 127.0.0.1
outgoing-interface: ::1

Besides this, the "All & All" works too, but you probably don't want listening on WAN ;)


My setup in 2.4.1 (upgraded from 2.4.0) about DNS:
 - No Forwarding with Resolver
 - Nothing set or checked for DNS in [System > General Setup]
 - No other DNS config for DHCP(6) servers || RA

Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 05:11:00 pm
^^^^
I'll give those a try.  DNS through pfSense has now failed completely.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 05:22:03 pm
Didn't work.  I still have complete DNS failure with pfSense.  I cannot resolve either Internet or local host names.  Something is clearly messed up here.  Is there any way to revert back to 2.4.0?
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 30, 2017, 05:23:42 pm
For a test.  Disable resolver and enable forwarder.  See what happens.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 06:08:09 pm
For a test.  Disable resolver and enable forwarder.  See what happens.

That appears to work, though I no longer have the local hosts available through it.
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 30, 2017, 06:11:22 pm
Yeah - I'm having the same troubles on both a pfsense vm and opnsense vm.  In vmware with a private IP at wan. 
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 06:22:44 pm
If there isn't a fix for the resolver soon, I'll have to copy all my local devices into the forwarder.
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 30, 2017, 06:25:36 pm
I think its a resolver specific issue and it will be fixed.   til then, I like your fix.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 07:15:59 pm
No idea what you guys are doing. Resolver works fine in 2.4.1.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 08:21:45 pm
No idea what you guys are doing. Resolver works fine in 2.4.1.

I updated to 2.4.1.  I guess I shouldn't have done that.
Title: Re: Poor performance with 2.4.1
Post by: Derelict on October 30, 2017, 08:22:18 pm
Resolver works fine.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 30, 2017, 08:32:10 pm
Resolver works fine.

I just tried again and resolver does not work.  Forwarder does.  I have been using resolver almost since I started using pfSense 1.5 years ago but it now fails.

Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 31, 2017, 02:35:41 am
In my case, I think its something in the network at this one place giving unbound trouble.  I haven't seen this anywhere else.  Since in my case, its just for testing I didn't worry about it much.  However in this 1 location both opnsense and pfsense had resolver issues, so I turned it off.

Went with dnsmasq on opnsense and forwarder on pfsense and suddenly it all worked.  I think its something strange going on with the machine hosting the VMs in my case because this only happened in one place. 

The only things odd about this install is its in vmware and the IP on the WAN is private.  Like I said...   For testing only, so no public on this one.  Other than that, its vanilla as can be. 
Title: Re: Poor performance with 2.4.1
Post by: johnpoz on October 31, 2017, 04:24:40 am
No problems with resolver here..

Prob timeouts with its ULA address.. Because your RA failed and its using your "backup" plan of ULA addresses..

How about some info on how its failing.. So you do a query for www.domainx.com and it doesn't walk down from roots?  You looked in the cache of unbound for how it would look up this domain, what it has in its cache, etc.  You sniffed on wan and don't see this, but there is nothing in the logs?

example

Code: [Select]
unbound-control -c /var/unbound/unbound.conf lookup forum.pfsense.org
The following name servers are used for lookup of forum.pfsense.org.
;rrset 1279 2 0 7 3
pfsense.org.    1279    IN      NS      ns2.netgate.com.
pfsense.org.    1279    IN      NS      ns1.netgate.com.
;rrset 1279 1 0 8 0
ns1.netgate.com.        1279    IN      A       192.207.126.6
;rrset 84078 1 0 1 0
ns1.netgate.com.        170478  IN      AAAA    2610:160:11:3::6
;rrset 1279 1 0 8 0
ns2.netgate.com.        1279    IN      A       162.208.119.38
;rrset 84078 1 0 1 0
ns2.netgate.com.        170478  IN      AAAA    2610:1c1:3::108
Delegation with 2 names, of which 0 can be examined to query further addresses.
It provides 4 IP addresses.
2610:1c1:3::108         not in infra cache.
162.208.119.38          rto 328 msec, ttl 840, ping 4 var 81 rtt 328, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
2610:160:11:3::6        rto 376 msec, ttl 840, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed.
192.207.126.6           rto 347 msec, ttl 840, ping 7 var 85 rtt 347, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
[2.4.1-RELEASE][root@pfsense.local.lan]/root:

Is there anything in the log for unbound?  Did you up the verbosity of what it logs, etc..

Resolver does not work... Like telling your mechanic - car is broke..
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 31, 2017, 04:40:20 am
For me, I wasn't all that worried because I was more interested in stepping through the menues and comparing menues, options, features of two firewall distros than anything.  I need to move several older machines to something else when the AES-NI requirement kicks in. 

I wonder what unbound would do if you turned off DNSSEC/hardening?  I'm going to try because I suspect for me it could be an ISP issue. 
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 31, 2017, 05:07:37 am
I tried changing quite a few things in unbound but nothing works.  This isn't specific to pfsense either.  For these test VMs, I'm fine with forwarder.  Done fooling with it.
Title: Re: Poor performance with 2.4.1
Post by: johnpoz on October 31, 2017, 05:43:07 am
"Done fooling with it."

Well from what you posted that's all you were doing with it anyway.  I see no info from you either on what is not actually working?  Nothing from logs, nothing from how it would look up anything.  No checking to see if actually sends query to roots, and then walks down the tree, etc..

Does a dig +trace work from a client.. This would simulate walking down tree like unbound would do, etc.

example - I got rid of the dnssec info so it was cleaner looking trace

Code: [Select]
> dig forum.pfsense.org +trace +nodnssec

; <<>> DiG 9.11.2 <<>> forum.pfsense.org +trace +nodnssec
;; global options: +cmd
.                       502339  IN      NS      e.root-servers.net.
.                       502339  IN      NS      f.root-servers.net.
.                       502339  IN      NS      l.root-servers.net.
.                       502339  IN      NS      b.root-servers.net.
.                       502339  IN      NS      i.root-servers.net.
.                       502339  IN      NS      k.root-servers.net.
.                       502339  IN      NS      m.root-servers.net.
.                       502339  IN      NS      g.root-servers.net.
.                       502339  IN      NS      a.root-servers.net.
.                       502339  IN      NS      j.root-servers.net.
.                       502339  IN      NS      d.root-servers.net.
.                       502339  IN      NS      c.root-servers.net.
.                       502339  IN      NS      h.root-servers.net.
;; Received 239 bytes from 192.168.3.10#53(192.168.3.10) in 3 ms

org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      d0.org.afilias-nst.org.
;; Received 448 bytes from 192.203.230.10#53(e.root-servers.net) in 14 ms

pfsense.org.            86400   IN      NS      ns1.netgate.com.
pfsense.org.            86400   IN      NS      ns2.netgate.com.
;; Received 93 bytes from 199.19.53.1#53(c0.org.afilias-nst.info) in 33 ms

forum.pfsense.org.      300     IN      A       208.123.73.18
pfsense.org.            300     IN      NS      ns1.netgate.com.
pfsense.org.            300     IN      NS      ns2.netgate.com.
;; Received 141 bytes from 162.208.119.38#53(ns2.netgate.com) in 37 ms

I don't get this thought process... I clicked some stuff.. Not working.. Well just use forwarder then... Do you not want to know why something is not working?  Could be something completely broken in your firewall causing the problem.. Could be your isp is intercepting your dns traffic, and while you think your forwarding to X.. Your really just getting whatever your ISP wants to send you..  Some simple testing would tell you why your having a problem with resolving.. Maybe your isp just blocks outbound to 53 and only allows their NS or specific NS, etc.??
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 31, 2017, 06:01:26 am
Quote
I think its something strange going on with the machine hosting the VMs in my case because this only happened in one place. 

I'm not running pfSense in a VM.  It's a bare metal install on a computer.
Title: Re: Poor performance with 2.4.1
Post by: JKnott on October 31, 2017, 06:06:06 am
Quote
Resolver does not work... Like telling your mechanic - car is broke..

Well, prior to 2.4.1, resolver worked fine but failed immediately on the upgrade and I hadn't changed anything else.  Sure sounds like a resolver issue to me.

Quote
I don't get this thought process... I clicked some stuff.. Not working.. Well just use forwarder then... Do you not want to know why something is not working?  Could be something completely broken in your firewall causing the problem.. Could be your isp is intercepting your dns traffic, and while you think your forwarding to X.

While I would like to know what caused the problem and how to fix it, having a working network is more important.  Switching to forwarder does that.  I think it's extremely unlikely that my ISP would change things at the precise time I upgraded to 2.4.1.
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 31, 2017, 06:17:57 am
Yep - I'm just mashing random buttons on this thing...   Like a monkey with a keyboard. 
I will figure it out eventually.  It isn't anything simple.
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on October 31, 2017, 06:28:38 am
; <<>> DiG 9.11.2 <<>> forum.pfsense.org +trace +nodnssec
;; global options: +cmd
;; connection timed out; no servers could be reached


When I find something helpful I will post it.  This just isn't informative.  I'm still trying to get past "it brokted"
I'm wondering if the ISP can somehow break this?
Title: Re: Poor performance with 2.4.1
Post by: isopede on January 21, 2018, 01:08:55 pm
Just as another data point, I upgraded from 2.4.1 to 2.4.2 a few days ago and started noticing these symptoms.

My connection is fine, I'm able to ping the gateway and monitoring shows no degradation in quality (no packet drops).

DNS lookups using the resolver however, will occasionally fail for many seconds before returning a result. This includes internal lookups (static entries and DHCP lookups).

I switched to the forwarder and everything seems fine.
Title: Re: Poor performance with 2.4.1
Post by: kejianshi on February 08, 2018, 08:55:45 am
I think that ISPs can impact the reliability of resolver.  I don't really care what anyone thinks about that.

I think some ISPs are living in the 80s and 90s and just havent dropped some bad practices, like blocking all dns other than their own.