Netgate SG-1000 microFirewall

Author Topic: Poor performance with 2.4.1  (Read 1406 times)

0 Members and 1 Guest are viewing this topic.

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Poor performance with 2.4.1
« on: October 30, 2017, 05:45:12 am »
Yesterday, after I updated to 2.4.1, I noticed web sites took a lot longer to load.  I ran speedtest.net and got only about 14 Mb down, when I normally get mid 70s.  Upload was unaffected at the normal about 11 Mb.  I rebooted both pfSense and cable modem and now speedtest download is normal, but the web sites are still slow to load.  For example, when I reload the page for this site in the Chrome browser, it normally happens so fast I have to watch closely to verify it actually reloaded.  Now it takes a few seconds.  Firefox is sluggish too.

Has anyone else noticed this?

Offline NogBadTheBad

  • Sr. Member
  • ****
  • Posts: 497
  • Karma: +45/-0
    • View Profile
Re: Poor performance with 2.4.1
« Reply #1 on: October 30, 2017, 07:18:15 am »
Nope still getting my regular speeds.

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #2 on: October 30, 2017, 07:50:38 am »
Nope still getting my regular speeds.

After rebooting, I'm now getting normal bandwidth from speedtest, but just connecting is taking much longer.  Even getting new messages headers with IMAP seems to be taking longer.  I'm thinking perhaps a DNS issue.  I'm using the resolver.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 15120
  • Karma: +1411/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Poor performance with 2.4.1
« Reply #3 on: October 30, 2017, 12:57:48 pm »
If I had to guess prob something with your boxes trying to use ULA addresses.. ;)

If you believe its dns related, and your running the resolver.. Then why don't you troubleshoot simple process of resolving something.  Once something is cached then resolving drops off the table as a problem.. 

What does simple query from your client look like when you try and resolve something.. Use your fav tool, dig, nslookup, host, etc.  While a dns problem might cause you not to be able to resolve a specific host, or possible delay in the lookup... Once you talked to server for your IMAP.. it would have nothing to do with downloading the message headers.. DNS would no longer be in the loop after you looked up the imap server via its name, etc.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #4 on: October 30, 2017, 01:06:47 pm »
At the moment, I'm using the Google DNS, instead of pfSense on this computer.  It appears to work better.  I'll try your suggestions later.  However, this happened immediately after I updated to 2.4.1 yesterday.  Also, I'm aware of the cache effect.


« Last Edit: October 30, 2017, 01:18:03 pm by JKnott »

haleakalas

  • Guest
Re: Poor performance with 2.4.1
« Reply #5 on: October 30, 2017, 01:58:43 pm »
Yesterday, after I updated to 2.4.1, I noticed web sites took a lot longer to load.  I ran speedtest.net and got only about 14 Mb down, when I normally get mid 70s.  Upload was unaffected at the normal about 11 Mb.  I rebooted both pfSense and cable modem and now speedtest download is normal, but the web sites are still slow to load.  For example, when I reload the page for this site in the Chrome browser, it normally happens so fast I have to watch closely to verify it actually reloaded.  Now it takes a few seconds.  Firefox is sluggish too.

Has anyone else noticed this?

@JKnott : We do maintain a large number of pfSense boxes for our SOHO users at large, all built on relatively modest hardware, some of them as old as 10, some brand new, mostly Intel but also some AMD cpus.

Our internal stats show that pfsense 241 is significantly more sensitive to hardware component mix than 2.3.4-p1 was.
We can't figure out if the main cause is simply FreeBSD 11.1 or there are other reasons.

On practically all our hardware (which was tentatively upgraded) the upgrade process has failed at the first try!
One a very few, a second or even a third attempt allowed to get the upgrade to go through to full completion, but always with some dysfunctional package or setting somewhere.

So as a matter of procedure we decided to backup 2.3.4-p1's configuration (for each piece of hardware individually), then run a clean 2.4.0 or 2.4.1 install and then restore the old config. That's how we got to get most of our hardware up and running.

But then we started to observe performance issues or freezing or disappearance of some hardware.
For instance in many cases the USB-GPS dongles (which we use as a time source for NTP) would first work fine but an hour or two later would simply disappear.

We see lots of WAN connection issues where the internet connection suddenly dies out for 5-10 seconds and comes back. As we heavily run 2-way video and lots of voip that kind of disturbance becomes visible by the users immediately.

The worst part is that on a large number of hardware we observe slow but constant performance degradation. Initially just a speed issue, with some GUI sluggishness and gradual freezing of the whole box. In some cases this happens 2-3 days later.

Many of our SOHO users decided to switch back to 2.3.4-p1.

We are a Linux shop but not really FreeBSD specialists, so we have just started to dig into the root cause analysis with some help from BSD folks.

But all in all we are concerned about the evolution of this platform, but unfortunately it is not as if there are tons of alternatives that suit our budgetary constraints.

There was a time the pfsense routers of ours ran 70 to 90 days in a row untouched, undisturbed, at peak performance. (Version 2.2.2 gave us that kind of reliable performance)
Since the 2.3.x generation it's hard to see a machine running more than 2 weeks without having to be rebooted for one reason or another.


Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #6 on: October 30, 2017, 02:17:56 pm »
Quote
@JKnott : We do maintain a large number of pfSense boxes for our SOHO users at large, all built on relatively modest hardware, some of them as old as 10, some brand new, mostly Intel but also some AMD cpus.

My system is built on an refurb HP computer with an AMD CPU.  There was no problem upgrading, but the performance hit was immediately noticeable.  I had not seen a performance change with any other update in the 1.5 years I've been running pfSense.  I'm also a lot stronger on Linux than FreeBSD.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9787
  • Karma: +1105/-311
    • View Profile
Re: Poor performance with 2.4.1
« Reply #7 on: October 30, 2017, 02:21:41 pm »
If you think it's DNS, dig/drill are your friends.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #8 on: October 30, 2017, 02:46:10 pm »
I just verified it's the pfSense DNS.  I set my computer's DNS back to pfSense and the first time I reloaded the forum index page, it took several seconds.  Subsequent reloads were quick.  I also tried the Google news page.  The first time is took about 18 seconds, the next 2.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9787
  • Karma: +1105/-311
    • View Profile
Re: Poor performance with 2.4.1
« Reply #9 on: October 30, 2017, 02:50:35 pm »
dig/drill
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

haleakalas

  • Guest
Re: Poor performance with 2.4.1
« Reply #10 on: October 30, 2017, 02:56:40 pm »
@JKnott : What is your RTT and RTTsd values under WAN Gateway? Have you seen any significant change from version 234 to 241?
If you have a spare disk with your 234 backup copy and you can swap between 234 and 241 you can quickly get to the bottom of the speed issue.

Offline hda

  • Sr. Member
  • ****
  • Posts: 599
  • Karma: +32/-4
    • View Profile
Re: Poor performance with 2.4.1
« Reply #11 on: October 30, 2017, 03:07:21 pm »
... I'm thinking perhaps a DNS issue.  I'm using the resolver.
Do you also use in "General DNS Resolver Options" Network Interfaces :: "All" and Outgoing Network Interfaces :: "All" ?

I myself see better performance if using Network Interfaces :: "All" (or any iface selections) and Outgoing Network Interfaces :: "WAN"

But then... the DNS Resolver Log records like mad with the address of my WAN Link-Local IPv6 like:
Quote
Oct 30 17:30:29    unbound    45462:3    error: can't bind socket: Can't assign requested address for fe80::20d:b9ff:fe40:79b8
Oct 30 17:30:29    unbound    45462:3    error: can't bind socket: Can't assign requested address for fe80::20d:b9ff:fe40:79b8
....
Why ? I did not select it...  Is this error an unwanted feature ?
And why does the logging keep quiet when selecting "All & All".

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #12 on: October 30, 2017, 03:21:38 pm »
I just ran dig.

When I don't specify server:
dig cnn.com

; <<>> DiG 9.9.9-P1 <<>> cnn.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59675
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                59      IN      A       151.101.129.67
cnn.com.                59      IN      A       151.101.193.67
cnn.com.                59      IN      A       151.101.1.67
cnn.com.                59      IN      A       151.101.65.67

;; Query time: 410 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Mon Oct 30 16:12:31 EDT 2017
;; MSG SIZE  rcvd: 100

The server the response comes from is the 2nd in resolv.conf.  PfSense is the first.

When I specify that same DNS server:

dig cnn.com

; <<>> DiG 9.9.9-P1 <<>> cnn.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59675
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                59      IN      A       151.101.129.67
cnn.com.                59      IN      A       151.101.193.67
cnn.com.                59      IN      A       151.101.1.67
cnn.com.                59      IN      A       151.101.65.67

;; Query time: 410 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Mon Oct 30 16:12:31 EDT 2017
;; MSG SIZE  rcvd: 100


Now when I specify the pfSense firewall:

dig @<address removed> cnn.com

; <<>> DiG 9.9.9-P1 <<>> @<address removed> cnn.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


Looks to me like my pfSense DNS resolver is not working at all for servers on the Internet.  It does appear to work for local hosts.  The delay when I first try to access a site would be caused by the failure and then trying the 2nd DNS listed in resolv.conf.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9787
  • Karma: +1105/-311
    • View Profile
Re: Poor performance with 2.4.1
« Reply #13 on: October 30, 2017, 03:24:32 pm »
;; connection timed out; no servers could be reached

Not responding at all. check the config on whatever <address removed> is. Make sure you can reach that. Make sure that query is not blocked by firewall rules, etc etc etc
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Online JKnott

  • Hero Member
  • *****
  • Posts: 1176
  • Karma: +50/-11
    • View Profile
Re: Poor performance with 2.4.1
« Reply #14 on: October 30, 2017, 03:27:38 pm »
;; connection timed out; no servers could be reached

Not responding at all. check the config on whatever <address removed> is. Make sure you can reach that. Make sure that query is not blocked by firewall rules, etc etc etc

That <address removed> is the public address for the LAN side of my firewall.  Since I can get to the Internet through pfSense, I can certainly reach it, access the configuration etc..