pfSense Support Subscription

Author Topic: The HFSC wizard gives about 50% of the download speed I expect.  (Read 379 times)

0 Members and 1 Guest are viewing this topic.

Offline ryan29

  • Jr. Member
  • **
  • Posts: 53
  • Karma: +19/-0
    • View Profile
Hi,

As I'm trying to learn how traffic shaping works in pfSense, the configuration I get from the wizard has me a bit confused.  I'll list my questions first and then explain how I tested.

  • What's the purpose of the qLink queue?  I've read that it's intended for multi-LAN setups, but it's limited to the speeds I set for upload download and that doesn't make sense to me.  The LAN-to-LAN transfers get limited to those speeds.  The only explanation I can think of is that I've misunderstood the upload / download speed settings and I'm supposed to be setting them close to the interface speed instead of my internet speeds.
  • Why does my super simple, wizard base config limit download speeds to half the speed I expect?  This is my main question for now and I describe how I set up and tested below.
  • Why does the wizard bump my WAN download speed to 51200Kbit/s when I enter 50000Kbit/s?
  • Why does the wizard include outbound in the floating rules description when the direction is set to all?  Based on my testing they match incoming (port forwarded) traffic and outgoing traffic.  Did I get it wrong?
  • What are the implications of un-categorized traffic having ACKs going into the default queue?  I noted this on one of my listings of queue stats (note qACK isn't used).  Is it something I should be trying to avoid?

As for question #2, I've tried to create the simplest 1 WAN, 1 LAN configuration possible.  I've done the following:
  • Run the multi-WAN, multi-LAN wizard and select 1 WAN, 1 LAN.
  • Set the WAN to 50000Kbit/s download and 5000Kbit/s upload.
  • Enable Other networking protocols, but don't adjust any of the pre-defined services.
This creates the following queues:

Code: [Select]
QUEUE                 BW SCH  PRIO
root_em0           5000K hfsc    0
 qInternet         5000K hfsc     
  qACK             1000K hfsc     
  qDefault          500K hfsc     
  qOthersHigh       500K hfsc     
  qOthersLow        250K hfsc     
root_em1             51M hfsc    0
 qLink               10M hfsc     
 qInternet           40M hfsc     
  qACK             8192K hfsc     
  qOthersHigh      4096K hfsc     
  qOthersLow       2048K hfsc     

Next I create 2 floating firewall rules (in/out aka all directions):

Code: [Select]
Source   Port   Destination   Port   Gateway   Queue              Description
*        *      *             5201   *         qACK/qOthersHigh   iperf
*        *      *             5202   *         qACK/qOthersLow    iperf

I also create LAN interface rules to block all traffic except the ports I'm going to use for testing with iperf (5201, 5202, 5203).  I run iperf3 in server mode on a machine outside the WAN:

Code: [Select]
10.10.10.1 (iperf server) -- WAN (pfSense) LAN -- 192.168.70.10 (iperf client)
At this point, I run some tests with iperf.  The first test transfers data from the client to the server.  It should be limited to 5Mbit/s.  It looks reasonable to me:

Code: [Select]
iperf3 --client 10.10.10.1
Code: [Select]
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  5.98 MBytes  5.02 Mbits/sec  141             sender
[  4]   0.00-10.00  sec  5.71 MBytes  4.79 Mbits/sec                  receiver

The queue stats after the above look like this:

Code: [Select]
QUEUE                 BW SCH  PRIO    PKTS   BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN   P/S    B/S
root_em0           5000K hfsc    0       0       0        0        0    0                   0      0
 qInternet         5000K hfsc            0       0        0        0    0                   0      0
  qACK             1000K hfsc           10     660        0        0    0                   0      0
  qDefault          500K hfsc            1      42        0        0    0                   0      0
  qOthersHigh       500K hfsc         4155 6276586      141   213474    0                   0      0
  qOthersLow        250K hfsc            0       0        0        0    0                   0      0
root_em1             51M hfsc    0       0       0        0        0    0                   0      0
 qLink               10M hfsc           21    4050        0        0    0                 1.0    149
 qInternet           40M hfsc            0       0        0        0    0                   0      0
  qACK             8192K hfsc         2535  179010        0        0    0                   0      0
  qOthersHigh      4096K hfsc            6     601        0        0    0                   0      0
  qOthersLow       2048K hfsc            0       0        0        0    0                   0      0

Next I run the test in the reverse mode so data is transferred from the server to the client.  Based on the Upper Limit of 40Mbit/s that's set on the qInternet queue I expect this test to be limited to 40Mbit/s.  However, it's not.  It's much slower:

Code: [Select]
iperf3 --client 10.10.10.1 --reverse
Code: [Select]
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  21.8 MBytes  18.2 Mbits/sec                  sender
[  4]   0.00-10.00  sec  21.6 MBytes  18.2 Mbits/sec                  receiver

The queue stats look like this:

Code: [Select]
QUEUE                 BW SCH  PRIO    PKTS   BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN   P/S    B/S
root_em0           5000K hfsc    0       0       0        0        0    0                   0      0
 qInternet         5000K hfsc            0       0        0        0    0                   0      0
  qACK             1000K hfsc        12566  900668        0        0    0                   0      0
  qDefault          500K hfsc            0       0        0        0    0                   0      0
  qOthersHigh       500K hfsc           10    1079        0        0    0                   0      0
  qOthersLow        250K hfsc            0       0        0        0    0                   0      0
root_em1             51M hfsc    0       0       0        0        0    0                   0      0
 qLink               10M hfsc           13    3138        0        0    0                 1.0    293
 qInternet           40M hfsc            0       0        0        0    0                   0      0
  qACK             8192K hfsc            6     412        0        0    0                   0      0
  qOthersHigh      4096K hfsc        15766  23277K      536   809547    0                   0      0
  qOthersLow       2048K hfsc            0       0        0        0    0                   0      0

When I run the same tests on port 5202 so they get put into the qOthersLow queue the results are almost identical.  However, if I run the second test on port 5203 which defaults into the qLink (default) queue, I get results that I expect.  There's no other traffic on the link and there's no Upper Limit, so it runs at 50Mbit/s.  My SSH connection to pfSense also puts a small amount of traffic into the qLink (default) queue, but it's negligible:

Code: [Select]
iperf3 --client 10.10.10.1 --port 5203 --reverse
Code: [Select]
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  58.6 MBytes  49.2 Mbits/sec                  sender
[  4]   0.00-10.00  sec  58.6 MBytes  49.2 Mbits/sec                  receiver

The queue stats look like this after the above test (note qACK isn't used):

Code: [Select]
QUEUE                 BW SCH  PRIO    PKTS   BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN   P/S    B/S
root_em0           5000K hfsc    0       0       0        0        0    0                   0      0
 qInternet         5000K hfsc            0       0        0        0    0                   0      0
  qACK             1000K hfsc            0       0        0        0    0                   0      0
  qDefault          500K hfsc        20874 1401547        0        0    0                   0      0
  qOthersHigh       500K hfsc            0       0        0        0    0                   0      0
  qOthersLow        250K hfsc            0       0        0        0    0                   0      0
root_em1             51M hfsc    0       0       0        0        0    0                   0      0
 qLink               10M hfsc        42524  62774K       40    60560    0                 1.0    149
 qInternet           40M hfsc            0       0        0        0    0                   0      0
  qACK             8192K hfsc            0       0        0        0    0                   0      0
  qOthersHigh      4096K hfsc            0       0        0        0    0                   0      0
  qOthersLow       2048K hfsc            0       0        0        0    0                   0      0

If I run my high priority and low priority tests simultaneously, the aggregate bandwidth is still less than I expect.

High priority:

Code: [Select]
iperf3 --client 10.10.10.1 --reverse
Code: [Select]
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  16.8 MBytes  14.1 Mbits/sec                  sender
[  4]   0.00-10.00  sec  16.6 MBytes  13.9 Mbits/sec                  receiver

Low priority:

Code: [Select]
iperf3 --client 10.10.10.1 --port 5202 --reverse
Code: [Select]
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  15.5 MBytes  13.0 Mbits/sec                  sender
[  4]   0.00-10.00  sec  15.3 MBytes  12.9 Mbits/sec                  receiver

Queue stats:

Code: [Select]
QUEUE                 BW SCH  PRIO    PKTS   BYTES   DROP_P   DROP_B QLEN BORROW SUSPEN   P/S    B/S
root_em0           5000K hfsc    0       0       0        0        0    0                   0      0
 qInternet         5000K hfsc            0       0        0        0    0                   0      0
  qACK             1000K hfsc        19659 1419178        0        0    0                   0      0
  qDefault          500K hfsc            1      42        0        0    0                   0      0
  qOthersHigh       500K hfsc           11    1133        0        0    0                   0      0
  qOthersLow        250K hfsc           11    1133        0        0    0                   0      0
root_em1             51M hfsc    0       0       0        0        0    0                   0      0
 qLink               10M hfsc           20    4840        0        0    0                 1.0    165
 qInternet           40M hfsc            0       0        0        0    0                   0      0
  qACK             8192K hfsc           14     956        0        0    0                   0      0
  qOthersHigh      4096K hfsc        12155  17937K      467   705838    0                   0      0
  qOthersLow       2048K hfsc        11245  16596K      441   667437    0                   0      0

Are the results I'm seeing are consistent with what I should expect?  Should I be seeing 40Mbit/s on the test I think is too slow or am I misunderstanding something about the way the wizard configures things?

Offline moscato359

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +10/-6
    • View Profile
Re: The HFSC wizard gives about 50% of the download speed I expect.
« Reply #1 on: April 21, 2017, 06:47:43 am »
Hi,

As I'm trying to learn how traffic shaping works in pfSense, the configuration I get from the wizard has me a bit confused.  I'll list my questions first and then explain how I tested.

  • What's the purpose of the qLink queue?  I've read that it's intended for multi-LAN setups, but it's limited to the speeds I set for upload download and that doesn't make sense to me.  The LAN-to-LAN transfers get limited to those speeds.  The only explanation I can think of is that I've misunderstood the upload / download speed settings and I'm supposed to be setting them close to the interface speed instead of my internet speeds.
  • Why does my super simple, wizard base config limit download speeds to half the speed I expect?  This is my main question for now and I describe how I set up and tested below.
  • Why does the wizard bump my WAN download speed to 51200Kbit/s when I enter 50000Kbit/s?
  • Why does the wizard include outbound in the floating rules description when the direction is set to all?  Based on my testing they match incoming (port forwarded) traffic and outgoing traffic.  Did I get it wrong?
  • What are the implications of un-categorized traffic having ACKs going into the default queue?  I noted this on one of my listings of queue stats (note qACK isn't used).  Is it something I should be trying to avoid?


qLink is for multi lan, so you can maintain high speed lan to lan connections while traffic shaping internet. The traffic shaper wizard is insane, and sets things terribly. It conceptually had the right idea and then did it wrong.
Q2: the traffic wizard is insane
Q3: binary math is weird, but it's better that way. Rounds to the nearest binary multiple, in theory.
Q4: the traffic wizard is insane
Q5: putting uncategorized traffic into a prioritized ack queue will increase their total throughput, which may cause situations where non utp bit torrent traffic gets prioritized




Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2157
  • Karma: +194/-11
    • View Profile
Re: The HFSC wizard gives about 50% of the download speed I expect.
« Reply #2 on: April 21, 2017, 10:56:48 am »
That's a lot of dropped packets. Did you size your queues correctly? They default to 50, which is crazy small. I recommend just checking the Codel box under each queue, if you don't know what you're doing.

Offline ryan29

  • Jr. Member
  • **
  • Posts: 53
  • Karma: +19/-0
    • View Profile
Re: The HFSC wizard gives about 50% of the download speed I expect.
« Reply #3 on: April 23, 2017, 05:53:19 pm »
Harvy66, you nailed it.  I left the default queue sizes and didn't realize they were too small.  I guess pfTop must be averaging QLEN or something because it was rarely hitting double digits.  Bumping the queue limit to 500 (since I know qLink works with 500) gives me the results I was expecting.  It was the first thing I tried, so you saved me a bunch of time.  Thanks.

moscato359, it helps to know the wizard might not be making the best choices.  When I'm trying to learn something I usually assume the defaults are well tuned and that if they don't make sense to me it's a misunderstanding on my end.  Also, thanks for the explanation regarding my 5th question.