pfSense Forum

pfSense English Support => Traffic Shaping => Topic started by: qubit on March 04, 2017, 04:49:31 pm

Title: playing with fq_codel in 2.4
Post by: qubit on March 04, 2017, 04:49:31 pm
If you use limiters on 2.4 and check the system log you may have seen this pop up

Code: [Select]
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded

FQ_CODEL was added to FreeBSD in 11.0 in dummynet/ipfw, and since 2.4 is based on that we can enable it by hand without recompiling anything.

Note: This doesn't look like it will officially be in 2.4 via the GUI, and may need more testing. Since we're messing around with the command line, bad things may happen so use at your own risk.

Start with a recent 2.4 snapshot. Create two root limiters, Download and Upload, and put 95% your maximum values in bandwidth. Create two queues under each, say LAN and WAN. For LAN, selection destination addresses for mask and source addresses for WAN. Modify the default outgoing firewall rule to use WAN under "in" pipe and LAN under "out" pipe.

This generates /tmp/rules.limiter with something like the following:

Code: [Select]
pipe 1 config  bw 85Mb
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff


pipe 2 config  bw 9Mb
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff

and the firewall rule adds a "dnqueue( 2,1)" in /tmp/rules.debug for the outgoing lan rule.

Without messing with php we can manually change this to fq_codel and have it persist across reboots and ruleset reloads.

cp /tmp/rules.limiter to /root/rules.limiter

I edited /etc/inc/shaper.inc as follows:

Code: [Select]
4599c4599,4600
<               mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
---
>               #mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
>               mwexec("/sbin/ipfw /root/rules.limiter");

replace /root/rules.limiter with:

Code: [Select]
pipe 1 config  bw 85Mb
sched 1 config pipe 1 type fq_codel
queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

pipe 2 config  bw 9Mb
sched 2 config pipe 2 type fq_codel
queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff

replace your bandwidth numbers with your own

Trigger a rule reload (disable, apply, reenable a rule) and kill states. Might want to run "ipfw pipe flush" before doing that. then verify in command line:

Code: [Select]
[2.4.0-BETA][admin@pfsense.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     1450  2175000 31 46500   0
00002:   9.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       21      840  0    0   0

[2.4.0-BETA][admin@pfsense.lan]/root: ipfw queue show
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

fq_codel is running, and we're passing traffic. Cool.


I tried using limiters a long time ago but had to stop due to some problems with dropped traffic probably relating to my hardware and igb. I then just had two interface shapers, WAN and LAN with CODELQ, set to 95% my upload and download. This stopped bufferbloat, but I noticed that most real traffic would actually be half of these values.

So far this has been far superior to the altq CODELQ with some of the following observations from the top of my head:

Downloads not randomly halved versus codel.
Twitch streams don't buffer when under heavy load such as steam
Two heavy bandwidth, multiple connection programs will share bandwidth evenly.
No more "sendto: No buffer space available" for unbound
Slight latency increase versus intermittent packet loss at load

Works just as good as cake in openwrt/lede from my limited home testing.

Some points:

1. Since I haven't been able to use plain limiters until now, this may just be better performance due to dummynet just limiting my bandwidth instead of fq_codel actually shaping. But it seems to perform better than plain limiters with reaching my bandwidth values versus the default WF2Q+.
2. Traffic isn't shown under queues, but 0.0.0.0/0 will show under ipfw sched, so I guess the traffic is still being shaped. I noticed this in the original dummynet aqm paper on the developers' website, so maybe it's by design.

Discuss if you've tried this or have any input. If you use limiters I'm interested if you can actually measure a difference since I'm coming from altq.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on March 04, 2017, 06:38:07 pm
ipfw -a list

That shows your rules and you can see what is matching to validate you have your rules correctly.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on March 04, 2017, 07:49:42 pm
Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.
Title: Re: playing with fq_codel in 2.4
Post by: qubit on March 04, 2017, 08:11:23 pm
ipfw -a list

That shows your rules and you can see what is matching to validate you have your rules correctly.
Code: [Select]
[2.4.0-BETA][admin@pfsense.lan]/root: ipfw -a list
ipfw: retrieving config failed: Protocol not available

dummynet is used with pf via "dnqueue" in pf rules which shows up in firewall rules via pfctl with limiters enabled.

Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.

Again probably related to my hardware. speedtests would show the full limited speeds on altq but most downloads wouldn't even reach that. Oddities like: dslreports would max out but fast.com would top out to about 40 megabits, as well as downloads via multiple browsers. Works fine without altq now. I think it was related to the igb driver as on my 2440 all networking would sometimes die and require a reboot if I disable the altq codel. Probably fixed recently by https://github.com/pfsense/FreeBSD-src/commit/42a5f2897e93d1e42833eac551c64c1373119ff9 but I haven't touched it in a while as this setup has been working great.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on March 05, 2017, 12:50:24 am
I got all three A+ on dslreports, even having active porn downloading on qbittorent. 8)
Title: Re: playing with fq_codel in 2.4
Post by: w0w on March 05, 2017, 02:51:12 am
I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.
Title: Re: playing with fq_codel in 2.4
Post by: qubit on March 05, 2017, 09:04:19 am
I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.


These are the default sysctls which may need tweaking depending on traffic and bandwidth

Code: [Select]
net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

Technical details can be found here: http://caia.swin.edu.au/freebsd/aqm/papers.html

So far I found the default ok
Title: Re: playing with fq_codel in 2.4
Post by: w0w on March 07, 2017, 10:06:50 am
I've played a bit, but I think that default are really OK.
Now I am using only IPFW FQ_CODEL shaper and disabled ALTQ, this gives me about +4Mbps on 300Mbps bandwidth if I compare with ALTQ shaper tested maximum.
So far, so good.
Title: Re: playing with fq_codel in 2.4
Post by: shinzo on March 08, 2017, 12:16:39 pm
Thanks alot.  i have been looking for something like this for a while.  I plan to use it for a while to see how things go. ;D
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on March 08, 2017, 01:28:11 pm
interesting i may try this out at some point thanks for sharing the information.
Title: Re: playing with fq_codel in 2.4
Post by: shinzo on March 09, 2017, 07:39:51 pm
I already had the limiters setup.  i Was looking on how to make it easier and i came up with this. Instead of modifying anything i just input the ipfw command to enable fq_codel and worked.

Limiters:
00001:  30.000 Mbit/s    0 ms burst 0
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00002:   6.00 Mbit/s    0 ms burst 0
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active


Queues:
q00001  50 sl. 0 flows (256 buckets) sched 3 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 4 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

ipfw sched 1 config pipe 1 type fq_codel
ipfw sched 2 config pipe 2 type fq_codel
ipfw sched show

And Done.  While it wont survive a reboot i am sure i can set something up.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on March 10, 2017, 10:59:48 am
Survivng reboot and update also is that what I needed.
Here is my patch (use System_patches package)
Code: [Select]
--- shaper.inc Mon Feb 20 18:14:04 2017
+++ shaper.inc Sun Mar 05 07:33:23 2017
@@ -4596,7 +4596,8 @@
  "net.inet.ip.dummynet.pipe_slot_limit" => $max_qlimit
  ));
  file_put_contents("{$g['tmp_path']}/rules.limiter", $dn_rules);
- mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+ #mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+ mwexec("/sbin/ipfw /root/rules.limiter");
  }
 }
 


Also, remember, you need to reboot firewall manually after update is completed or disable/enable rule where you have limiters used, like in OP first post.

Title: Re: playing with fq_codel in 2.4
Post by: Nullity on March 10, 2017, 02:39:53 pm
Regarding the sysctl defaults, this link is likely the most official source for details, particularly the "Parameters" section: https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 02, 2017, 01:38:59 am
One Sunday morning I have found that bufferbloat rating is B or even C  and no drops on my side. I've tried to play with bandwidth limiting and after changing it to twice smaller I got A rating again, looks like it's a problem on the ISP side. OK, I was thinking there is nothing to do, but why not to try to use delay instead of limiting bandwidth.
SO I changed limiter config to
pipe 1 config delay 0ms   for both pipes
And looks like this did the trick, now I have A+ bufferbloat and A or A+ Quality ratings.
Certainly, I need to do advanced tests before draw some conclusions, but it looks hopefully.
Title: Re: playing with fq_codel in 2.4
Post by: obrienmd on April 04, 2017, 06:05:14 pm
Can't WAIT for this to get into the UI.

FQ_codel's fair queuing is incredible, and HFSC + CODEL, FAIRQ + CODEL and CODELQ in pfSense can't provide multi-bucket fair queuing nearly as well.

I tested this using shellcmd so it will persist through reboots: "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" runs on reboot, with limiters and firewall pipes configured in the UI. It performs just as good as Linux's fq_codel that I have running on LEDE, IPFire and a few other boxes. pfSense getting fq_codel and wireguard would let me move entirely to pfSense / BSD on the networking side :)
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 06, 2017, 09:52:05 am
As for GUI I was thinking about building some package, but I am not any kind of php programmer and  the best would be mainstream implementation into pfsense by professionals, core team.
We can also vote for bounty and see what happens.
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on April 11, 2017, 12:16:55 pm
It's literally an on/off setting, and a kernel module
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 14, 2017, 01:46:30 am
It's literally an on/off setting, and a kernel module
Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit. 
BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 14, 2017, 03:10:09 am
It's literally an on/off setting, and a kernel module
Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit. 
BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.

Thanks for trying to explain it. When it comes to traffic-shaping, even from a user perspective (disregarding the developer implementation), rarely is anything as simple as "It's literally an on/off setting, and a kernel module".

I've been guilty of back-seat driving myself... and I'm totally, fully, absolutely awesome.  ::)
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on April 14, 2017, 11:12:45 pm
Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 15, 2017, 02:14:57 am
Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

Why don't we "just" send humans to Mars? We already have robots there.

Like I said, back-seat driving is easy.
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on April 15, 2017, 11:03:17 am
Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

Why don't we "just" send humans to Mars? We already have robots there.

Like I said, back-seat driving is easy.

Why is it under limiter, when the rest of them are under altq?
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on April 15, 2017, 04:06:46 pm
ALTQ and Limiters are two different systems. My understanding is ALTQ is PF traffic shaping and Limiters are IPFW traffic shaping. Two competing firewall systems that FreeBSD has.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 15, 2017, 10:53:44 pm
Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

Why don't we "just" send humans to Mars? We already have robots there.

Like I said, back-seat driving is easy.

Why is it under limiter, when the rest of them are under altq?

I'm a bit unclear about what you're asking but if you are asking why fq_codel was implemented in dummynet rather than ALTQ you'd need to ask the devs: http://caia.swin.edu.au/freebsd/aqm/

I'd like to know as well. Maybe they think ipfw/dummynet is more future-proof than ALTQ? I dunno...
Title: Re: playing with fq_codel in 2.4
Post by: nallar on April 18, 2017, 09:08:02 am
By default, fq_codel uses ECN.

This often doesn't work properly for upload (https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#ecn-issues) so you may need to try without it. For my config this meant using:

ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

Swap ecn/noecn as needed depending on the order you created the limiters in.
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on April 21, 2017, 08:31:02 am
Interestingly, on Linux, fq_codel is in mainstream kernel, and enabled by default now.no settings required.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 21, 2017, 01:22:50 pm
By default, fq_codel uses ECN.

This often doesn't work properly for upload (https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#ecn-issues) so you may need to try without it. For my config this meant using:

ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

Swap ecn/noecn as needed depending on the order you created the limiters in.

I know what are you talking about.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
But FQ_CODEL revision was updated several times since this article was published and no official remarks about ECN and recommended settings in docs.
I have read a lot and played a bit with ECN option, but in my case it have no effect directly. If anybody suggest some simple way to test ECN I will be much thankful.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 21, 2017, 02:15:47 pm
By default, fq_codel uses ECN.

This often doesn't work properly for upload (https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#ecn-issues) so you may need to try without it. For my config this meant using:

ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

Swap ecn/noecn as needed depending on the order you created the limiters in.

I know what are you talking about.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
But FQ_CODEL revision was updated several times since this article was published and no official remarks about ECN and recommended settings in docs.
I have read a lot and played a bit with ECN option, but in my case it have no effect directly. If anybody suggest some simple way to test ECN I will be much thankful.

You can use tcpdump to see whether ECN has been negotiated/used, then run downloads & uploads with ECN disabled/enabled to see if there's any difference in speeds and/or latencies.

For me, it improved download (or was it upload? or both?) speeds by a few percent but over a few days of using ECN (Linux client /proc/sys/net/ipv4/tcp_ecn = 1) had a couple of sites completely fail to work so I set tcp_ecn back to it's default (2).


Whether your pfSense router supports ECN is a separate condition from your client supporting it, so make sure to configure it appropriately on both.

I only played with ECN very quickly so take my input with a grain of salt... ;)
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 22, 2017, 08:28:30 am
...
For me, it improved download (or was it upload? or both?) speeds by a few percent but over a few days of using ECN (Linux client /proc/sys/net/ipv4/tcp_ecn = 1) had a couple of sites completely fail to work so I set tcp_ecn back to it's default (2).


Whether your pfSense router supports ECN is a separate condition from your client supporting it, so make sure to configure it appropriately on both.

I only played with ECN very quickly so take my input with a grain of salt... ;)
Do you remember URLs of sites failed to work with ECN?
I've seen some reports like "Measuring the State of ECN Readiness in Servers, Clients" and others too, all of them stated that there is some % of servers that have wrongly configured ECN and this is the real problem, even if percentage of those servers lowered over years, but the real quantity raised up, so the simplest way is to test ECN enabled FQ_CODEL against some of those " ECN-failed" sites.
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 22, 2017, 03:15:26 pm
Setting my bandwidth to 95% of my always results in about 20mb off of my total bandwidth in tests. It seems that to use this you have to take a bandwidth hit....
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on April 23, 2017, 11:05:41 am
I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 23, 2017, 11:20:25 am
I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

This is my experience as well. Only when I was beginning my traffic-shaping journey did I experience strange things like that. My assumption is that I was misconfiguring.

I suppose it's possible that these algorithms incorrectly calculate bitrates but that is very unlikely since transmitting at the configured bitrate is perhaps the most fundamental aspect of any traffic-shaping algorithm.
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 23, 2017, 01:52:43 pm
I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

I also have 150mb connection and am running an i5 mini PC with PFsense. It seems like a simple configuration so I'm not sure what could actually be misconfigured but I'm not ruling it out. Any ideas?
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 23, 2017, 01:56:50 pm
I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

Full disclosure, I am running a VPN, but it pins at 147mb no matter what....until this config.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 23, 2017, 02:26:36 pm
HeatmiserNYC
So, with FQ_CODEL you have 130Mbps max, right? You said -20Mbps...
The misconfiguration can be interference with other limiters or rules if you have used same limiter twice or more ó I did not checked but it was possible in certain conditions.
Also TS mentioned that this FQ_CODEL setup equalizes traffic and with VPN it can be a real problem if you have concurrent or even the same traffic on both.
Anyway, I did tests some time ago and there was 1-2 Mbps difference with bandwidth limit, if we compare to traditional HFSC this is about twice less. Now I don't use bandwidth limit but delay limit that is set to 0ms, this causes FQ_CODEL scheduler to process all traffic by using only internal parameters, I think. Double check everything and if problem persists, please provide some configuration sample.
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 23, 2017, 09:54:10 pm
Cool, thanks for replying.

Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

I use the VPN for all outbound traffic, it's not a separate situation.

I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

Just need a successful example of this to get running...

Again, thanks.
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on April 23, 2017, 10:16:20 pm
Cool, thanks for replying.

Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

I use the VPN for all outbound traffic, it's not a separate situation.

I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

Just need a successful example of this to get running...

Again, thanks.

Perhaps your speed drop is related to overhead like VPN, TCP, etc. I assume you are referring to goodput (https://en.wikipedia.org/wiki/Goodput) bitrates?

On downloads you will commonly see below the configured bitrate because each time you hit the limit pfSense will tell the sender to slow down below the limit. Personally, I found very little useful benefit by limiting downloads because my ISP has minimal bufferbloat and allowing them to do the rate-limiting gives me 100% speeds.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 23, 2017, 10:48:27 pm
What about to try to move shaper/limiters from LAN side to VPN side firewall rules?
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 23, 2017, 11:22:14 pm
That's an idea, I'll give that a shot!
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 25, 2017, 09:28:05 pm
Yea, that didn't work.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 25, 2017, 11:01:43 pm
Just for testing purpose, try to change bw limiting to delay limiting :
Code: [Select]
pipe 1 config delay 0msfor both pipes
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 29, 2017, 11:45:40 pm
Sorry gone for a few days, vacation.

I gave that a shot by changing the /tmp file, it doesn't seem to have an affect. I am only changing the /tmp file, maybe it needs to be rebooted and hardcoded into the file? The only reason I haven't done this is because I haven't seen the results everybody is reporting...
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 30, 2017, 03:05:35 am
Yes it's need to be rebooted or reloaded with
Code: [Select]
/etc/rc.reload_all in console regardless of where you coded it into. I think it's better to be used in the same file as stated by topic starter, ex.  "/root/rules.limiter" if you really did everything the right way.
After you did that run the following command
Code: [Select]
ipfw sched show and you should see something like
Code: [Select]
00001: unlimited         0 ms burst 0 for the both pipes you have.
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on April 30, 2017, 10:12:18 pm
Yes it's need to be rebooted or reloaded with
Code: [Select]
/etc/rc.reload_all in console regardless of where you coded it into. I think it's better to be used in the same file as stated by topic starter, ex.  "/root/rules.limiter" if you really did everything the right way.
After you did that run the following command
Code: [Select]
ipfw sched show and you should see something like
Code: [Select]
00001: unlimited         0 ms burst 0 for the both pipes you have.

Yes, all relatively simple and you've been great at walking through the steps you put in place.

I'm getting this for both pipes.

00003: unlimited         0 ms burst 0

00004: unlimited         0 ms burst 0

Yet I can't get better than a B rating for bufferbloat, which is the same if I literally do nothing at all....
Title: Re: playing with fq_codel in 2.4
Post by: w0w on April 30, 2017, 11:37:16 pm
But what about VPN bandwidth? Are you still getting 120Mbps?
Title: Re: playing with fq_codel in 2.4
Post by: HeatmiserNYC on May 01, 2017, 09:25:21 pm
That part HAS improved, looks like it does get about 145-ish or so which is about right. It just does nothing for bufferbloat.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on May 01, 2017, 10:45:08 pm
Can you post the full output of
Code: [Select]
ipfw sched show?
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on May 03, 2017, 10:45:06 am
I'd like to look at implementing this, but I was wondering

Anyone know the status of pfsync + limiters?
Title: Re: playing with fq_codel in 2.4
Post by: w0w on May 03, 2017, 11:34:41 am
I'd like to look at implementing this, but I was wondering

Anyone know the status of pfsync + limiters?
What was the last status you know?  :D
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on May 03, 2017, 06:05:21 pm
The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why

Title: Re: playing with fq_codel in 2.4
Post by: w0w on May 04, 2017, 12:52:26 pm
The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why
This is actual. https://redmine.pfsense.org/issues/4310 have 0% progress.
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on May 04, 2017, 10:19:46 pm
The last status I know is that the pfsense book says not to use pfsync and limiters together, but doesn't explain why
This is actual. https://redmine.pfsense.org/issues/4310 have 0% progress.


D=
Title: Re: playing with fq_codel in 2.4
Post by: moscato359 on June 08, 2017, 08:56:14 am
Is there any chance fq_codel will make it into the 2.4 GUI in limiters?
Title: Re: playing with fq_codel in 2.4
Post by: w0w on June 11, 2017, 02:30:59 am
Definitely not!
They are keeping eyes on it, but currently no plans, no moves, AFAIK.
Title: Re: playing with fq_codel in 2.4
Post by: sofakng on June 26, 2017, 10:36:11 am
Darn.  I'm thinking about switching back to pfSense but I really want fq_codel.
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on June 26, 2017, 09:41:00 pm
fq_codel, the ZFS of AQMs, or nearly. Cake aims to be the "ZFS", but close enough.
Title: Re: playing with fq_codel in 2.4
Post by: superbree on July 07, 2017, 12:44:29 pm
Is the command of "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" the same if I only have 2 root limiters?  Both of them are root limiters one has a mask of source and the other has a mask of destination.

I would like to try this out but wondering if the command is different for just root limiters without "child" queues.  Obviously I am highly dependent on the gui I am a bit confused with the ipfw command since it references both sched and pipe.

Thanks for any reply!
Title: Re: playing with fq_codel in 2.4
Post by: w0w on July 07, 2017, 10:49:14 pm
TS sample is for the root limiters also, if you have  some troubles understanding, post the content of your /tmp/rules.limiter
Title: Re: playing with fq_codel in 2.4
Post by: superbree on July 10, 2017, 11:55:28 am
here is the content of my /tmp/rules.limiter

pipe 1 config  bw 100Mb mask dst-ip6 /128 dst-ip 0xffffffff
 

pipe 2 config  bw 10Mb mask src-ip6 /128 src-ip 0xffffffff
 
I need help with the ipfw command to enable fq_codel on pipes 1 and 2 because i don't have any child queues. 

thanks in advance
Title: Re: playing with fq_codel in 2.4
Post by: w0w on July 11, 2017, 12:41:50 am
According to documentation posted in this thread you need to configure sheduler at least to make things work.

Code: [Select]
pipe 1 config bw 100Mb mask dst-ip6 /128 dst-ip 0xffffffff
sched 1 config pipe 1 type fq_codel

pipe 2 config bw 10Mb mask src-ip6 /128 src-ip 0xffffffff
sched 2 config pipe 2 type fq_codel

EDIT:
Tested, it will not work. You need to configure child queues and use them in ruleset, exactly as described by TS. Default automatically created pipe queue always uses FIFO sheduler and I am not sure it is possible to change this.

So after changes made in GUI also, you must edit and create your own rules.limiter that should look like this. 
Code: [Select]
pipe 1 config bw 100Mb
sched 1 config pipe 1 type fq_codel
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff

pipe 2 config bw 10Mb mask
sched 2 config pipe 2 type fq_codel
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff
So the right answer is no you can not shape with fq_codel using only root limiters.
Title: Re: playing with fq_codel in 2.4
Post by: superbree on July 11, 2017, 05:37:26 pm
Thats really too bad.  We use PFsense primarily to "specify bandwidth limits per host." for a small ISP.

I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs.  And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

Is it possible to combine and use ALTQ and Dummynet at the same time?  Has anyone tried that or have a config example?

I guess I could use limiters on 2 PFsense boxes.  First one limiting each host to 5 Mbps using limiters with a destination/source mask.  And the second limiting the entire subnet to 100Mbs using limiters without a mask and changing the type from WF2Q+ to FQ_Codel by issuing the command "ipfw pipe 1 config bw 100Mb type fq_codel"

I hope thats not too confusing.  Anyone have a more eloquent way of trying this?

As always, thank you for any reply.

Title: Re: playing with fq_codel in 2.4
Post by: w0w on July 12, 2017, 02:09:03 am
Yes it's possible, but  you will have some overheads and losses, you can try it at least, I think. Just set your per host limits on ALTQ shaper side and do your evenly shared FQ_CODEL enabled limiters exactly as TS described for you entire subnet.
I am sure it is possible to build ipfw only shaper model that works like you want it to work, but it would be complicated not only with pfSense and can cause some errors on pfSense.
Title: Re: playing with fq_codel in 2.4
Post by: cplmayo on August 09, 2017, 06:40:52 pm
Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.


My last speed test.


(http://www.dslreports.com/speedtest/20053541.png) (http://www.dslreports.com/speedtest/20053541)
Title: Re: playing with fq_codel in 2.4
Post by: meruem on August 25, 2017, 06:34:47 am
Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.


My last speed test.


(http://www.dslreports.com/speedtest/20053541.png) (http://www.dslreports.com/speedtest/20053541)

enable re-enable or disable re-enable or enable re-disable or .. ?
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on August 27, 2017, 03:15:19 am
fq_codel, the ZFS of AQMs, or nearly. Cake aims to be the "ZFS", but close enough.

This is very interesting.

Any chance someone(s) knowledgeable would be willing to put together a single post along the lines of this - https://forum.pfsense.org/index.php?topic=126597.0

Kind of like an fq_codel one-stop shop for the layman?
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 03, 2017, 09:33:19 am
ok am finally testing this and got it working.

I had observed some iptv/vpn issues that seemed to only occur when my ingress altq config was active, so am now testing this configuration.  I have not yet tested if this is as effective as hsfc alt for keeping steam downloads in check, I had to set the dummynet limiter to 95% of downstream cap to even get a 6 threaded downstream test to stop causing packetloss, so not confident that will be enough for a 30+ stream steam download but will see.

How granular is this? can I e.g. route steam etc. all through it but at the same time applying a limit less than 95% for steam download whilst keeping things like youtube able to burst higher.  All on dummynet.  As I have a feeling I will need to drop this to at least 90% to manage steam but I consider that too low for lighter threaded stuff.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 03, 2017, 11:11:13 am
The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

It's a granular as firewall rules can be
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 03, 2017, 11:21:45 am
The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

It's a granular as firewall rules can be

shaping is nowhere near that simple.

The reason steam is harder to shape is it opens so many threads.

I always get 100% from my isp but it doesnt mean 95% will always work well for all types of traffic.

Looking at dummynet configuration it looks like multiple specific rate limits cannot be set within one pipe however weighting can be applied so I can put steam downloads on a low weight and things like dns lookups and emails on a high weight, this is what I will look into on my config next. Thanks to the OP giving me a starting point. :)

AltQ on PFSense is incredibly granular but of course someone has put the effort into integrating it all into the GUI, and AltQ itself allows children in a queue to have their own limits set.

Already got some good results.

When I setup the dummynet config (basic as in the OP) I had a iptv stream running to my STB and I can see from my ping monitoring on my connection, the peak latency has plummeted, it was an almost steady increase in peak latency, now its spikes instead of constant and the spikes been generally much lower.

I will post back on how my steam downloads testing goes.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 03, 2017, 01:21:09 pm
That's kind of the point of fq_codel, KISS. It is by design intended to be simple.

Now you might be trying to get it to do something it wasn't designed to do, in which case yes you will have to do some weird stuff - bit more likely you should just look elsewhere.

Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 03, 2017, 01:29:00 pm
I am a fan of simple, if steam works well in the current config then the current config stays, the weighting is a fallback plan if it doesnt work well.

From what I understand all the weighting is dummynet side, it simply dynamically adjusts the throughput allowance of each thread based on the weight assigned, by default everything has the same weight.

All fully documented on the dummynet man page, I dont think its a non supported feature.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 03, 2017, 01:42:05 pm
Yeah man page is very helpful.

You should just be able to weight steam as you desire, apply the queue to a rule that catches ports and protocols for steam and let fq_codel do the rest.

I think you'll be happy with it. I just set it up and it's working very well for me.

It's awesome for weighting a guest net and primary net!
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 04, 2017, 02:44:33 am
I played some more.

First I misunderstood the man page, the weight flag does nothing on fq_codel, it only has an affect on another queuing type.

I tested steam and the result wasnt good, lots of packet loss during a steam download, the packet loss only goes close to 0% when the pipe size is below 40% of my line capacity, as I said steam is probably the most brutal traffic I have seen on my home connection.

HFSC can manage it at anything below 90%, however latency during saturation is vastly superior on fq_codel, packet loss is worse but latency better.

If I increase the queue slots to 500 (default 50), then packet loss almost stops at pipe size below 75%, with a small hit to latency.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 04, 2017, 05:37:33 am
It sounds like something else is going wrong on your box.

Weighting definitely applies to fq_codel, I've tested it on my own system and it matches weighted values every time.

I've also tested with both steam and flent rrul. No packet loss.

What is your line bandwidth? Are you trying to use dummynet and altq at the same time?
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 04, 2017, 06:29:33 am
altq is off during this testing to ensure no conflict.

Thanks for clarifying weighting has an affect I will try it on my test config as originally planned.  The reason I said it wasnt valid is because this is in the man page from dummynet section on ipfw.

Quote
     weight weight
             Specifies the weight to be used for flows matching this queue.
             The weight must be in the range 1..100, and defaults to 1.

     The following case-insensitive parameters can be configured for a
     scheduler:

     type {fifo | wf2q+ | rr | qfq}
             specifies the scheduling algorithm to use.
             fifo    is just a FIFO scheduler (which means that all packets
                     are stored in the same queue as they arrive to the
                     scheduler).  FIFO has O(1) per-packet time complexity,
                     with very low constants (estimate 60-80ns on a 2GHz
                     desktop machine) but gives no service guarantees.
             wf2q+   implements the WF2Q+ algorithm, which is a Weighted Fair
                     Queueing algorithm which permits flows to share bandwidth

     type {fifo | wf2q+ | rr | qfq}
             specifies the scheduling algorithm to use.
             fifo    is just a FIFO scheduler (which means that all packets
                     are stored in the same queue as they arrive to the
                     scheduler).  FIFO has O(1) per-packet time complexity,
                     with very low constants (estimate 60-80ns on a 2GHz
                     desktop machine) but gives no service guarantees.
             wf2q+   implements the WF2Q+ algorithm, which is a Weighted Fair
                     Queueing algorithm which permits flows to share bandwidth
                     according to their weights.  Note that weights are not
                     priorities; even a flow with a minuscule weight will
                     never starve.  WF2Q+ has O(log N) per-packet processing
                     cost, where N is the number of flows, and is the default
                     algorithm used by previous versions dummynet's queues.
             rr      implements the Deficit Round Robin algorithm, which has
                     O(1) processing costs (roughly, 100-150ns per packet) and
                     permits bandwidth allocation according to weights, but
                     with poor service guarantees.
             qfq     implements the QFQ algorithm, which is a very fast
                     variant of WF2Q+, with similar service guarantees and
                     O(1) processing costs (roughly, 200-250ns per packet).

This made me think its exclusive to wf2q+ however, fq_codel is omitted on the type section, so it didnt confirm that fq_codel has no weighting algorithm so I made the assumption.

My downstream throughput is around 71603kbit. Calculated after removing vdsl overheads, and also confirmed with experimentation when rate limiting to see when a rate limit starts having an affect.  The bandwidth is very consistent whether its on peak or off peak, if I remove shaping and download via steam it flatlines at the max speed with no dips.

Steam on average opens between 20 and 40 connections when downloading, most of these connections appear to be short lived making them very difficult to shape. Instead of downloading a large compressed file and uncompressing it steam seems to either download individual files on their own sessions or download files in fragments with the aim of maximising tcp sessions.  The problem is significantly reduced if I choose a server that has a high rtt such as in america (I am in the UK), swamping a connection with low RTT high bandwidth sessions will murder it.

I have been looking at the box configuration itself, the hardware tuning etc.  As I understand, if packets are batched together with things like interrupt moderation as well as a low kernel hertz timer, then shaping is less efficient as it cannot intervene at frequent enough intervals.   These are all things I am investigating as an ongoing process and I havent given up on this.

I have just reported back here how things went on the configuration suggested in the OP, with the only alternative config tried so far been to increase the queue depth.

So so far on my unit on my usage test patterns fq_codel via dummynet is much better for latency but worse at packet loss compared to HFSC on altq.

Also to add steam itself supports throttling the speeds, in that case I have tested on "unlimited", "7MB sec, which seems to be just below what my line can do" and "5MB sec", when steam throttles its not clean tho, it works by spiking to full speed, then pausing, then full speed again so it evens out that way.  If I set my pipe size lower and leave steam at unlimited its a clean reduction in speed flatlined at the pipe throughput rate.  throttling via steam vs the pipe size is more effective at higher speeds, but the pipe gets better when set very low.  I will report back after trying more stuff and welcome suggestions that are reasonable (trying entirely new kit is not reasonable in case you about to suggest it).
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 04, 2017, 07:15:46 am
You could try applying a different shaping algorithm to steam and see if it works better.
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 04, 2017, 11:39:49 am
I have stopped playing for now and moved back to HFSC ingress and fairq egress.

Suddenly thats improved due to the changes I made to try and help dummynet.

I did some tuning based on the information here (the author of isr code)

http://alter.org.ua/soft/fbsd/netisr/

I disabled interrupt moderation on my 2 intel ports.

I increased kernel timer rate which doesnt help interrupt overheads but I have the spare cpu clocks idle.

Now ALTQ+ HFSC is handing steam no problem at 97%, the latency performance is much closer to fq_codel as well.

I dont know yet if my iptv issues will improve or if I will still need to temporarily disable ingress ALTQ when relying on the iptv stuff.

The netisr changes I consider experimental of course as it goes against what I considered good practice (and obviously what pfsense considers good practice) the author seems to suggest deffere mode for netisr is superior for routing and direct is only best when running as a server, especially the case when you have more cpu cores than network cards.

Hopefully more people will contribute to this thread on their experience of dummynet fq_codel vs ALTQ shaping (particurly HFSC).
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 05, 2017, 06:29:31 pm
Increased the timer rate? It defaulted to 1000hz for me or at least I don't remember changing it. Was it lower for you or did you increase it even more? FreeBSD 10 added tickless kernel scheduling. I'm not sure kern.hz even applies anymore.

I'm also getting good results from ALTQ+HFSC+Codel
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 06, 2017, 06:38:28 am
yeah by default it runs double the kern hz on my unit, I had set it back to legacy behaviour so 1000khz timer, but for this I removed my override so its 2000 per second on each core.

By the way I think now this is a PPS issue that my unit is failing to handle when its too high.

It seems HFSC drops the PPS quite aggressively when it is in play and this is why it is working better for me, from what I can tell my unit starts dropping random packets when PPS is over about 1500 when it reaches 2500 or so it gets quite bad, although the download is never affected visibly.

I do have NAT, no cpu cores are been saturated, pfsense is behind a vdsl modem I have in bridge mode so there is also a chance that modem is not handling the higher pps itself.

So at least for now I will keep my feedback out of this thread given the problem for me might be related to me hitting a PPS bottleneck.
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 06, 2017, 08:12:44 am
Sounds like something is wrong. My i5-3.2ghz Haswell Intel-i350T2 is handling 1.44Mpps with HFSC+Codel just fine. 400Kpps is about 15-20% load and 1.44Mpps is about 20%-25% load. And I still get about 1ms pings while doing this.

I have some forum posts around about what changes I made, like enabling MSI-X (enable soft interrupts if your NIC properly supports it) and removing the limitation on the number of packets to process per interrupt, which is by default 40 I think.

I used to process 1.44Mpps at only 600 interrupts per second, but over the years, it is now about 1,200 interrupts per second. All I know is CPU is low, interrupts are low, latency and jitter and loss are low. Even with only 300 interrupts per core per second.
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 06, 2017, 09:03:49 am
agree something is wrong, problem is I dont know what at the moment. :)

Will diagnose some more at some point.

MSIX is enabled by default on my i350, I actually temporarily disabled it already to try and get to bottom of it but MSIX in this case is not the solution. The packets per interrupt is something I not heard off, you know where that is configured?
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 06, 2017, 10:35:24 am
agree something is wrong, problem is I dont know what at the moment. :)

Will diagnose some more at some point.

MSIX is enabled by default on my i350, I actually temporarily disabled it already to try and get to bottom of it but MSIX in this case is not the solution. The packets per interrupt is something I not heard off, you know where that is configured?

I guess it's not 40. Not sure what "40" is. I definitely remembering something.

https://calomel.org/freebsd_network_tuning.html

Quote
# Intel igb(4): FreeBSD puts an upper limit on the the number of received
# packets a network card can process to 100 packets per interrupt cycle. This
# limit is in place because of inefficiencies in IRQ sharing when the network
# card is using the same IRQ as another device. When the Intel network card is
# assigned a unique IRQ (dmesg) and MSI-X is enabled through the driver
# (hw.igb.enable_msix=1) then interrupt scheduling is significantly more
# efficient and the NIC can be allowed to process packets as fast as they are
# received. A value of "-1" means unlimited packet processing and sets the same
# value to dev.igb.0.rx_processing_limit and dev.igb.1.rx_processing_limit .
hw.igb.rx_process_limit="-1"  # (default 100 pps, packets per second)

I want to say that when I echo'd my system settings, msix was disabled and I had to explicitly add it to pfSense to enabled it. I'm also under the impression that many NICs that claim to support MSIX, do not correctly and have odd bugs when msix is enabled, so it was turned off. I'm going off of memory from a long while ago when I was researching.
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 06, 2017, 10:53:36 am
yeah mine has definitely been on, checked via dmesg.

I will turn it back on, I only turned off just to check if somehow it was not working properly.
Title: Great Results
Post by: belt9 on September 07, 2017, 04:40:58 am
I've got some great results with fq_codel, it makes significant improvements on wifi where latency can be a real problem!


https://forum.pfsense.org/index.php?topic=135843.msg745944#msg745944
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 09, 2017, 01:17:18 pm
I think its interrupt related.

I enabled polling on both nic's in use and the packet loss when downloading of steam or meganz is gone.

Before I enabled polling I was observing the interrupts/sec, first the AIM (adaptive interrupt moderation) seems to not be working on my i350 ports as it has no affect, secondly I will generate 8000/sec interrupts for a download of about 70mbit/sec.  I see people on here reporting similar usage but for half a gigabit/sec, so it seems two things pointing to moderation not working.

altq reports around 2000 pps, assuming thats accurate then some how I have 4 interrupts for every packet.

I have seen threads on here regarding fake i350's, I dont think its impossible mine is fake which could explain the broken AIM.
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 10, 2017, 01:00:11 am
You can get the number off of your nic and run it through Intel.com to see if it's valid.
https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007074.html
Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 10, 2017, 07:56:28 am
How did you enable polling?
I am not quite sure this is good idea anyway, but I can't see this option in GUI  anymore, is it moved somewhere?
You'd better try to add boot.conf.local "safe" settings for Intel cards
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0
Also I have disabled hardware TCP segmentation offload and hardware large receive offload.
P.S. In addition to Harvy66 post
https://forums.servethehome.com/index.php?threads/chinese-intel-i350-adapters.3746/#post-58686
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 10, 2017, 08:57:07 am
I compiled it into the kernel and then enabled it in the cli via ifconfig, I dont think you can enable it with a module.

Polling is seen as pointless now days, but only because modern intel and broadcom hardware do interrupt moderation, for some reason my i350 isnt moderating the interrupts.

I have already disabled flow control and played with the queues, all that made no difference.

When I can be bothered I will check how harvy said, although I am already pretty sure I had no such label on my card.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 10, 2017, 09:01:24 am
Looks like most of those chinese Intel cards are failing because of cheap fake Delta ethernet transformers used on their boards, but it always come up with packet loss and I don't think that AIM could be affected. I think they all using Intel chipset not some fake remarked crap realtek.
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 10, 2017, 09:10:49 am
Well I am just speculating.

There is no difference in the interrupts with aim on or off, and the amount of interrupts generated seems high for the pps/bw used.  So that suggests to me aim isnt working.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 10, 2017, 10:47:29 am
I have not tested AIM in any way you did it, so I can't say why it's working or not.
Also I've never seen packet loss on my igb cards, some chinese, not fake, but Winayo branded Intel NICs.
https://forums.freebsd.org/threads/62406/#post-360467 I see you are already digging deeper.
Did you already read that https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/ ? We must be 100 percent sure which card you have exactly. May be some photos?
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 10, 2017, 11:25:38 am
yep read that thread which is why I made that comment about the fake cards.  When I can be bothered I will take closer photos of the card both sides.

I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 10, 2017, 01:38:55 pm
Most of the UEFI motherboards will eat the bootmgr.efi placed in EFI\Microsoft\Boot\bootmgr.efi on FAT32 formatted USB flash.
Here you can find how to obtain it https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Obtaining_UEFI_Shell
I've used binaries from https://github.com/tianocore/edk2/tree/master/ShellBinPkg
From what I've learned reading forums the first sign that this is a fake is the price and redrawn "Delta-like" ethernet transformers on it.
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 11, 2017, 09:55:06 am
thanks for the help guys I do appreciate it.

I havent really got alternative kit to test on, but I managed to get pfsense running in esxi, after spending a couple of hours looking for a second nic (dual port server class intel nic I put in there both ports dead) and using vmx drivers in 2.4-RC is fine without polling.  So I am putting this down to a dodgy hardware setup and I think I am going to replace my unit.

But I am now going to leave this VM running so I can take the photos and play around with the firmware in the meantime without worrying about downtime.
Title: Re: playing with fq_codel in 2.4
Post by: pf3000 on September 11, 2017, 03:54:39 pm
I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.
I described how to flash via EFI here https://forum.pfsense.org/index.php?topic=112968.msg629211#msg629211
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on September 11, 2017, 05:40:23 pm
thanks is flashed, will post pics tomorrow but in a different thread as I am derailing this thread too much. I will edit this post with the link after I posted.

thread here https://forum.pfsense.org/index.php?topic=136561.new#new
Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 13, 2017, 10:52:53 pm
Moved to https://forum.pfsense.org/index.php?topic=136561.new#new
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 14, 2017, 10:01:53 am
If it's an official Intel NIC, it will have a YottaMark sticker
(https://www.intel.com/content/dam/support/us/en/images/network/sb/img/yotta_label.jpg)

According to Intel, if your NIC does not have a YottaMark, it is defective and should be returned. Without the YottaMark, you have no access to any warranty claims or support.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on September 20, 2017, 08:01:32 pm
I recently upgraded to 2.4.0-RC so I could give fq_codel a try.  Up to now I had been using the ALTQ FAIRQ scheduler together with codel managed queues to sort of emulate fq_codel.  I disabled my ALTQ shaping settings and I followed the steps in the original post.  After configuring everything I did a:

ipfw sched show

I could see fq_codel enabled but I could not see any traffic passing through.  However, I then also tried a speed test over at DSL Reports and could see fq_codel working, i.e. I had a A+ on the bufferbloat score.  This left me a little perplexed.  I could not see traffic passing through the fq_codel queues but yet it seemed to be working.  Is there a step I might have missed to make sure I can also see traffic passing through the queues?

Thanks in advance for your help.
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on September 21, 2017, 03:14:37 pm
Did you look at show when you know there was data flowing?

So for example... I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

Code: [Select]
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

Got A+ for quality, A for bufferbloat and A+ overall... This really is easy to implement too.. Be nice when can be fully done in the gui though.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on September 22, 2017, 05:42:09 pm
Did you look at show when you know there was data flowing?

So for example... I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

Code: [Select]
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

Got A+ for quality, A for bufferbloat and A+ overall... This really is easy to implement too.. Be nice when can be fully done in the gui though.

Thanks John, I really appreciate your help on this.  After I setup everything and ran a test at DSL Reports I actually did not continue to refresh ipfw sched show manually.  I naively thought that it would update automatically when traffic is passing through the queues.  That being said, is this the case though for the "Limiter Information" section under Diagnostics in the Web UI (i.e. does that section refresh automatically and show traffic passing through the queues)?

I think I'm going to try this out again.  However, I think my implementation needs to be a little different than was suggested in the original post.  I actually have more than 1 LAN interface on my pfSense box and each interface handles a different subnet.  In order to not limit the amount of bandwidth between the subnets, I don't think I should put the limiter queues on the default allow all rule on the subnets.   Instead, should I just setup a floating firewall rule that matches on WAN traffic?  If that makes sense, what other settings do I need configure (besides choosing the limiter queues at the bottom)?  For instance, what should be the source and destination?

Thanks again for all your help.


Title: Re: playing with fq_codel in 2.4
Post by: w0w on September 23, 2017, 03:13:40 am
Limiter info GUI do not show anything just by design of current fq_соdel implementation in both FreeBSD and pfSense, so it's normal that you don't see traffic there.

Title: Re: playing with fq_codel in 2.4
Post by: tman222 on September 24, 2017, 04:03:18 pm
So I tried setting up fq_codel again and confirm that it is working just fine (just had to refresh the "ipfw sched show" command as traffic was passing through the queues).  I did however, notice a lot of dropped packets while running a speed test.  I have symmetric gigabit connection and also had this issue with the ALTQ traffic shappers.  The solution was to increase the queue size sufficiently.   When using dummynet limiters is the default queue size adjusted under Advanced Options for the queue (that was created under the root limiter) or is it somewhere else?

Also, I'm still not quite sure how to configure my firewall rules so I don't accidentally limit traffic between local subnets, i.e. I only want to shape traffic internet bound traffic.  What would be the best way to do this (as using the default allow all rule on the LAN will impact subnet traffic too)?

Thanks again for your help, I really appreciate it.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 24, 2017, 07:23:20 pm
You have to pick your poison, dropped packets or large queue size.

Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

Which is better for you will depend on your network traffic.
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on September 25, 2017, 12:31:31 pm
@tman222

I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on September 25, 2017, 06:02:48 pm
You have to pick your poison, dropped packets or large queue size.

Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

Which is better for you will depend on your network traffic.

Thanks for this info - you are right.  I guess my point was that with very fast (high bandwidth) connections if the queue is too short, packets may drop unnecessarily (irrespective of any AQM such as Codel), which would limit the ability to realize (close to) full bandwidth on the link.  That being said, going too large on the queue size does increase the risk of bufflerbloat, but does this also impact the efficacy of AQM?  In other words if the queue is too large will Codel no longer work effectively?


@tman222

I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..

Thanks John, that makes perfect sense and is probably the best way to ensure that RFC 1918 traffic (or traffic on local subnets) does not get pushed into queues.   With a symmetric gigabit internet connection this is not necessarily a big deal as there are essentially no slow and fast links in the network topology, but in most other cases this configuration is very important so one does not unnecessarily limit bandwidth on local traffic.

------------------

I have now tried both fq_codel on dummynet and FAIRQ with Codel AQM on ALTQ on 2.4.0-RC and the results so far (at least for me) have been similar.  I'm curious if anyone has any ideas for additional testing to better demonstrate the superiority of one traffic shaping solution over the other?

Thanks again for all your help.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on September 25, 2017, 06:10:55 pm
I'm not sure one is better than the other.

It depends on what you're trying to do.

ALTQ gives you a lot of options and control.

Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

So really it depends on what you are trying to do and why.

As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

https://flent.org/index.html
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on September 25, 2017, 06:19:55 pm
I'm not sure one is better than the other.

It depends on what your trying to do.

ALTQ gives you a lot of options and control.

Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

So really it depends on what you are trying to do and why.

As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

https://flent.org/index.html

Thanks again for your help and explanation - I will check out that link.
Title: Re: playing with fq_codel in 2.4
Post by: EricE on September 30, 2017, 06:37:09 pm
I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs.  And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

Take a look at the message in thread: https://forum.pfsense.org/index.php?topic=99503.msg642533#msg642533

You should be able to tweak the config files from that thread to match your networks and do exactly what you want. 
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on September 30, 2017, 08:38:33 pm
fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

I hope they get the kinks worked out of Cake, but they keep trying to add the kitchen sink and has caused some back-and-forth regression.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 01, 2017, 08:15:04 am
fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.


Netgate should implement some sort of automatic bandwidth limiting (https://forum.pfsense.org/index.php?topic=137239.0), and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?


The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a  sub 5 minute configuration for even the least experienced user.

I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.


fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.
Title: Re: playing with fq_codel in 2.4
Post by: EricE on October 01, 2017, 02:32:40 pm
If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread (https://forum.pfsense.org/index.php?topic=137359.msg751240#msg751240).  I don't want to limit it, just make it a lower priority for any other traffic that happens, including starving out Backblaze entirely if there is ANY other traffic. But if there isn't other traffic, let Backblaze consume all the available bandwidth of the internet connection.

I would have thought not only would this be a simple thing to do, but it would also be fairly common - ha!  I have found precious few examples of how to do ANY traffic shaping to specific subnets - everything I have found so far is all around specific ports or services, which won't work in this instance since all the Backblaze traffic all over SSL.  They do provide a list of all the subnets their servers are on, so I have (one would think!) an easy way to classify their traffic
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 01, 2017, 02:38:23 pm
If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread (https://forum.pfsense.org/index.php?topic=137359.msg751240#msg751240).  I don't want to limit it, just make it a lower priority for any other traffic that happens - but if there isn't other traffic, consume all the available bandwidth.


For this, set your up and down limiters like normal.

Within each limit set two queues, lets say you call one normal and the other backblaze.

Set the subnet to match your network (probably /24). Down=destination up=source

If you wanted to prioritize normal traffic to have 90% bandwidth and backblaze to get 10% when the pipe is full. Then weight normal as 90 and backblaze as 10.

If the pipe is empty backblaze can still use it all.
Title: Re: playing with fq_codel in 2.4
Post by: EricE on October 01, 2017, 02:56:00 pm
I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions.  Thanks in advance.

For this, set your up and down limiters like normal.

So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?


Within each limit set two queues, lets say you call one normal and the other backblaze.

OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat.  Sounds like I need to be in the limiters instead - that might be where I went wrong.

Set the subnet to match your network (probably /24). Down=destination up=source

I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue.  If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right?  I have an Alias with all the subnets for the BackBlaze servers. 

I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section.  It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible.  But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

It sounds like I really need to play with the limiters instead.  Thanks again for the hints.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 01, 2017, 06:01:31 pm
I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions.  Thanks in advance.

For this, set your up and down limiters like normal.

So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?
Firewall/Traffic Shaper/Limiters

Within each limit set two queues, lets say you call one normal and the other backblaze.

OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat.  Sounds like I need to be in the limiters instead - that might be where I went wrong.
CODELQ is under the ALTQ system - which does certainly work, it's just a much more involved setup.

Set the subnet to match your network (probably /24). Down=destination up=source

I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue.  If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right?  I have an Alias with all the subnets for the BackBlaze servers.
When you make the queues in dummynet there will be an area to enter your subnet size so that it can share traffic between clients, set that to match which probably means setting it to /24, and the Download limiter is "destination" and Upload Limiter is "source".
You will also need to apply your queues to firewall rules.

In order to make sure everything is working and set correctly, I would temporarily set your up and down speeds to something way under your upload speed and set a unique number so that you will easily recognize it on speedtest.
What I mean by that is, if your normal download/up speeds are 40/10, then on dummynet set download to something like 4200Kbps and set upload to something like 650Kbps.
The only point of this is so that if you've accidentally reversed the upload and download queues in your firewall rules you will easily recognize that and fix it when you run a speedtest if you see upload at 4.2Mbps and download at 0.65Mbps. If you already know it's all setup correctly then just skip all that stuff.



I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section.  It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible.  But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

It sounds like I really need to play with the limiters instead.  Thanks again for the hints.

I'm sorry for the convoluted explanation. I'm not near a pfSense box I can access and won't be for awhile. Otherwise I would just give you a screenshot to explain this, it's very easy I'm just trying to explain this from memory of what the config GUI looks like.

Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 02, 2017, 01:33:22 am
Patch for Limiter Info page with schedulers information and refresh interval of 500ms

Code: [Select]
--- diag_limiter_info.php Wed Sep 07 00:26:47 2016
+++ diag_limiter_info.php Sun Oct 01 08:20:33 2017
@@ -40,5 +40,5 @@
  echo $text;
- $text = `/sbin/ipfw queue show`;
+ $text = `/sbin/ipfw sched show`;
  if ($text != "") {
- echo "\n\n" . gettext("Queues") . ":\n";
+ echo "\n\n" . gettext("Shedulers") . ":\n";
  echo $text;
@@ -72,3 +76,3 @@
  events.push(function() {
- setInterval('getlimiteractivity()', 2500);
+ setInterval('getlimiteractivity()', 500);
  getlimiteractivity();

Title: Re: playing with fq_codel in 2.4
Post by: Chrismallia on October 02, 2017, 12:30:15 pm
fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.


Netgate should implement some sort of automatic bandwidth limiting (https://forum.pfsense.org/index.php?topic=137239.0), and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?


The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a  sub 5 minute configuration for even the least experienced user.

I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.


fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.

Agreed with all you said. They should look into implementing it asap
Title: Re: playing with fq_codel in 2.4
Post by: superbree on October 02, 2017, 03:49:23 pm
Patch for Limiter Info page with schedulers information and refresh interval of 500ms

Code: [Select]
--- diag_limiter_info.php Wed Sep 07 00:26:47 2016
+++ diag_limiter_info.php Sun Oct 01 08:20:33 2017
@@ -40,5 +40,5 @@
  echo $text;
- $text = `/sbin/ipfw queue show`;
+ $text = `/sbin/ipfw sched show`;
  if ($text != "") {
- echo "\n\n" . gettext("Queues") . ":\n";
+ echo "\n\n" . gettext("Shedulers") . ":\n";
  echo $text;
@@ -72,3 +76,3 @@
  events.push(function() {
- setInterval('getlimiteractivity()', 2500);
+ setInterval('getlimiteractivity()', 500);
  getlimiteractivity();


Would love to try this patch out.  This will show fq_codel on the limiter info page?  Is there are kind soul who could explain how to implement this to the lay person?
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 02, 2017, 04:14:51 pm
There's a redmine feature request to get an automatic bandwidth limiter added to dummynet.

If anyone is interested and technically inclined please chime in!

Check out the links in my signature for more info.

https://redmine.pfsense.org/issues/7904
Title: Re: playing with fq_codel in 2.4
Post by: cplmayo on October 03, 2017, 12:08:17 pm
I finally got fq_codel limiters applied to just my WAN connection via floating rules.

From what I am seeing I think I like it better than using my vlan's interfaces. From what I am seeing in my own testing the jitter seems lower and I see fewer latency spikes on my upload bandwidth tests. Also since this is queuing all traffic on the WAN interface I feel like it is handling separate flows better than it did before.

I could be wrong and all of this is anecdotal or a placebo affect from all of my messing around with shappers and limiters.

If anyone is interested in trying it out the setup is fairly easy.

Firewall > Rules > Floating

*Add new rule

*Change "Action" from "Pass" to "Match"

*Select "WAN" in Interface

*Set "Direction" to "Out"

*Set "Protocol" to "any"

*Source to "any"

*Destination to "any"

Advanced settings

*Set Gateway (Cannot leave as default; you have to specifically set it to your configured gateway)

*Set In/Out (Because it is a floating rule and it is set to "Out" it gets a little confusing. It reverses In/Out ie In is for outgoing and Out is for incoming.)




Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on October 03, 2017, 01:25:37 pm
dslreports.com has a good bufferbloat test.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 03, 2017, 10:28:32 pm

Would love to try this patch out.  This will show fq_codel on the limiter info page?  Is there are kind soul who could explain how to implement this to the lay person?
You need "System patches" package.
Create new patch and apply it. See attachment.
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 12, 2017, 11:30:14 am
I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.

Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any.  Put a rule above to allow access to your other vlans without the pipe's applied.

These settings changed my bufferbloat tests on dslreports to A..

Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 12, 2017, 11:45:35 am
Why a /32 IPv4 mask?
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 12, 2017, 12:28:51 pm
Because that is what comes up in the gui when  this is the rules.limiter

[2.4.0-RELEASE][root@pfsense.local.lan]/root: cat /tmp/rules.limiter

pipe 1 config  bw 85Mb
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff
 

pipe 2 config  bw 11Mb
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff

Is something wrong there?  It was working great!!!
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 12, 2017, 01:39:22 pm
Haha, I don't know to be honest. I had mine set the same way until I noticed that, then set it to /24 to match my network (I'm IPv4 only). I haven't been on that network in awhile now but I don't remember noticing a difference. My config is otherwise pretty much the same as yours.

Maybe someone can chime in on whether that setting matters or not and exactly what it is doing?

I know that in some parts of traffic shaping GUI there are options presented that don't apply to all types of shaping.
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 12, 2017, 01:50:20 pm
The person that asked for the screenshot says its working great for him as well..

I just am not knowledgeable enough when it comes to shaping and limiters to know one way or the other either. I understand the basic principles is about all.  I just took the settings as given and applied them to my bandwidth at the time and yeah it drastically reduced the bufferbloat test without noticing any serious hit to the top end numbers on speedtest or during normal use.

But to be honest I had not really noticed any issues before that ;)  Other than the test showing me my bufferbloat was bad..

Looking forward to when I can apply it to my new 500/50 line when get new pfsense hardware.  I can tell you for sure that on the usg that currently stuck with that when you turn on their smart queues my download is limited to 80ish down vs the 530 I see on speedtest currently.  Seems to handle the upload ok but the download gets shit on..
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 12, 2017, 02:21:42 pm
Yikes, that's pretty limited!
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 13, 2017, 05:21:52 am
Which is why its not on ;)  When you turn on their queues you loose the hardware offload it seems.. So yeah speed takes a hit ;)
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 13, 2017, 07:36:59 am
And that is why I am thankful for pfSense!
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 13, 2017, 07:55:07 am
Oh believe me I will be back to pfsense as soon as get new hardware that can handle the speed.. The usg was a temp solution that was cheap enough to sneak through the budget committee (wife).. its was only a 100$ ;)

It can handle the speed in hardware offload.. But its feature set is so lacking.. Still running my pfsense vm for dhcp and dns since those features on usg need a huge amount of work to be viable in anything other than the most basic of home user networks.. And really just forget about ipv6 and or openvpn without manipulate of json files and having to reload them any time you reprovision the usg from the controller.. And the firewall rules are just nuts to setup on it as well..  I counting the days til I have pfsense back that is for sure ;)
Title: Re: playing with fq_codel in 2.4
Post by: sideout on October 15, 2017, 06:15:07 pm
I ran this on my router at my LAN party and it worked out great.   184 people with a 300mbit modem and 2 100mbit modems , made 2 download shapers and 1 upload shaper.

i made the system patches as well so it would apply after updates.
Title: Re: playing with fq_codel in 2.4
Post by: gsmornot on October 16, 2017, 03:59:38 pm
I should skip this since I don't know what I'm doing but still really curious to make it work. I have gigabit service and get D's and F's on buffer bloat.

I'm sure its in the post and I have indeed read though but still don't understand. What are the steps to enable this? I have 2.4 installed.

Looks like install patches package, run patch posted on page 8 which I was going to do until it said I could not remove this so I thought I better study a bit before I keep going. If you have the energy, please tell me what are the steps and I will follow them. Thanks.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 16, 2017, 04:27:42 pm
You don't have to install the patch.

Just set up limiters (look at Johns screenshots a few pages above this) then run the ipfw commands for fq_codel and add them to shellcmd.

Run a speed test and set your limiters to 95% of the speeds you get.

Now go to your firewall rules to pass traffic and in the advanced section just select the queues you just made.

That's it.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 16, 2017, 10:36:16 pm
You don't have to install the patch.

Just set up limiters (look at Johns screenshots a few pages above this) then run the ipfw commands for fq_codel and add them to shellcmd.

Run a speed test and set your limiters to 95% of the speeds you get.

Now go to your firewall rules to pass traffic and in the advanced section just select the queues you just made.

That's it.

I don't think it's that simple. If you don't override rules.limiter with own one like TS suggests by patching php code, then any firewall config or even WAN IP change that wants and would reload this file will destroy your manually configured fq_codel, until you manually run ipfw commands again or restart firewall to let shellcmd to do it. Am I wrong?
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on October 17, 2017, 04:58:57 am
No sorry it is that simple.. You do not need to make any files changes at all..  Just create the limiters and then put in the commands via shellcmd to put them in every time you reboot, etc.

Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 17, 2017, 07:20:23 am
Yeah, I just tried adding and deleting firewall rules then checking ipfw and it still has my fq_codel flows.

If there's some other action you're worried might remove fq_codel then just try doing that action then check ipfw after to see if fq_codel is still in place.


Code: [Select]
ipfw sched show
Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 17, 2017, 10:52:30 am
OK so may be quick start quide?

1. RTFM for FQ_CODEL http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt
2. Config limiters (pipes) via GUI.
3. View /tmp/rules.limiter

for example it will be

Code: [Select]

pipe 1 config  bw 280576Kb
queue 1 config pipe 1 mask src-ip6 /128 src-ip 0xffffffff
 

pipe 2 config  bw 280576Kb
queue 2 config pipe 2 mask dst-ip6 /128 dst-ip 0xffffffff

4. USE shellcmd package to recreate pipes with commands like

Code: [Select]
ipfw pipe flush

ipfw pipe 1 config  bw 280576Kb
ipfw sched 1 config pipe 1 type fq_codel target 7ms quantum 2000 flows 2048
ipfw queue 1 config pipe 1 mask src-ip6 /128 src-ip 0xffffffff
 
ipfw pipe 2 config  bw 280576Kb
ipfw sched 2 config pipe 2 type fq_codel target 7ms quantum 2000 flows 2048
ipfw queue 2 config pipe 2 mask dst-ip6 /128 dst-ip 0xffffffff


5. Add your limiters to firewall rules (IN/OUT pipes), this step can be any after step 2 actually.

Is it correct?
Maybe it's better to run script at startup? Just placing it into /usr/local/etc/rc.d? I found that using shellcmd is a little bit uncomfortable with multiple command lines at once, have I missed something?
Title: Re: playing with fq_codel in 2.4
Post by: JTravers on October 17, 2017, 11:49:30 am
Excuse my ignorance on this. I've just learned about and started using pfSense a couple weeks ago.

I have my limiters attached to my "Default allow LAN to any rule" in order to evenly split bandwidth to my LAN clients. And then fq_codel applied to those limiters. Seems to be working great for reducing bufferbloat, ensuring low latency for all clients, etc. Thanks for all the guidance in this thread!

Is there any benefit or harm to doing it that way vs. attaching the limiters to a floating rule as @johnpoz did?

Also, how does all this apply to OpenVPN clients (with pfSense as the server)? Would either setup also work with the OpenVPN clients, or is one setup better than the other?

Thanks for all your help!

Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 17, 2017, 12:19:46 pm
Floating rules vs interface rules won't make a difference. It will also work well on VPN clients. VPN traffic will always have higher latency relative to the same traffic not routed through a VPN. fq_codel can't fix that, but it will still work with fairly queuing the traffic and reducing bufferbloat.
Title: Re: playing with fq_codel in 2.4
Post by: gsmornot on October 17, 2017, 12:33:29 pm
I came back here to say thanks because it works well. I completed my setup differently than some of what has just been posted.

I setup limiters just as seen in the screenshots. (post 121)(upload, download, wan, lan)
I ran the single command for IPFW pipes. (ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2)
I installed shellcmd and added the single IPFW statement.
Modified the two stock LAN firewall rules (IPV4 and IPV6 advanced configuration) so that wan and lan would be used just as seen in the screenshots.
I restarted the firewall.

That is all I have done. Prior my buffer bloat was a D to F. Post I get an A each time. I may/may not be setup correctly but whatever it is works. I originally used the wizard for setup of traffic shaping which used HFSC and which gave @425 upload on my gigabit connection. This new setup gives @750. So, good for me.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 17, 2017, 01:51:28 pm
Definitely I am blind what screenshots are you all talking about? :D
Title: Re: playing with fq_codel in 2.4
Post by: gsmornot on October 17, 2017, 02:04:58 pm
Definitely I am blind what screenshots are you all talking about? :D
Reply 121 of this thread.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on October 17, 2017, 10:25:20 pm
Thanks.  :)
Title: Re: playing with fq_codel in 2.4
Post by: JTravers on October 18, 2017, 12:27:37 am
Floating rules vs interface rules won't make a difference. It will also work well on VPN clients. VPN traffic will always have higher latency relative to the same traffic not routed through a VPN. fq_codel can't fix that, but it will still work with fairly queuing the traffic and reducing bufferbloat.
I tested floating rules vs. lan rules and they both give excellent results. Latency results in bufferbloat tests seemed to be just slightly lower with the lan rules, but that's just splitting hairs.

I had very poor bufferbloat results when testing through my OpenVPN connection as a client connected to the OpenVPN server in pfSense. Is there any way to fix this? Should I be creating limiters to apply to the OpenVPN interface rules in the firewall and then selecting fq_codel on those limiters, as well?
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 18, 2017, 12:57:32 am
Yes you would need to apply limiters to your openvpn interface in order to queue your clients traffic. However, you can only fix your end, if the client is connecting to you via a poor connection then you can't get any better than the worst link.
Title: Re: playing with fq_codel in 2.4
Post by: JTravers on October 18, 2017, 02:04:32 pm
Yes you would need to apply limiters to your openvpn interface in order to queue your clients traffic. However, you can only fix your end, if the client is connecting to you via a poor connection then you can't get any better than the worst link.
Thanks, that makes sense.
Iíll try it out and see how much it helps.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on October 20, 2017, 07:47:31 pm
Finally got around to trying this again, and everything worked great!  John's screenshots in reply 121 are spot on and there is no need to edit any files if one uses shellcmd.

I actually recently changed to a 100/100 Fiber connection - here are results (using the DSL Reports speed test which has a nice Bufferbloat check):

Before (no shaping):
(http://www.dslreports.com/speedtest/23652529.png)

Using ALTQ FAIRQ + Codel Active Queue Management; 100Mbit Limit on Both WAN and LAN:
(http://www.dslreports.com/speedtest/23691758.png)

Using fq_codel and 100Mbit Limit on Both Upload and Download:
(http://www.dslreports.com/speedtest/23698259.png)

What's interesting to me here is that fq_codel appears to perform a bit better than the ALTQ emulation of fq_codel (using FAIRQ + Codel) - I find this very interesting.  Anyone have any thoughts as to why?

I also ran a more intense FLENT test on another system with fq_codel enabled and the results looked great as well (stable ping and stable download/upload over the course of the test).

Given the relatively little effort required to get this to work on pfSense, it's a fantastic way to improve the stability of a connection.


Title: Re: playing with fq_codel in 2.4
Post by: Nullity on October 21, 2017, 07:13:26 am
Finally got around to trying this again, and everything worked great!  John's screenshots in reply 121 are spot on and there is no need to edit any files if one uses shellcmd.

I actually recently changed to a 100/100 Fiber connection - here are results (using the DSL Reports speed test which has a nice Bufferbloat check):

Before (no shaping):
(http://www.dslreports.com/speedtest/23652529.png)

Using ALTQ FAIRQ + Codel Active Queue Management; 100Mbit Limit on Both WAN and LAN:
(http://www.dslreports.com/speedtest/23691758.png)

Using fq_codel and 100Mbit Limit on Both Upload and Download:
(http://www.dslreports.com/speedtest/23698259.png)

What's interesting to me here is that fq_codel appears to perform a bit better than the ALTQ emulation of fq_codel (using FAIRQ + Codel) - I find this very interesting.  Anyone have any thoughts as to why?

I also ran a more intense FLENT test on another system with fq_codel enabled and the results looked great as well (stable ping and stable download/upload over the course of the test).

Given the relatively little effort required to get this to work on pfSense, it's a fantastic way to improve the stability of a connection.

As I understand it, the biggest difference between FAIRQ + CoDel and fq_codel is that fq_codel individually applies codel to each per-flow pseudo-queue while FAIRQ + CoDel applies codel to the entire queue. There are also other subtle differences between codel and fq_codel, like the "fq" in fq_codel being a bit smarter than standard "fair queueing".

Either way, the 4ms difference you observed in best-case latency could just be a fluke.


Thanks for sharing the comparisons, btw.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on October 21, 2017, 07:28:17 am
I really don't get much difference. I was using OPNSense and fq_codel prior as it seemed to just work better for me.

With the new release, I changed back and just use HFSC queues with codel checked and some very basic rules to make sure my gaming traffic is first and my non important (downloads for media and other odd plex related download stuff) is limited. Works like a champ.

(https://i.imgur.com/iRKnDan.png)

Only thing for me always comes back to making sure my upload and download limits match close to reality what I expect out of my link so I use 940 down and 880 on Verizon's Gigabit FIOS with 1000 queue. No drops and no bufferbloat that I've been able to make happen.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on October 21, 2017, 02:24:10 pm
Thanks all for the feedback.  i do have a quick follow up question as I think that I may have misconfigured something:

I actually ended up creating two limiters, one at 100Mbit up/down, the other at 25Mbit up/down to use on a guest network.   Went through the same process and enabled fq_codel on the second set of limiters.  Applied the limiters inside the firewall rules on the guest network, but for some reason when I try to test out the configuration with a machine on the guest network I'm able to go faster than the limited speed of 25Mbit.  However, the interesting thing is that does not seem to be consistent - for instance:

1) When running a speedtest on speedtest.net I'm limited to just 25Mbit (as expected)
2) When running a speedtest on DSLReports I'm able to go well beyond 25Mbit (almost to full speed).

I haven't been able to try an iperf3 test yet unfortunately.  Could it be that something is misconfigured and that the 25Mbit limit is applied per flow vs. the queue as a whole?

Thanks in advance for any insight you might have.

P.S. Some thoughts regarding fq_codel vs. FAIRQ + Codel:  At least in my case, using fq_codel consistently results in a bufferbloat average (for both upload/download) under 10ms.  Using FAIRQ + Codel it often goes beyond that, but never higher than 15-20ms.  Ultimately, I suppose it's not really a big deal, but I found it interesting nonetheless.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on October 22, 2017, 12:12:31 pm
Thanks all for the feedback.  i do have a quick follow up question as I think that I may have misconfigured something:

I actually ended up creating two limiters, one at 100Mbit up/down, the other at 25Mbit up/down to use on a guest network.   Went through the same process and enabled fq_codel on the second set of limiters.  Applied the limiters inside the firewall rules on the guest network, but for some reason when I try to test out the configuration with a machine on the guest network I'm able to go faster than the limited speed of 25Mbit.  However, the interesting thing is that does not seem to be consistent - for instance:

1) When running a speedtest on speedtest.net I'm limited to just 25Mbit (as expected)
2) When running a speedtest on DSLReports I'm able to go well beyond 25Mbit (almost to full speed).

I haven't been able to try an iperf3 test yet unfortunately.  Could it be that something is misconfigured and that the 25Mbit limit is applied per flow vs. the queue as a whole?

Thanks in advance for any insight you might have.

P.S. Some thoughts regarding fq_codel vs. FAIRQ + Codel:  At least in my case, using fq_codel consistently results in a bufferbloat average (for both upload/download) under 10ms.  Using FAIRQ + Codel it often goes beyond that, but never higher than 15-20ms.  Ultimately, I suppose it's not really a big deal, but I found it interesting nonetheless.

Looks like the issue I was experiencing has to do with the Squid Proxy running on the guest network.  Similar to what was described here:

https://forum.pfsense.org/index.php?topic=132960.0

I'll go ahead and start a separate thread as I may need some help  configuring the proper rules to get this work.
Title: Re: playing with fq_codel in 2.4
Post by: Chrismallia on October 22, 2017, 01:04:26 pm
Implementing fq_codel improved the dsl reports TO A AND B  but USING hfsc and CODEL I get better results  ALL A+, I tried a linux distro with fq_codel got same A,B and sometime C  but again with Pfsense HFSC and codel I get all A+, so for me I am getting better results with HFSC and Codel
Title: Re: playing with fq_codel in 2.4
Post by: gsmornot on October 22, 2017, 01:43:47 pm
Implementing fq_codel improved the dsl reports TO A AND B  but USING hfsc and CODEL I get better results  ALL A+, I tried a linux distro with fq_codel got same A,B and sometime C  but again with Pfsense HFSC and codel I get all A+, so for me I am getting better results with HFSC and Codel
Did you configure manually or use the wizard? I used the wizard with HFSC selected and received better grades on dslreports but speed was much lower overall. The scores were better because the throttle was more aggressive. Would you be willing to share your config? Screenshots maybe. I would like to compare what I get using fq_codel as described in this thread.
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on October 23, 2017, 10:34:13 am
It's possible that HFSC+ALTQ gives better rate limiting characteristics compared to IPFW.
Title: Re: playing with fq_codel in 2.4
Post by: Chrismallia on October 23, 2017, 11:23:19 am
Implementing fq_codel improved the dsl reports TO A AND B  but USING hfsc and CODEL I get better results  ALL A+, I tried a linux distro with fq_codel got same A,B and sometime C  but again with Pfsense HFSC and codel I get all A+, so for me I am getting better results with HFSC and Codel
Did you configure manually or use the wizard? I used the wizard with HFSC selected and received better grades on dslreports but speed was much lower overall. The scores were better because the throttle was more aggressive. Would you be willing to share your config? Screenshots maybe. I would like to compare what I get using fq_codel as described in this thread.

Sure in dsl buffer bloat test I get half the speed but thats cos if it goes over that speed I get buffer bloat , but running a  normal speed test with same setup I get my full speed, so I only get half with dsl reports so for me HFSC and codel are  doing a fine Job but I am sure many more experts here can correct me. A other thing using ipfw limiters when using  the full upload speed it does not give example enough bandwidth plex remote users need, in hfsc it takes bandwidth from example the upload backup to the cloud and gives plex itS full 5mbps it needs
Title: Re: playing with fq_codel in 2.4
Post by: cwagz on October 23, 2017, 08:23:54 pm
I have fq_codel working on my system without issue.  I followed the screenshots from post #121.

Question:

If I apply the same lan / wan queues to the In / Out on my IPsec interface rule will bandwidth then be shared evenly between multiple IPsec clients?

I have several people that access server resources and it would be great if the bandwidth was shared evenly when everyone was trying to perform a get operation.

Thanks
Title: Re: playing with fq_codel in 2.4
Post by: chrcoluk on October 23, 2017, 08:56:55 pm
to the guys saying they only had to enable in cli and "nothing" else.

You didnt do this step?

Quote
Start with a recent 2.4 snapshot. Create two root limiters, Download and Upload, and put 95% your maximum values in bandwidth. Create two queues under each, say LAN and WAN. For LAN, selection destination addresses for mask and source addresses for WAN. Modify the default outgoing firewall rule to use WAN under "in" pipe and LAN under "out" pipe.

Also the limiter is surviving all filter reload's?
Title: Re: playing with fq_codel in 2.4
Post by: gsmornot on October 23, 2017, 09:09:49 pm
to the guys saying they only had to enable in cli and "nothing" else.

You didnt do this step?

Quote
Start with a recent 2.4 snapshot. Create two root limiters, Download and Upload, and put 95% your maximum values in bandwidth. Create two queues under each, say LAN and WAN. For LAN, selection destination addresses for mask and source addresses for WAN. Modify the default outgoing firewall rule to use WAN under "in" pipe and LAN under "out" pipe.

Also the limiter is surviving all filter reload's?
Yes I did that step. When I say I only used the command line I mean I did not install a patch of any kind. I use Shellcmd package to run the command line again each time my system boots.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on October 25, 2017, 09:15:26 am
Part of the challenge is trying to figure out what gives better performance is your ISP and what may or may not be going on with your local network.

I've got a 1Gb FIOS line and a pretty 'quiet' neighborhood so I tend to get a very consistent speed for up and download when I'm testing. Since it's not a pure 'lab' scenario, you can't really be sure of the variables in your testing.

I've noticed:
- FQ_Codel seems to have a bit less overhead than HFCS/Codel
- If I get my upload and download speeds set properly, I can straight A+s on any buffer bloat tests
- If I have multiple things going on or something not configured correctly, I tend to get problems
- If you are using a straight up limiter and equally sharing bandwidth across all LAN connections for an example, you won't see your max upload/download as you have it shared equally. To that point, in OPNSense, you would configure a limiter and "weight" your FW rules to prioritize what you wanted.

My rules would look like something like:

Code: [Select]
Limiters:
10000: 940.000 Mbit/s    0 ms burst 0
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active
10001: 880.000 Mbit/s    0 ms burst 0
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x0 0 buckets 0 active


Queues:
q10002  50 sl. 0 flows (1 buckets) sched 10001 weight 100 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN
q10003  50 sl. 0 flows (1 buckets) sched 10001 weight 10 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN
q10000  50 sl. 0 flows (1 buckets) sched 10000 weight 100 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN
q10001  50 sl. 0 flows (1 buckets) sched 10000 weight 10 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms NoECN

Which created some buckets and than weighted by my firewall rules.

I try to use the concept simple is better as I have very limited rules and only really lower my plex download traffic and prioritize my gaming traffic. Everything else just falls into the defaults.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 25, 2017, 10:20:33 am
To that point, in OPNSense, you would configure a limiter and "weight" your FW rules to prioritize what you wanted.

It works the same way in pfSense. I weight my guest Network to 10% of my bandwidth.
So if there is no lan traffic then guest can use all the bandwidth. When someone on lan starts using bandwidth then it will throttle guest all the way until they get down to 10% as necessary.
It's great, limits without wasting bandwidth. Of course you can set hard limits as well if you need to.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on October 25, 2017, 10:30:58 am
To that point, in OPNSense, you would configure a limiter and "weight" your FW rules to prioritize what you wanted.

It works the same way in pfSense. I weight my guest Network to 10% of my bandwidth.
So if there is no lan traffic then guest can use all the bandwidth. When someone on lan starts using bandwidth then it will throttle guest all the way until they get down to 10% as necessary.
It's great, limits without wasting bandwidth. Of course you can set hard limits as well if you need to.

Apologies as I don't mean to state the obvious so don't read into other than a statement, there is always traffic going on so if the plan is to share out across a LAN.

I always see some traffic going on which is specifically why I avoided equal sharing across my LAN and focused more on prioritizing hosts. All those Echos, ATVs and such are chatty :)
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 25, 2017, 02:07:45 pm
I don't think you're understanding.

Example:
On a 100/100 limiter.
LAN is weight 90, Guest is weight 10.

LAN is unused, background traffic only (let's say ~2Kbps) - Guest has up to 99998Kbps of bandwidth available.
In short, guest is free to use as much of the available bandwidth as they want less whatever LAN is using (Guest can only ever take away 10% of the total available bandwidth from LAN. Likewise, LAN can only ever take away 90% of the total available from Guest).

So, neither network will be limited at all until the pipe is full. The same principle is true for clients within each individual network.

Equal sharing does not mean that your bandwidth is automatically divided up between the number of clients on the network and each is given a hard limit.
I.e., 100Mbps limiter with 10 clients on the network automatically limits those clients to 10Mbps each all the time. That does not happen. That scenario would only ever happen if the pipe was full and ALL 10 clients were asking for >10Mbps simultaneously. The instant even one client backed off, that clients bandwidth would be distributed back out into the pool of available bandwidth.

Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on October 25, 2017, 02:11:16 pm
I don't think you're understanding.

Example:
On a 100/100 limiter.
LAN is weight 90, Guest is weight 10.

LAN is unused, background traffic only (let's say ~2Kbps) - Guest has up to 99998Kbps of bandwidth available.
In short, guest is free to use as much of the available bandwidth as they want less whatever LAN is using (Guest can only ever take away 10% of the total available bandwidth from LAN. Likewise, LAN can only ever take away 90% of the total available from Guest).

So, neither network will be limited at all until the pipe is full. The same principle is true for clients within each individual network.

I understood what you said. I used the term "equally sharing bandwidth across all LAN connections" in my post and you repeated my example of weighting, which I said I used.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 25, 2017, 02:16:43 pm
Ok I see.

My point was that you made it sound like only opensense offered this feature, which is incorrect.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on October 25, 2017, 02:27:39 pm
Ok I see.

My point was that you made it sound like only opensense offered this feature, which is incorrect.

Ah, ok as that wasn't my point. I just wanted to share that both the FQ-Codel and HFSC/Codel work well when configured right and my findings with quite a bit of testing was that FQ-Codel was more efficient but not by much and I had working results with both.
Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 25, 2017, 02:34:24 pm
My bad, my bad! It was a really late couple of nights haha.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on October 25, 2017, 06:08:28 pm
Hi guys,

Have been following the discussion on how to setup weights on the queues.  Wanted to go through an example to make sure I understand correctly:

Let's assume I have 3 subnets (LAN1 - 3) and one guest network.  I'd like to make sure that when under load, no LAN (or guest network) can hog all the bandwidth.

To set this up with limiters, I would:

Create an upload and download limiter and then create under each:

Download:  Create 4 queues (one for each subnet with weight 30, and one for the guest network with weight 10)
Upload:   Create 4 queues (one for each subnet with weight 30, and one for the guest network with weight 10).

Assuming I had a 100/100 connection, this would ensure that:

With no load, any of the subnets including guest network could consume up to 100Mbit.
Assuming the connection is maxed out, this will ensure the that LAN1 - 3 are limited to 30Mbit each, and the guest network is limited 10Mbit each.

In the situation where e.g. only LAN1 and LAN2 are trying to use all the bandwidth, how would it work (i.e. not traffic on LAN 3 and guest network)?  Depending on on which subnet started using the bandwidth first, is either able to go up to 70Mbit as the other is guaranteed at least 30Mbit? 

Thanks in advance for your help and explanation I really appreciate it.

Title: Re: playing with fq_codel in 2.4
Post by: belt9 on October 25, 2017, 07:44:03 pm
You got it, in the lan 1 and 2 only scenario it would go to 50 Mbps for each since they are weighted equally.

The speeds will certainly have transient periods of assymetric throughput but will balance out.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on October 26, 2017, 12:11:29 pm
You got it, in the lan 1 and 2 only scenario it would go to 50 Mbps for each since they are weighted equally.

The speeds will certainly have transient periods of assymetric throughput but will balance out.

Thanks!  I configured everything as described and was able to test it out by running a speed test on the three LAN's concurrently.  Was a nice to see speeds adjusting so that every LAN got its faire share as determined by the weights, yet if the other two LAN's are busy the third LAN could still use all the bandwidth.

Thanks again for the help - I think it's great how with proper traffic shaping one can really get the most out of a lower bandwidth connection, e.g. 50/50 or 75/75 will go a long way with proper shaping vs. spending extra $ to upgrade to more bandwidth to try to solve the problem.
Title: Re: playing with fq_codel in 2.4
Post by: darkcrucible on October 26, 2017, 04:56:02 pm
The problem I'm having with fq_codel is shaping OpenVPN. It's not clear how best to apply fq_codel to OpenVPN for my setup.
There are two options here.
1) Apply fq_codel to the WAN firewall rule for OpenVPN. This works well for site-to-site VPNs. If I send highly-compressable data, then the LZ4 compression works and I get a higher throughput. Uncompressable data is shaped normally and works well. This doesn't work well for a road-warrior connection. When the road-warrior accesses the Internet, that traffic is not handled by fq_codel. If it saturates the link then it's like not have fq_codel at all.

2) Apply fq_codel to the OpenVPN interface firewall rules. This breaks compression apparently as I couldn't get rates that exceeded the limiter speed.

With the old codelq applied to WAN, it didn't seem to matter what I did, as it would always do a pretty good job of keeping latency under control with/without OpenVPN, highly-compressable data, etc. fq_codel does a better job but having to apply it to every firewall rule is a bit of configuration tangle.

*Applying fq_codel to the WAN firewall rule for OpenVPN and sending highly-compressable data does introduce a lot of latency for me but still ok. It's much worse without fq_codel.
For reference:
Idle: 8ms, regular upstream saturation with fq_codel: 12-18ms, highly-compressable upstream saturation: 100ms, no fq_codel/codel upstream saturation: 1500ms.
Title: Re: playing with fq_codel in 2.4
Post by: deagle on October 28, 2017, 09:42:20 pm
I have two queues created under the "download" limiter and they show up in Limiter Info, but when I create the schedule only one queue gets added...

Does the command "ipfw sched 1 config pipe 1 type fq_codel" need to be modified to tell it to include all queues? I'm trying to add a lower weight to the guest network.

edit: one other observation, I followed the screenshots from post 121 but I needed to set the mask to match my subnets or multiple clients were clashing and still causing buffer bloat.
Title: Re: playing with fq_codel in 2.4
Post by: tibere86 on November 03, 2017, 02:45:50 pm
I really don't get much difference. I was using OPNSense and fq_codel prior as it seemed to just work better for me.

With the new release, I changed back and just use HFSC queues with codel checked and some very basic rules to make sure my gaming traffic is first and my non important (downloads for media and other odd plex related download stuff) is limited. Works like a champ.

(https://i.imgur.com/iRKnDan.png)

Only thing for me always comes back to making sure my upload and download limits match close to reality what I expect out of my link so I use 940 down and 880 on Verizon's Gigabit FIOS with 1000 queue. No drops and no bufferbloat that I've been able to make happen.
I have been using ALTQ FAIRQ + Codel Active Queue Management on my 150/150 link along with the queue set to 1024 in the child queue. My question is, does it make more sense to set the queue in the Codel child or the FAIRQ parent? Will I see a performance difference?
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 05, 2017, 08:01:21 am

I have been using ALTQ FAIRQ + Codel Active Queue Management on my 150/150 link along with the queue set to 1024 in the child queue. My question is, does it make more sense to set the queue in the Codel child or the FAIRQ parent? Will I see a performance difference?

I've stuck with HFSC and codel on the child queues with a queue limit of 1000. Works perfect for me. I use a very simplistic setup as I only have a high, default, low queue and gaming/voip is high and my download/sync traffic is low. Everything is just defaults.
Title: Re: playing with fq_codel in 2.4
Post by: Elhazgul on November 10, 2017, 03:40:12 am
Hello,
I try to configure the weight on the queus with fq_codel.
I have a LAN, a WAN and a VPN (ipsec), I want to put a higher weight on the VPN (weight 90) and on my WAN a lower weight (weight 10).
I use floating rules on the LAN interface to match my flows.

My limiters :
* Download : 1800kbit/s
 + Download_LOW : weight 10 (mask /24 with "destination addresses")
 + Download_HIGH : weight 90 (mask /24 with "destination addresses")
* Upload :  1800kbit/s
 + Upload_LOW : weight de 10 (mask /24 with "source addresses")
 + Upload_HIGH : weight de 90 (mask /24 with "source addresses")

Here is my /root/rules.limiter file:
Code: [Select]
pipe 1 config  bw 1800Kb
sched 1 config pipe 1 type fq_codel
queue 1 config pipe 1 weight 10 mask dst-ip6 /128 dst-ip 0xffffff00
queue 2 config pipe 1 weight 90 mask dst-ip6 /128 dst-ip 0xffffff00

pipe 2 config  bw 1800Kb
sched 2 config pipe 2 type fq_codel
queue 3 config pipe 2 weight 10 mask src-ip6 /128 src-ip 0xffffff00
queue 4 config pipe 2 weight 90 mask src-ip6 /128 src-ip 0xffffff00

It works without fq_codel but with fq_codel, limiters work but not weight on queues, my queues are not used.

When I do a "ipfw sched show", I do not understand why my queue 2 is present in the second limiter.

Code: [Select]
# ipfw sched show
00001:   1.800 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 2 1
00002:   1.800 Mbit/s    0 ms burst 0
[color=red]q00002  50 sl. 0 flows (256 buckets) sched 1 weight 90 lmax 0 pri 0 droptail[/color]
    mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 4 3

# ipfw queue show
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 10 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 1 weight 90 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
q00003  50 sl. 0 flows (256 buckets) sched 2 weight 10 lmax 0 pri 0 droptail
    mask:  0x00 0xffffff00/0x0000 -> 0x00000000/0x0000
q00004  50 sl. 0 flows (256 buckets) sched 2 weight 90 lmax 0 pri 0 droptail
    mask:  0x00 0xffffff00/0x0000 -> 0x00000000/0x0000

What is wrong ? Thanks for your help
Title: Re: playing with fq_codel in 2.4
Post by: Cardnyl on November 10, 2017, 09:55:53 am
I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 10, 2017, 09:59:23 am
I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

You should be able to. You are sacrificing some bandwidth cap so you don't get bloat. It's just finding that magical number for your connection. If you can't, your provider might be a little more sporadic than you think.
Title: Re: playing with fq_codel in 2.4
Post by: darkcrucible on November 10, 2017, 12:45:36 pm
I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.

In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-for
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on November 10, 2017, 05:29:25 pm
You could get some benefit FairQ.
Title: Re: playing with fq_codel in 2.4
Post by: strangegopher on November 11, 2017, 08:03:21 pm
This has to be the best thread on the forum right now. Helped me a lot.
Title: Re: playing with fq_codel in 2.4
Post by: tibere86 on November 14, 2017, 08:12:37 am
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 14, 2017, 09:14:35 am
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

Refresh your browser window and under Services->Shell Command
Title: Re: playing with fq_codel in 2.4
Post by: tibere86 on November 14, 2017, 10:18:00 am
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

Refresh your browser window and under Services->Shell Command
Doh. That's the one thing I didn't try. Many thanks!
Title: Re: playing with fq_codel in 2.4
Post by: Elhazgul on November 15, 2017, 09:52:46 am
Hi
It is possible to use queues (different weights) with fq_codel ?
Title: Re: playing with fq_codel in 2.4
Post by: darkcrucible on November 17, 2017, 12:24:32 am
So I have a pretty odd situation with fq_codel.
I created a floating rule:
Code: [Select]
Action - Match
Interface - WAN
Direction - out
Address Family - IPv4
Protocol - Any
In / Out pipe - lan wan

Now, when I do traceroute from a LAN machine, all the hops show up as the destination! If I disable this floating rule, then traceroute works normally. If it's enabled and I do traceroute from pfsense itself, that traceroute is fine. ICMP ping itself seems oddly unreliable because every first ping is lost.

I have a regular firewall rule to allow IPv4 ICMP to the WAN address but disabling/enabling this does nothing. I also do manual outbound NAT but just the regular rules.

What's causing this? As soon as I disable the out floating rule, traceroute works normally. The bad traceroute even has latency corresponding to what the hops would normally be. traceroute through a site-to-site VPN works normally (no floating rules for VPN).

IPv6 traceroute is unaffected.

Does anyone else who uses a floating rule like mine see this? Any known solutions?

A LAN machine example traceroute:
Code: [Select]
# traceroute -I forum.pfsense.org
traceroute to forum.pfsense.org (208.123.73.70), 30 hops max, 60 byte packets
 1  uac.localdomain (192.168.112.1)  0.163 ms  0.143 ms  0.138 ms
 2  * 208.123.73.70 (208.123.73.70)  14.600 ms  15.180 ms
 3  208.123.73.70 (208.123.73.70)  15.136 ms  15.379 ms  15.528 ms
 4  * * *
 5  208.123.73.70 (208.123.73.70)  16.811 ms  16.837 ms  16.848 ms
 6  208.123.73.70 (208.123.73.70)  17.816 ms  16.261 ms  17.627 ms
 7  208.123.73.70 (208.123.73.70)  17.644 ms  19.906 ms  20.173 ms
 8  208.123.73.70 (208.123.73.70)  20.262 ms  19.962 ms  19.813 ms
 9  208.123.73.70 (208.123.73.70)  19.594 ms  19.867 ms  19.898 ms
10  208.123.73.70 (208.123.73.70)  60.034 ms  59.998 ms  60.353 ms
11  208.123.73.70 (208.123.73.70)  59.594 ms  55.994 ms  55.201 ms
12  208.123.73.70 (208.123.73.70)  55.441 ms  55.469 ms  56.421 ms
13  208.123.73.70 (208.123.73.70)  55.350 ms  57.401 ms  57.401 ms
14  208.123.73.70 (208.123.73.70)  54.101 ms  55.283 ms  62.990 ms
15  208.123.73.70 (208.123.73.70)  62.392 ms  62.218 ms *
16  * * *
17  * * *
18  208.123.73.70 (208.123.73.70)  61.501 ms  62.231 ms  59.980 ms
19  208.123.73.70 (208.123.73.70)  67.374 ms  68.414 ms  68.897 ms
20  208.123.73.70 (208.123.73.70)  63.797 ms  74.896 ms  70.074 ms

The same traceroute from pfsense itself:
Code: [Select]
traceroute -I forum.pfsense.org
traceroute to forum.pfsense.org (208.123.73.70), 64 hops max, 48 byte packets
 1  173-228-88-1.dsl.dynamic.fusionbroadband.com (173.228.88.1)  14.262 ms  20.740 ms  17.791 ms
 2  gig1-29.cr1.lsatca11.sonic.net (70.36.243.77)  17.422 ms  21.650 ms  14.505 ms
 3  * * *
 4  50.ae4.gw.pao1.sonic.net (50.0.2.5)  15.070 ms  21.048 ms  19.681 ms
 5  ae6-102.cr1-pao1.ip4.gtt.net (69.22.130.85)  14.544 ms  13.726 ms  21.730 ms
 6  xe-8-1-6.cr0-sjc1.ip4.gtt.net (89.149.142.18)  16.210 ms  14.649 ms  14.841 ms
 7  as6461.ip4.gtt.net (216.221.158.110)  18.493 ms  18.531 ms  20.437 ms
 8  ae16.cr1.sjc2.us.zip.zayo.com (64.125.31.12)  22.334 ms  20.133 ms  20.252 ms
 9  ae27.cs1.sjc2.us.eth.zayo.com (64.125.30.230)  60.536 ms  79.274 ms  54.876 ms
10  ae2.cs1.lax112.us.eth.zayo.com (64.125.28.145)  55.359 ms  57.712 ms  63.770 ms
11  ae3.cs1.dfw2.us.eth.zayo.com (64.125.29.52)  59.466 ms  64.721 ms  63.456 ms
12  ae27.cr1.dfw2.us.zip.zayo.com (64.125.30.181)  60.663 ms  63.960 ms  59.337 ms
13  ae11.er1.dfw2.us.zip.zayo.com (64.125.20.66)  58.965 ms  64.998 ms  61.659 ms
14  ae8.er2.dfw2.us.zip.zayo.com (64.125.29.122)  64.904 ms  60.535 ms  59.003 ms
15  te-6-1-aus-core-11.zip.zayo.com (64.125.32.202)  65.484 ms  63.259 ms  63.947 ms
16  net64-20-229-170.static-customer.corenap.com (64.20.229.170)  65.048 ms  61.145 ms  59.658 ms
17  gw1.netgate.com (66.219.34.173)  67.752 ms  63.029 ms  63.543 ms
18  fw2.pfmechanics.com (208.123.73.4)  66.281 ms  68.793 ms  67.865 ms
19  208.123.73.70 (208.123.73.70)  66.834 ms  66.319 ms  66.047 ms
Title: Re: playing with fq_codel in 2.4
Post by: Nullity on November 17, 2017, 01:25:01 am
Hi
It is possible to use queues (different weights) with fq_codel ?

I'm unsure about exactly what you are asking but ALTQ (pfSense "queues") only supports codel and dummynet (pfSense "limiters") supports fq_codel.

You should probably create your own thread, including more details about your setup & intentions, if you need more help.
Title: Re: playing with fq_codel in 2.4
Post by: dimangelid on November 20, 2017, 11:36:28 am
I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.

Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any.  Put a rule above to allow access to your other vlans without the pipe's applied.

These settings changed my bufferbloat tests on dslreports to A..

Hello to all!

I have followed these instructions, together with the fq_codel trick on /etc/inc/shaper.inc and /root/rules.limiter . All my tests on dslreports are A or A+ . Also the download speed between my computers is shared evenly but with a serios issue:

When i download one file at one computer and one file at another computer, both from HTTP servers and a different server per computer, each computer takes the half speed which is what i want.

When any of the two computers start downloading a second file, from the same server or another, then it gets more speed and the second computer download speed slows down. If it starts a third download the other computers gets even less speed.

For example, when i download one file at each of the two computers, each one gets about 1.5 MB/s (my line maxes out at about 3 MB/s to 3.5 MB/s) . When any one starts a second download, the other computer gets about 1.2 MB/s and the other with the two downloads about 2.3 MB/s.

I am searching for a lot of days and have not found a solution. What i need to achieve, is to evenly share the speed between my computers no mater how many simultaneous downloads each of them runs.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 20, 2017, 12:15:23 pm
Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

Refresh your web browsers and it's under Services->Shellcmd
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 21, 2017, 01:49:12 pm
I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

I have the Shell Command to create the proper queue setup:

https://i.imgur.com/k08PJQZ.png (https://i.imgur.com/k08PJQZ.png)

I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

https://i.imgur.com/6JZTEXd.png (https://i.imgur.com/6JZTEXd.png)

https://i.imgur.com/6cDzTe5.png (https://i.imgur.com/6cDzTe5.png)

Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

https://i.imgur.com/HeMy45B.png (https://i.imgur.com/HeMy45B.png)

My rules examples are a bit big so I linked them a little different:

Default queue
http://i.imgur.com/CQDQGcf.png (https://imgur.com/CQDQGcf)

http://i.imgur.com/CQDQGcf.png (https://imgur.com/CQDQGcf)

Low priority rule
http://i.imgur.com/MDuvFFe.png (https://imgur.com/MDuvFFe)

http://i.imgur.com/CADTE77.pn (https://imgur.com/CADTE77)

For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

My speedtest results made me happy:

(http://www.dslreports.com/speedtest/25448513.png)

Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.

Title: Re: playing with fq_codel in 2.4
Post by: strangegopher on November 21, 2017, 07:46:48 pm
I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

I have the Shell Command to create the proper queue setup:

https://i.imgur.com/k08PJQZ.png (https://i.imgur.com/k08PJQZ.png)

I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

https://i.imgur.com/6JZTEXd.png (https://i.imgur.com/6JZTEXd.png)

https://i.imgur.com/6cDzTe5.png (https://i.imgur.com/6cDzTe5.png)

Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

https://i.imgur.com/HeMy45B.png (https://i.imgur.com/HeMy45B.png)

My rules examples are a bit big so I linked them a little different:

Default queue
http://i.imgur.com/CQDQGcf.png (https://imgur.com/CQDQGcf)

http://i.imgur.com/CQDQGcf.png (https://imgur.com/CQDQGcf)

Low priority rule
http://i.imgur.com/MDuvFFe.png (https://imgur.com/MDuvFFe)

http://i.imgur.com/CADTE77.pn (https://imgur.com/CADTE77)

For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

My speedtest results made me happy:

(http://www.dslreports.com/speedtest/25448513.png)

Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.

Are you sure you need both in and out direction floating rules? Mine seems to work with just outbound floating rule.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 21, 2017, 09:34:27 pm
I'm definitely not sure as I was trying to figure out a way to match my private lan traffic to drop a few hosts specifically into my 'low' queue.

I tried something a little different and deleted all my floating rules and made a few LAN rules and since they are PASS rules, they logged fine on my private IPs and looks to have dropped them in the proper queue.

I also changed my WAN NAT rules to have a proper queue for in/out as well.

I get the same results on speed tests and tried saturating my low traffic and my high/default worked fine.

Going to give that a try for a few days and see how it works and I might write that up instead as it just seems easier to me than the floating rules.
Title: Re: playing with fq_codel in 2.4
Post by: pete35 on November 24, 2017, 09:51:11 am
Hi,

just trying this on Pfsense 2.4.2 and 2.4.1 Release.

Even i install this with   ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

the resulting scheduler type is WF2Q+ ...

ipfw sched show

00001:  25.000 kbit/s    0 ms burst 0
 sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
   Children flowsets: 1
00002:  5.000 kbit/s    0 ms burst 0
 sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
   Children flowsets: 2

What im doing wrong ?

Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on November 24, 2017, 10:18:20 am
Hi,

just trying this on Pfsense 2.4.2 and 2.4.1 Release.

Even i install this with   ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

the resulting scheduler type is WF2Q+ ...

ipfw sched show

00001:  25.000 kbit/s    0 ms burst 0
 sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
   Children flowsets: 1
00002:  5.000 kbit/s    0 ms burst 0
 sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
   Children flowsets: 2

What im doing wrong ?

You should see an error on the command line when you type it in as Unix/Linux is case sensitive. Make sure to use lower case for FQ_Codel:

Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
Title: Re: playing with fq_codel in 2.4
Post by: pete35 on November 24, 2017, 11:24:01 am
Thank you, that worked!
Title: Re: playing with fq_codel in 2.4
Post by: darkcrucible on November 27, 2017, 06:40:30 pm
I setup fq_codel using floating rules on another system and the same IPv4 traceroute/ICMP problem I mentioned earlier occurs.

Anyone else who uses floating rules to match traffic for fq_codel, do you see IPv4 ICMP traceroute working properly?
Title: Re: playing with fq_codel in 2.4
Post by: Cardnyl on November 30, 2017, 05:09:39 pm
I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.

In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-for

Actually this article does help quite a bit. There was also mention of adjusting some of these tuneables by the OP; I just didn't realize how critical it was until now. Here is the list:

Code: [Select]

net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

The article you linked suggests the following for a slow connection like mine:
net.inet.ip.dummynet.fqcodel.limit < 1000
net.inet.ip.dummynet.fqcodel.quantum < 300
ECN OFF

Is the right way to make these adjustments by adding/modifying them in the GUI (System->Advanced->System Tunables)?
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on December 01, 2017, 10:35:20 am
Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.
Title: Re: playing with fq_codel in 2.4
Post by: Cardnyl on December 03, 2017, 02:45:50 pm
Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.

What is the correct way to set these targets?
Title: Re: playing with fq_codel in 2.4
Post by: Harvy66 on December 04, 2017, 11:29:40 am
I assume the "net.inet.ip.dummynet.fqcodel" settings you mentioned in your just prior post.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on December 04, 2017, 07:46:23 pm
Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

The reason for this is you don't want a single MTU sized packet to trip the drop logic.

What is the correct way to set these targets?

So I experimented with this a little bit, and changing the values below either in the System Tunables section or by editing loader.conf.local didn't seem to work because these fq_codel defaults don't appear to show up until fq_codel is actually loaded.  So if you are loading fq_codel using shellcmd this won't really work.

Code: [Select]
net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

You can make it work, but you would have to load fq_codel using ipfw so the defaults show up, change the default values to what you want them to be using sysctl and then load fq_codel again with the new values.

However, I think there is an easier way as you can change these values just by editing the ipfw command that loads fq_codel.

From a link posted earlier in this thread:

http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

From section 3, the syntax for loading fq_codel via ipfw is:

Code: [Select]
ipfw sched x config [...] type fq_codel [target t] [interval t] [ecn | noecn]
        [quantum n] [limit n] [flows n]
 
 where
    t is time in seconds (s), milliseconds (ms) or microseconds (us). The
        default interpretation is milliseconds.
    n is an integer number

The command to load fq_codel with the defaults (from earlier in the thread):

Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel


Using similar syntax as above, in your case all you would have to change is:

Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel target 15 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 15 noecn quantum 300 limit 600

This does the following:
Set target to 15ms (you'll need higher than default of 5ms given the low upload on your 15/1 connection)
Turn off ECN
Set fq_codel quantum to 300
Set queue size limit to 600 packets

(feel free to tweak these values as necessary)

Now of course, this all assumes that you have already created two limiters and queues underneath them.

You can issue the command above from the command line and make it persistent between reboots using shellcmd.

Hope this helps.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on December 04, 2017, 08:36:47 pm
I also have a question for everyone following this thread:

The command to enable fq_codel that has been shown in this thread to be:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

However, looking at the examples in 3.3 here:
http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

Doesn't the command have to be appended to include setting the the scheduler on the queue, i.e.:

Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw queue 1 config sched 1 && ipfw sched 2 config pipe 2 type fq_codel ipfw queue 2 config sched 2

The above assumes only two queues, one under each limiter.

Thanks in advance for the help.
Title: Re: playing with fq_codel in 2.4
Post by: w0w on December 04, 2017, 10:52:50 pm
I also have a question for everyone following this thread:

The command to enable fq_codel that has been shown in this thread to be:
Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

However, looking at the examples in 3.3 here:
http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

Doesn't the command have to be appended to include setting the the scheduler on the queue, i.e.:

Code: [Select]
ipfw sched 1 config pipe 1 type fq_codel && ipfw queue 1 config sched 1 && ipfw sched 2 config pipe 2 type fq_codel ipfw queue 2 config sched 2

The above assumes only two queues, one under each limiter.

Thanks in advance for the help.

I have used second, official variant, but have also found that first variant also works just fine if you have already configured limiters via GUI.
Title: Re: playing with fq_codel in 2.4
Post by: atrotter01 on December 05, 2017, 07:06:12 pm
I am trying to set this up on an SG-3100 with an asymmetrical gigabit connection.  I setup two queues and am configuring them as follows:

Code: [Select]
ipfw pipe 1 config  bw 920Mb
ipfw sched 1 config pipe 1 type fq_codel
ipfw queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

ipfw pipe 2 config  bw 40232Kb
ipfw sched 2 config pipe 2 type fq_codel
ipfw queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff

On the firewall side I am using a floating rule to apply the queues.  This all works, with the floating rule enabled I get A / A+ on the DSL Reports speed test, whereas without the rule I get a C at best for bufferbloat.  The issue I am having is with the rule enabled, I get at most ~650Mb/s bandwidth downstream, I have no problem hitting 940 with the rule disabled.  Am I running into a CPU limitation of the SG-3100? Is there anything I can try to tweak?
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on December 06, 2017, 10:19:17 am
I am trying to set this up on an SG-3100 with an asymmetrical gigabit connection.  I setup two queues and am configuring them as follows:

Code: [Select]
ipfw pipe 1 config  bw 920Mb
ipfw sched 1 config pipe 1 type fq_codel
ipfw queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

ipfw pipe 2 config  bw 40232Kb
ipfw sched 2 config pipe 2 type fq_codel
ipfw queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff

I'd love to have someone else chime in, but the way I "think" the masks work is that you are sharing bandwidth between source and destinations based on the "mask src/mask dst" rules going on. So that causes some dynamic queues to be setup and it'll share out bandwidth amongst those queues so assuming you have some other traffic going on even if it's only a little, it will limit the bandwidth out.

Maybe try without having the src and dst masks setup and just leave those blank and see how your results look. Also, you can check the CPU % or something? i'm not familiar with that on that particular to device to tell you how, but that would validate if you are having a CPU bottleneck instead.

On the firewall side I am using a floating rule to apply the queues.  This all works, with the floating rule enabled I get A / A+ on the DSL Reports speed test, whereas without the rule I get a C at best for bufferbloat.  The issue I am having is with the rule enabled, I get at most ~650Mb/s bandwidth downstream, I have no problem hitting 940 with the rule disabled.  Am I running into a CPU limitation of the SG-3100? Is there anything I can try to tweak?
Title: Re: playing with fq_codel in 2.4
Post by: atrotter01 on December 06, 2017, 03:57:12 pm
I tried leaving the masks off and that didn't seem to make a difference.  Then I tried disconnecting everything from the firewall except the laptop that I am using to test with, so that I could ensure nothing else was on the network.  That didn't make a difference either.  When the limiters are not enabled I can run a sustained 60 second download test and hit gig speeds 95% of the time, other than occasional dips.  When I enable the limiters I can't get over the 650ish mark.

I can run top from an SSH session and it looks to be pegging one of the cores both with and without fq_codel so I can't really tell one way or the other there.
Title: Re: playing with fq_codel in 2.4
Post by: johnpoz on December 07, 2017, 04:17:10 am
If you have symmetrical gig gig, why are you setting 40232Kb on pipe2

So 40mb?  Whats the amount of traffic for acks to hit 650 vs 940?  Have to do the math, etc.  But limiting your upload could have an effect on your max download.. Shoot 500mbps down would be like 8mbps or something up just in acks..

If your on gig/gig why would you want to limit your upload so much?

Not a qos guy, this was my real first test of any sort of qos in pfsense.. But I got asked to take a look, that is what jumps out at me to look odd..  In a few days when my 4860 is online I would be able to test your settings.  But not going to be ready until the weekend for sure..  But will be able try and duplicate the problem next week say.
 
Title: Re: playing with fq_codel in 2.4
Post by: atrotter01 on December 07, 2017, 05:30:08 am
John thanks, it's asymmetrical, not symmetrical.  :) The connection maxes out at about 940/42.
Title: Re: playing with fq_codel in 2.4
Post by: Animosity022 on December 07, 2017, 09:24:24 am
What's the CPU as well? I couldn't drive shaping until I did some upgrades. I'm overpowered now with an:

   Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz (4 cores)

But I wanted to future proof a bit. I can drive gig both directions at the same time without breaking my CPU atm.
Title: Re: playing with fq_codel in 2.4
Post by: atrotter01 on December 07, 2017, 05:45:38 pm
It's a dual core ARM v7 Cortex-A9 @ 1.6 GHz with NEON SIMD and FPU. I think I am either hitting some odd issue with traffic shaping on ARM architecture or a CPU limitation.
Title: Re: playing with fq_codel in 2.4
Post by: tman222 on December 08, 2017, 04:53:57 pm
It's a dual core ARM v7 Cortex-A9 @ 1.6 GHz with NEON SIMD and FPU. I think I am either hitting some odd issue with traffic shaping on ARM architecture or a CPU limitation.

1)  Do you run any type of IPS/IDS currently (e.g. Snort)?  If so, try disabling it temporarily to see if that helps.
2)  What's the output of "ipfw sched show"?  I think it would be good to doublecheck your fq_codel parameters.
3)  As alternative you can try ALTQ traffic shaping (instead of dummynet) to see if you experience similar limitations.  Instead of using limiters, try setting up the ALTQ FAIRQ scheduler and then enable Codel in the default queue.  You'll probably also need to increase the default queue size from 50 to something larger like 512 or 1024.  FAIRQ + Codel is not quite the same as fq_codel, but it it's similar (I have used both and the performance in my case was comparable).   If you try that, do you see a similar speed limitation that you currently see with dummynet/fq_codel?

Hope this helps.