Netgate SG-1000 microFirewall

Author Topic: playing with fq_codel in 2.4  (Read 12662 times)

0 Members and 1 Guest are viewing this topic.

Offline qubit

  • Newbie
  • *
  • Posts: 20
  • Karma: +8/-0
    • View Profile
playing with fq_codel in 2.4
« on: March 04, 2017, 04:49:31 pm »
If you use limiters on 2.4 and check the system log you may have seen this pop up

Code: [Select]
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded

FQ_CODEL was added to FreeBSD in 11.0 in dummynet/ipfw, and since 2.4 is based on that we can enable it by hand without recompiling anything.

Note: This doesn't look like it will officially be in 2.4 via the GUI, and may need more testing. Since we're messing around with the command line, bad things may happen so use at your own risk.

Start with a recent 2.4 snapshot. Create two root limiters, Download and Upload, and put 95% your maximum values in bandwidth. Create two queues under each, say LAN and WAN. For LAN, selection destination addresses for mask and source addresses for WAN. Modify the default outgoing firewall rule to use WAN under "in" pipe and LAN under "out" pipe.

This generates /tmp/rules.limiter with something like the following:

Code: [Select]
pipe 1 config  bw 85Mb
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff


pipe 2 config  bw 9Mb
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff

and the firewall rule adds a "dnqueue( 2,1)" in /tmp/rules.debug for the outgoing lan rule.

Without messing with php we can manually change this to fq_codel and have it persist across reboots and ruleset reloads.

cp /tmp/rules.limiter to /root/rules.limiter

I edited /etc/inc/shaper.inc as follows:

Code: [Select]
4599c4599,4600
<               mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
---
>               #mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
>               mwexec("/sbin/ipfw /root/rules.limiter");

replace /root/rules.limiter with:

Code: [Select]
pipe 1 config  bw 85Mb
sched 1 config pipe 1 type fq_codel
queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff

pipe 2 config  bw 9Mb
sched 2 config pipe 2 type fq_codel
queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff

replace your bandwidth numbers with your own

Trigger a rule reload (disable, apply, reenable a rule) and kill states. Might want to run "ipfw pipe flush" before doing that. then verify in command line:

Code: [Select]
[2.4.0-BETA][admin@pfsense.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     1450  2175000 31 46500   0
00002:   9.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       21      840  0    0   0

[2.4.0-BETA][admin@pfsense.lan]/root: ipfw queue show
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

fq_codel is running, and we're passing traffic. Cool.


I tried using limiters a long time ago but had to stop due to some problems with dropped traffic probably relating to my hardware and igb. I then just had two interface shapers, WAN and LAN with CODELQ, set to 95% my upload and download. This stopped bufferbloat, but I noticed that most real traffic would actually be half of these values.

So far this has been far superior to the altq CODELQ with some of the following observations from the top of my head:

Downloads not randomly halved versus codel.
Twitch streams don't buffer when under heavy load such as steam
Two heavy bandwidth, multiple connection programs will share bandwidth evenly.
No more "sendto: No buffer space available" for unbound
Slight latency increase versus intermittent packet loss at load

Works just as good as cake in openwrt/lede from my limited home testing.

Some points:

1. Since I haven't been able to use plain limiters until now, this may just be better performance due to dummynet just limiting my bandwidth instead of fq_codel actually shaping. But it seems to perform better than plain limiters with reaching my bandwidth values versus the default WF2Q+.
2. Traffic isn't shown under queues, but 0.0.0.0/0 will show under ipfw sched, so I guess the traffic is still being shaped. I noticed this in the original dummynet aqm paper on the developers' website, so maybe it's by design.

Discuss if you've tried this or have any input. If you use limiters I'm interested if you can actually measure a difference since I'm coming from altq.

Offline Animosity022

  • Jr. Member
  • **
  • Posts: 50
  • Karma: +4/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #1 on: March 04, 2017, 06:38:07 pm »
ipfw -a list

That shows your rules and you can see what is matching to validate you have your rules correctly.

Offline Nullity

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +96/-9
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #2 on: March 04, 2017, 07:49:42 pm »
Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline qubit

  • Newbie
  • *
  • Posts: 20
  • Karma: +8/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #3 on: March 04, 2017, 08:11:23 pm »
ipfw -a list

That shows your rules and you can see what is matching to validate you have your rules correctly.
Code: [Select]
[2.4.0-BETA][admin@pfsense.lan]/root: ipfw -a list
ipfw: retrieving config failed: Protocol not available

dummynet is used with pf via "dnqueue" in pf rules which shows up in firewall rules via pfctl with limiters enabled.

Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.

Again probably related to my hardware. speedtests would show the full limited speeds on altq but most downloads wouldn't even reach that. Oddities like: dslreports would max out but fast.com would top out to about 40 megabits, as well as downloads via multiple browsers. Works fine without altq now. I think it was related to the igb driver as on my 2440 all networking would sometimes die and require a reboot if I disable the altq codel. Probably fixed recently by https://github.com/pfsense/FreeBSD-src/commit/42a5f2897e93d1e42833eac551c64c1373119ff9 but I haven't touched it in a while as this setup has been working great.
« Last Edit: March 04, 2017, 08:16:51 pm by qubit »

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #4 on: March 05, 2017, 12:50:24 am »
I got all three A+ on dslreports, even having active porn downloading on qbittorent. 8)

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #5 on: March 05, 2017, 02:51:12 am »
I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.
« Last Edit: March 05, 2017, 05:05:35 am by w0w »

Offline qubit

  • Newbie
  • *
  • Posts: 20
  • Karma: +8/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #6 on: March 05, 2017, 09:04:19 am »
I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.


These are the default sysctls which may need tweaking depending on traffic and bandwidth

Code: [Select]
net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

Technical details can be found here: http://caia.swin.edu.au/freebsd/aqm/papers.html

So far I found the default ok

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #7 on: March 07, 2017, 10:06:50 am »
I've played a bit, but I think that default are really OK.
Now I am using only IPFW FQ_CODEL shaper and disabled ALTQ, this gives me about +4Mbps on 300Mbps bandwidth if I compare with ALTQ shaper tested maximum.
So far, so good.

Offline shinzo

  • Jr. Member
  • **
  • Posts: 78
  • Karma: +4/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #8 on: March 08, 2017, 12:16:39 pm »
Thanks alot.  i have been looking for something like this for a while.  I plan to use it for a while to see how things go. ;D

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #9 on: March 08, 2017, 01:28:11 pm »
interesting i may try this out at some point thanks for sharing the information.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline shinzo

  • Jr. Member
  • **
  • Posts: 78
  • Karma: +4/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #10 on: March 09, 2017, 07:39:51 pm »
I already had the limiters setup.  i Was looking on how to make it easier and i came up with this. Instead of modifying anything i just input the ipfw command to enable fq_codel and worked.

Limiters:
00001:  30.000 Mbit/s    0 ms burst 0
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00002:   6.00 Mbit/s    0 ms burst 0
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active


Queues:
q00001  50 sl. 0 flows (256 buckets) sched 3 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 4 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

ipfw sched 1 config pipe 1 type fq_codel
ipfw sched 2 config pipe 2 type fq_codel
ipfw sched show

And Done.  While it wont survive a reboot i am sure i can set something up.
« Last Edit: October 16, 2017, 10:25:20 pm by shinzo »

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #11 on: March 10, 2017, 10:59:48 am »
Survivng reboot and update also is that what I needed.
Here is my patch (use System_patches package)
Code: [Select]
--- shaper.inc Mon Feb 20 18:14:04 2017
+++ shaper.inc Sun Mar 05 07:33:23 2017
@@ -4596,7 +4596,8 @@
  "net.inet.ip.dummynet.pipe_slot_limit" => $max_qlimit
  ));
  file_put_contents("{$g['tmp_path']}/rules.limiter", $dn_rules);
- mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+ #mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+ mwexec("/sbin/ipfw /root/rules.limiter");
  }
 }
 


Also, remember, you need to reboot firewall manually after update is completed or disable/enable rule where you have limiters used, like in OP first post.


Offline Nullity

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +96/-9
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #12 on: March 10, 2017, 02:39:53 pm »
Regarding the sysctl defaults, this link is likely the most official source for details, particularly the "Parameters" section: https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #13 on: April 02, 2017, 01:38:59 am »
One Sunday morning I have found that bufferbloat rating is B or even C  and no drops on my side. I've tried to play with bandwidth limiting and after changing it to twice smaller I got A rating again, looks like it's a problem on the ISP side. OK, I was thinking there is nothing to do, but why not to try to use delay instead of limiting bandwidth.
SO I changed limiter config to
pipe 1 config delay 0ms   for both pipes
And looks like this did the trick, now I have A+ bufferbloat and A or A+ Quality ratings.
Certainly, I need to do advanced tests before draw some conclusions, but it looks hopefully.
« Last Edit: April 02, 2017, 01:58:28 am by w0w »

Offline obrienmd

  • Newbie
  • *
  • Posts: 15
  • Karma: +2/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #14 on: April 04, 2017, 06:05:14 pm »
Can't WAIT for this to get into the UI.

FQ_codel's fair queuing is incredible, and HFSC + CODEL, FAIRQ + CODEL and CODELQ in pfSense can't provide multi-bucket fair queuing nearly as well.

I tested this using shellcmd so it will persist through reboots: "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" runs on reboot, with limiters and firewall pipes configured in the UI. It performs just as good as Linux's fq_codel that I have running on LEDE, IPFire and a few other boxes. pfSense getting fq_codel and wireguard would let me move entirely to pfSense / BSD on the networking side :)