pfSense Support Subscription

Author Topic: playing with fq_codel in 2.4  (Read 12731 times)

0 Members and 2 Guests are viewing this topic.

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #90 on: September 10, 2017, 11:25:38 am »
yep read that thread which is why I made that comment about the fake cards.  When I can be bothered I will take closer photos of the card both sides.

I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #91 on: September 10, 2017, 01:38:55 pm »
Most of the UEFI motherboards will eat the bootmgr.efi placed in EFI\Microsoft\Boot\bootmgr.efi on FAT32 formatted USB flash.
Here you can find how to obtain it https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Obtaining_UEFI_Shell
I've used binaries from https://github.com/tianocore/edk2/tree/master/ShellBinPkg
From what I've learned reading forums the first sign that this is a fake is the price and redrawn "Delta-like" ethernet transformers on it.
« Last Edit: September 10, 2017, 09:45:14 pm by w0w »

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #92 on: September 11, 2017, 09:55:06 am »
thanks for the help guys I do appreciate it.

I havent really got alternative kit to test on, but I managed to get pfsense running in esxi, after spending a couple of hours looking for a second nic (dual port server class intel nic I put in there both ports dead) and using vmx drivers in 2.4-RC is fine without polling.  So I am putting this down to a dodgy hardware setup and I think I am going to replace my unit.

But I am now going to leave this VM running so I can take the photos and play around with the firmware in the meantime without worrying about downtime.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline pf3000

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +10/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #93 on: September 11, 2017, 03:54:39 pm »
I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.
I described how to flash via EFI here https://forum.pfsense.org/index.php?topic=112968.msg629211#msg629211

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 377
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #94 on: September 11, 2017, 05:40:23 pm »
thanks is flashed, will post pics tomorrow but in a different thread as I am derailing this thread too much. I will edit this post with the link after I posted.

thread here https://forum.pfsense.org/index.php?topic=136561.new#new
« Last Edit: September 12, 2017, 01:48:15 pm by chrcoluk »
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #95 on: September 13, 2017, 10:52:53 pm »
« Last Edit: September 14, 2017, 11:04:01 am by w0w »

Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2195
  • Karma: +202/-12
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #96 on: September 14, 2017, 10:01:53 am »
If it's an official Intel NIC, it will have a YottaMark sticker


According to Intel, if your NIC does not have a YottaMark, it is defective and should be returned. Without the YottaMark, you have no access to any warranty claims or support.

Offline tman222

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +6/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #97 on: September 20, 2017, 08:01:32 pm »
I recently upgraded to 2.4.0-RC so I could give fq_codel a try.  Up to now I had been using the ALTQ FAIRQ scheduler together with codel managed queues to sort of emulate fq_codel.  I disabled my ALTQ shaping settings and I followed the steps in the original post.  After configuring everything I did a:

ipfw sched show

I could see fq_codel enabled but I could not see any traffic passing through.  However, I then also tried a speed test over at DSL Reports and could see fq_codel working, i.e. I had a A+ on the bufferbloat score.  This left me a little perplexed.  I could not see traffic passing through the fq_codel queues but yet it seemed to be working.  Is there a step I might have missed to make sure I can also see traffic passing through the queues?

Thanks in advance for your help.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14299
  • Karma: +1330/-193
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #98 on: September 21, 2017, 03:14:37 pm »
Did you look at show when you know there was data flowing?

So for example... I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

Code: [Select]
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

Got A+ for quality, A for bufferbloat and A+ overall... This really is easy to implement too.. Be nice when can be fully done in the gui though.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.2-RELEASE on VM esxi 6.5 (home)

Offline tman222

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +6/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #99 on: September 22, 2017, 05:42:09 pm »
Did you look at show when you know there was data flowing?

So for example... I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

Code: [Select]
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

Got A+ for quality, A for bufferbloat and A+ overall... This really is easy to implement too.. Be nice when can be fully done in the gui though.

Thanks John, I really appreciate your help on this.  After I setup everything and ran a test at DSL Reports I actually did not continue to refresh ipfw sched show manually.  I naively thought that it would update automatically when traffic is passing through the queues.  That being said, is this the case though for the "Limiter Information" section under Diagnostics in the Web UI (i.e. does that section refresh automatically and show traffic passing through the queues)?

I think I'm going to try this out again.  However, I think my implementation needs to be a little different than was suggested in the original post.  I actually have more than 1 LAN interface on my pfSense box and each interface handles a different subnet.  In order to not limit the amount of bandwidth between the subnets, I don't think I should put the limiter queues on the default allow all rule on the subnets.   Instead, should I just setup a floating firewall rule that matches on WAN traffic?  If that makes sense, what other settings do I need configure (besides choosing the limiter queues at the bottom)?  For instance, what should be the source and destination?

Thanks again for all your help.



Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #100 on: September 23, 2017, 03:13:40 am »
Limiter info GUI do not show anything just by design of current fq_соdel implementation in both FreeBSD and pfSense, so it's normal that you don't see traffic there.


Offline tman222

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +6/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #101 on: September 24, 2017, 04:03:18 pm »
So I tried setting up fq_codel again and confirm that it is working just fine (just had to refresh the "ipfw sched show" command as traffic was passing through the queues).  I did however, notice a lot of dropped packets while running a speed test.  I have symmetric gigabit connection and also had this issue with the ALTQ traffic shappers.  The solution was to increase the queue size sufficiently.   When using dummynet limiters is the default queue size adjusted under Advanced Options for the queue (that was created under the root limiter) or is it somewhere else?

Also, I'm still not quite sure how to configure my firewall rules so I don't accidentally limit traffic between local subnets, i.e. I only want to shape traffic internet bound traffic.  What would be the best way to do this (as using the default allow all rule on the LAN will impact subnet traffic too)?

Thanks again for your help, I really appreciate it.

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #102 on: September 24, 2017, 07:23:20 pm »
You have to pick your poison, dropped packets or large queue size.

Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

Which is better for you will depend on your network traffic.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14299
  • Karma: +1330/-193
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #103 on: September 25, 2017, 12:31:31 pm »
@tman222

I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.2-RELEASE on VM esxi 6.5 (home)

Offline tman222

  • Jr. Member
  • **
  • Posts: 60
  • Karma: +6/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #104 on: September 25, 2017, 06:02:48 pm »
You have to pick your poison, dropped packets or large queue size.

Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

Which is better for you will depend on your network traffic.

Thanks for this info - you are right.  I guess my point was that with very fast (high bandwidth) connections if the queue is too short, packets may drop unnecessarily (irrespective of any AQM such as Codel), which would limit the ability to realize (close to) full bandwidth on the link.  That being said, going too large on the queue size does increase the risk of bufflerbloat, but does this also impact the efficacy of AQM?  In other words if the queue is too large will Codel no longer work effectively?


@tman222

I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..

Thanks John, that makes perfect sense and is probably the best way to ensure that RFC 1918 traffic (or traffic on local subnets) does not get pushed into queues.   With a symmetric gigabit internet connection this is not necessarily a big deal as there are essentially no slow and fast links in the network topology, but in most other cases this configuration is very important so one does not unnecessarily limit bandwidth on local traffic.

------------------

I have now tried both fq_codel on dummynet and FAIRQ with Codel AQM on ALTQ on 2.4.0-RC and the results so far (at least for me) have been similar.  I'm curious if anyone has any ideas for additional testing to better demonstrate the superiority of one traffic shaping solution over the other?

Thanks again for all your help.