pfSense Gold Subscription

Author Topic: playing with fq_codel in 2.4  (Read 12771 times)

0 Members and 1 Guest are viewing this topic.

Offline superbree

  • Newbie
  • *
  • Posts: 12
  • Karma: +1/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #60 on: July 11, 2017, 05:37:26 pm »
Thats really too bad.  We use PFsense primarily to "specify bandwidth limits per host." for a small ISP.

I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs.  And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

Is it possible to combine and use ALTQ and Dummynet at the same time?  Has anyone tried that or have a config example?

I guess I could use limiters on 2 PFsense boxes.  First one limiting each host to 5 Mbps using limiters with a destination/source mask.  And the second limiting the entire subnet to 100Mbs using limiters without a mask and changing the type from WF2Q+ to FQ_Codel by issuing the command "ipfw pipe 1 config bw 100Mb type fq_codel"

I hope thats not too confusing.  Anyone have a more eloquent way of trying this?

As always, thank you for any reply.

« Last Edit: July 11, 2017, 06:15:14 pm by superbree »

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #61 on: July 12, 2017, 02:09:03 am »
Yes it's possible, but  you will have some overheads and losses, you can try it at least, I think. Just set your per host limits on ALTQ shaper side and do your evenly shared FQ_CODEL enabled limiters exactly as TS described for you entire subnet.
I am sure it is possible to build ipfw only shaper model that works like you want it to work, but it would be complicated not only with pfSense and can cause some errors on pfSense.

Offline cplmayo

  • Jr. Member
  • **
  • Posts: 55
  • Karma: +1/-0
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #62 on: August 09, 2017, 06:40:52 pm »
Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.


My last speed test.


« Last Edit: August 11, 2017, 08:24:34 pm by cplmayo »

Offline meruem

  • Jr. Member
  • **
  • Posts: 69
  • Karma: +2/-1
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #63 on: August 25, 2017, 06:34:47 am »
Got this setup! Thank you so much! I have been waiting for a way to run FQ_Codel on my pfsense box for a while now. Granted it had to be hacked on but it worked!

Has anyone been running Suricata with 2.4 and fq_codel? Until I removed the suricata package my connection would keep dropping and I had lots of issues. So far so good.

I also had to enable Hardware checksum offloading and TCP Segmentation offloading. I may have to re-enable these at some point but at the moment everything is going well.


My last speed test.




enable re-enable or disable re-enable or enable re-disable or .. ?

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #64 on: August 27, 2017, 03:15:19 am »
fq_codel, the ZFS of AQMs, or nearly. Cake aims to be the "ZFS", but close enough.

This is very interesting.

Any chance someone(s) knowledgeable would be willing to put together a single post along the lines of this - https://forum.pfsense.org/index.php?topic=126597.0

Kind of like an fq_codel one-stop shop for the layman?

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #65 on: September 03, 2017, 09:33:19 am »
ok am finally testing this and got it working.

I had observed some iptv/vpn issues that seemed to only occur when my ingress altq config was active, so am now testing this configuration.  I have not yet tested if this is as effective as hsfc alt for keeping steam downloads in check, I had to set the dummynet limiter to 95% of downstream cap to even get a 6 threaded downstream test to stop causing packetloss, so not confident that will be enough for a 30+ stream steam download but will see.

How granular is this? can I e.g. route steam etc. all through it but at the same time applying a limit less than 95% for steam download whilst keeping things like youtube able to burst higher.  All on dummynet.  As I have a feeling I will need to drop this to at least 90% to manage steam but I consider that too low for lighter threaded stuff.
« Last Edit: September 03, 2017, 10:43:16 am by chrcoluk »
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #66 on: September 03, 2017, 11:11:13 am »
The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

It's a granular as firewall rules can be

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #67 on: September 03, 2017, 11:21:45 am »
The percentage of bandwidth you pay for is situationally dependent. If you always get 100% of what your isp says they'll give you then 95% works. If it dips to 94% of what you subscribe for and you set dummynet to 95% then dummynet can't do anything for you.

It's a granular as firewall rules can be

shaping is nowhere near that simple.

The reason steam is harder to shape is it opens so many threads.

I always get 100% from my isp but it doesnt mean 95% will always work well for all types of traffic.

Looking at dummynet configuration it looks like multiple specific rate limits cannot be set within one pipe however weighting can be applied so I can put steam downloads on a low weight and things like dns lookups and emails on a high weight, this is what I will look into on my config next. Thanks to the OP giving me a starting point. :)

AltQ on PFSense is incredibly granular but of course someone has put the effort into integrating it all into the GUI, and AltQ itself allows children in a queue to have their own limits set.

Already got some good results.

When I setup the dummynet config (basic as in the OP) I had a iptv stream running to my STB and I can see from my ping monitoring on my connection, the peak latency has plummeted, it was an almost steady increase in peak latency, now its spikes instead of constant and the spikes been generally much lower.

I will post back on how my steam downloads testing goes.
« Last Edit: September 03, 2017, 01:07:18 pm by chrcoluk »
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #68 on: September 03, 2017, 01:21:09 pm »
That's kind of the point of fq_codel, KISS. It is by design intended to be simple.

Now you might be trying to get it to do something it wasn't designed to do, in which case yes you will have to do some weird stuff - bit more likely you should just look elsewhere.


Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #69 on: September 03, 2017, 01:29:00 pm »
I am a fan of simple, if steam works well in the current config then the current config stays, the weighting is a fallback plan if it doesnt work well.

From what I understand all the weighting is dummynet side, it simply dynamically adjusts the throughput allowance of each thread based on the weight assigned, by default everything has the same weight.

All fully documented on the dummynet man page, I dont think its a non supported feature.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #70 on: September 03, 2017, 01:42:05 pm »
Yeah man page is very helpful.

You should just be able to weight steam as you desire, apply the queue to a rule that catches ports and protocols for steam and let fq_codel do the rest.

I think you'll be happy with it. I just set it up and it's working very well for me.

It's awesome for weighting a guest net and primary net!

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #71 on: September 04, 2017, 02:44:33 am »
I played some more.

First I misunderstood the man page, the weight flag does nothing on fq_codel, it only has an affect on another queuing type.

I tested steam and the result wasnt good, lots of packet loss during a steam download, the packet loss only goes close to 0% when the pipe size is below 40% of my line capacity, as I said steam is probably the most brutal traffic I have seen on my home connection.

HFSC can manage it at anything below 90%, however latency during saturation is vastly superior on fq_codel, packet loss is worse but latency better.

If I increase the queue slots to 500 (default 50), then packet loss almost stops at pipe size below 75%, with a small hit to latency.
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #72 on: September 04, 2017, 05:37:33 am »
It sounds like something else is going wrong on your box.

Weighting definitely applies to fq_codel, I've tested it on my own system and it matches weighted values every time.

I've also tested with both steam and flent rrul. No packet loss.

What is your line bandwidth? Are you trying to use dummynet and altq at the same time?

Offline chrcoluk

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +19/-50
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #73 on: September 04, 2017, 06:29:33 am »
altq is off during this testing to ensure no conflict.

Thanks for clarifying weighting has an affect I will try it on my test config as originally planned.  The reason I said it wasnt valid is because this is in the man page from dummynet section on ipfw.

Quote
     weight weight
             Specifies the weight to be used for flows matching this queue.
             The weight must be in the range 1..100, and defaults to 1.

     The following case-insensitive parameters can be configured for a
     scheduler:

     type {fifo | wf2q+ | rr | qfq}
             specifies the scheduling algorithm to use.
             fifo    is just a FIFO scheduler (which means that all packets
                     are stored in the same queue as they arrive to the
                     scheduler).  FIFO has O(1) per-packet time complexity,
                     with very low constants (estimate 60-80ns on a 2GHz
                     desktop machine) but gives no service guarantees.
             wf2q+   implements the WF2Q+ algorithm, which is a Weighted Fair
                     Queueing algorithm which permits flows to share bandwidth

     type {fifo | wf2q+ | rr | qfq}
             specifies the scheduling algorithm to use.
             fifo    is just a FIFO scheduler (which means that all packets
                     are stored in the same queue as they arrive to the
                     scheduler).  FIFO has O(1) per-packet time complexity,
                     with very low constants (estimate 60-80ns on a 2GHz
                     desktop machine) but gives no service guarantees.
             wf2q+   implements the WF2Q+ algorithm, which is a Weighted Fair
                     Queueing algorithm which permits flows to share bandwidth
                     according to their weights.  Note that weights are not
                     priorities; even a flow with a minuscule weight will
                     never starve.  WF2Q+ has O(log N) per-packet processing
                     cost, where N is the number of flows, and is the default
                     algorithm used by previous versions dummynet's queues.
             rr      implements the Deficit Round Robin algorithm, which has
                     O(1) processing costs (roughly, 100-150ns per packet) and
                     permits bandwidth allocation according to weights, but
                     with poor service guarantees.
             qfq     implements the QFQ algorithm, which is a very fast
                     variant of WF2Q+, with similar service guarantees and
                     O(1) processing costs (roughly, 200-250ns per packet).

This made me think its exclusive to wf2q+ however, fq_codel is omitted on the type section, so it didnt confirm that fq_codel has no weighting algorithm so I made the assumption.

My downstream throughput is around 71603kbit. Calculated after removing vdsl overheads, and also confirmed with experimentation when rate limiting to see when a rate limit starts having an affect.  The bandwidth is very consistent whether its on peak or off peak, if I remove shaping and download via steam it flatlines at the max speed with no dips.

Steam on average opens between 20 and 40 connections when downloading, most of these connections appear to be short lived making them very difficult to shape. Instead of downloading a large compressed file and uncompressing it steam seems to either download individual files on their own sessions or download files in fragments with the aim of maximising tcp sessions.  The problem is significantly reduced if I choose a server that has a high rtt such as in america (I am in the UK), swamping a connection with low RTT high bandwidth sessions will murder it.

I have been looking at the box configuration itself, the hardware tuning etc.  As I understand, if packets are batched together with things like interrupt moderation as well as a low kernel hertz timer, then shaping is less efficient as it cannot intervene at frequent enough intervals.   These are all things I am investigating as an ongoing process and I havent given up on this.

I have just reported back here how things went on the configuration suggested in the OP, with the only alternative config tried so far been to increase the queue depth.

So so far on my unit on my usage test patterns fq_codel via dummynet is much better for latency but worse at packet loss compared to HFSC on altq.

Also to add steam itself supports throttling the speeds, in that case I have tested on "unlimited", "7MB sec, which seems to be just below what my line can do" and "5MB sec", when steam throttles its not clean tho, it works by spiking to full speed, then pausing, then full speed again so it evens out that way.  If I set my pipe size lower and leave steam at unlimited its a clean reduction in speed flatlined at the pipe throughput rate.  throttling via steam vs the pipe size is more effective at higher speeds, but the pipe gets better when set very low.  I will report back after trying more stuff and welcome suggestions that are reasonable (trying entirely new kit is not reasonable in case you about to suggest it).
pfSense 2.4
Qotom Q355G4 or Braswell N3150 with Jetway mini pcie 2x intel i350 lan - 4 gig Kingston 1333 C11 DDR3L
 - 60 gig kingston ssdnow ssd - ISP Sky UK

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: playing with fq_codel in 2.4
« Reply #74 on: September 04, 2017, 07:15:46 am »
You could try applying a different shaping algorithm to steam and see if it works better.