pfSense Gold Subscription

Author Topic: Floating bandwidth value for shaper (bufferbloat checker)  (Read 981 times)

0 Members and 1 Guest are viewing this topic.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Floating bandwidth value for shaper (bufferbloat checker)
« on: September 27, 2017, 09:14:28 pm »
If we are using traffic shaper to eliminate bufferbloat, then we need to limit bandwidth as well, 80-95% according to some sources, but here comes another problem, sometimes this limit needs to be adjusted depending on ISP channel load or day of time or anything else or you just do not to waste those bandwidth percents you paid for, so the main idea is to use scripting to change bandwidth value depending on ping value. We know current limit and current load, we can ping ISP gateway every n seconds, so actually we can collect statistics of current link saturation and change bandwidth value according to it, ex. from 50% to 100%.
Is it a good idea? What cons do you see?

Offline Nullity

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +96/-9
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #1 on: September 27, 2017, 11:27:30 pm »
If we are using traffic shaper to eliminate bufferbloat, then we need to limit bandwidth as well, 80-95% according to some sources, but here comes another problem, sometimes this limit needs to be adjusted depending on ISP channel load or day of time or anything else or you just do not to waste those bandwidth percents you paid for, so the main idea is to use scripting to change bandwidth value depending on ping value. We know current limit and current load, we can ping ISP gateway every n seconds, so actually we can collect statistics of current link saturation and change bandwidth value according to it, ex. from 50% to 100%.
Is it a good idea? What cons do you see?

It's more complex than that.

From an upload perspective, your CPE modem is the most likely culprit of bufferbloat. (LAN)

From a download perspective, it's likely your ISP's network. (WAN)


Beyond both of those, there are many internet nodes that may be the bandwidth bottleneck at any given time. Even if you "know" certain aspects of your LAN network or your ISP's network, there are still unknowns beyond that.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline richb-hanover

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #2 on: September 28, 2017, 06:27:43 am »
... We know current limit and current load, we can ping ISP gateway every n seconds, so actually we can collect statistics of current link saturation and change bandwidth value according to it, ex. from 50% to 100%.

It's more complex than that, in a different way from that mentioned by @nullity.

Sometimes, traffic is below the current setting/limit because there isn't any traffic. You wouldn't want to adjust down the limit in that case. So you do need to know the load.

But it isn't easy to know the current load. You'd need to devise a measurement of the "percentage of the bottleneck link that's being used for a long duration", say over 2-5 seconds. (Obviously, it's 100% used during a packet transmission...) I'm not aware of any measurements in a stock kernel that provide this.

That's why bufferbloat tests create their own artificial load to ensure that the link is saturated while measuring.
« Last Edit: September 28, 2017, 06:44:06 am by richb-hanover »

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #3 on: September 28, 2017, 10:12:21 am »
If we are using traffic shaper to eliminate bufferbloat, then we need to limit bandwidth as well, 80-95% according to some sources, but here comes another problem, sometimes this limit needs to be adjusted depending on ISP channel load or day of time or anything else or you just do not to waste those bandwidth percents you paid for, so the main idea is to use scripting to change bandwidth value depending on ping value. We know current limit and current load, we can ping ISP gateway every n seconds, so actually we can collect statistics of current link saturation and change bandwidth value according to it, ex. from 50% to 100%.
Is it a good idea? What cons do you see?

It's more complex than that.

From an upload perspective, your CPE modem is the most likely culprit of bufferbloat. (LAN)

From a download perspective, it's likely your ISP's network. (WAN)


Beyond both of those, there are many internet nodes that may be the bandwidth bottleneck at any given time. Even if you "know" certain aspects of your LAN network or your ISP's network, there are still unknowns beyond that.
Thank you for the answer!
Yes it can be complex, but it would be a little bit better than nothing?

I don't care about those "many internet nodes that may be the bandwidth bottleneck at any given time", just because it's not my problem. We are talking about bufferbloat and for many users the first point will be ISP router or modem or other equipment that uses too long queue lenght when bandwidth limit is achieved, as for me ISP router is stable enough, but sometimes, under heavy traffic it's not so responsible and I can see it just pinging some ISP IPs outside the router, this what bufferbloat actually is.
When I tune manually my bandwidth value down, then bufferbloat goes away, but I don't want to stay on this value, just because it's too low for the rest of time.
Yes, you right the logic must be complex, there is something to think about.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #4 on: September 28, 2017, 10:34:21 am »
... We know current limit and current load, we can ping ISP gateway every n seconds, so actually we can collect statistics of current link saturation and change bandwidth value according to it, ex. from 50% to 100%.

It's more complex than that, in a different way from that mentioned by @nullity.

Sometimes, traffic is below the current setting/limit because there isn't any traffic. You wouldn't want to adjust down the limit in that case. So you do need to know the load.

But it isn't easy to know the current load. You'd need to devise a measurement of the "percentage of the bottleneck link that's being used for a long duration", say over 2-5 seconds. (Obviously, it's 100% used during a packet transmission...) I'm not aware of any measurements in a stock kernel that provide this.

That's why bufferbloat tests create their own artificial load to ensure that the link is saturated while measuring.

It's a little bit different, the bufferbloat test creators don't really know your bandwidth limits and load, thats why they need to emulate artificial load on you and try to achieve those limits, causing ISP modem or any equipment to bloat.
In our case we already have all data, limits, current traffic, states and so on.
I think it is possible to detect some slowdowns even just monitoring queue lengths, something like CODEL already do...
The logic must be complex and not just lowering bandwidth but detect also is there some result after lowering to desired minimum value, if this result is missing, the bandwidth should be restored to previous maximum value and stay untouched for period of time this should prevent from lowering  bandwidth limit in case when bufferbloat is not possible to eliminate by limiting bandwidth on user side.

Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2209
  • Karma: +204/-12
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #5 on: September 28, 2017, 02:08:46 pm »
There are other distros with scripts that do exactly what the OP mentioned. Ping a known target and when the ping goes above some threshold, the script starts to lower the provisioned bandwidth in the shaper down to some configured floor.

I guess it works well enough to be better than nothing. Or do what I do and pay $50/m for a 150/150 dedicated fiber connection with guaranteed zero congestion within the ISPs network and the ISP has 6x more trunk bandwidth than their 95% percentile. /sigh I you move out here, you'd better get used to bar hopping or cow tipping for entertainment, but at least you'll have crazy awesome pings to Chicago game servers. Why ever leave home?

Offline Nullity

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +96/-9
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #6 on: September 28, 2017, 03:54:58 pm »
Gargoyle has a feature called "Active Congestion Controller". I've seen other scripts that either monitor bandwidth/ping or run occasional speedtests to determine optimal QoS bandwidth.

My connection is stable though, so I never tried any of them.


It'd be great to have dynamically adjusting QoS for unstable internet connections but I dunno if any of the mentioned solutions work well in practice.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #7 on: September 28, 2017, 08:39:54 pm »
For my connection the WAN bandwidth avaioable to me varies a lot depending on the time.

Just like the OP if I limit the value enough then bufferbloat isn't a problem, but I don't want to limit myself to 60% of my connection all the time just because it dips there occasionally.

For me at least, a script that would temporarily adjust bandwidth to a definable floor value based on a ping (or maybe pings to several different places? Would be a LOT better than nothing.

If someone writes that script for pfSense, please share on this thread!

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #8 on: September 29, 2017, 10:48:59 am »
Thanks Harvy66 and Nullity!
I will look at Gargoyle and it's "Active Congestion Controller".

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #9 on: September 29, 2017, 10:54:25 am »
https://github.com/ericpaulbishop/gargoyle/blob/master/package/qos-gargoyle/src/qosmon.c
https://dev.openwrt.org/attachment/ticket/8536/qosmon.patch (openwrt implementation)

I do think it's possible to use the same or similar logic on FreeBSD/pfSense, but we really need some help from professionals. Should I create a bounty for it or there are other plans at Netgate already?
« Last Edit: September 29, 2017, 10:24:29 pm by w0w »

Offline Nullity

  • Hero Member
  • *****
  • Posts: 973
  • Karma: +96/-9
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #10 on: September 30, 2017, 06:00:00 am »
https://github.com/ericpaulbishop/gargoyle/blob/master/package/qos-gargoyle/src/qosmon.c
https://dev.openwrt.org/attachment/ticket/8536/qosmon.patch (openwrt implementation)

I do think it's possible to use the same or similar logic on FreeBSD/pfSense, but we really need some help from professionals. Should I create a bounty for it or there are other plans at Netgate already?

I highly doubt that the Linux-centric code (I see references to iproute2 & tc) is useful to FreeBSD.

I think I saw someone on these forums say they were working on a pfSense script (python, sh, etc?) that accomplishes practically what qosmon does. I dunno if they ever released anything... the post was also a few years old IIRC.


Really, it should not be too hard to script something that monitors a ping command and/or bandwidth.


I'll try to remember where I put another Linux-based QoS (shell) script that dynamically adjusted bandwidth based on bandwidth/ping, since it's likely more portable and it would also be useful to see their approach. I got it from some GPL request.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #11 on: September 30, 2017, 01:54:42 pm »
Yep, I did not mention that we need to port some linux code, but at least we have working piece of code that can be used as logic sample in a new bash script or better PHP code... Just dreaming  8)
Thanks Nullity, I hope you'll find something good enough!

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #12 on: October 01, 2017, 09:35:56 am »

Offline belt9

  • Full Member
  • ***
  • Posts: 233
  • Karma: +24/-6
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #13 on: October 01, 2017, 05:38:16 pm »
The feature request was accepted but needs your input!

Quote from: Jim Pingle
It's possible in some specific circumstances, but I don't see one of those being a way that would work properly with dummynet (limiters). It also still requires you to probe an external source at a specific destination (which would always have to respond as fast as your circuit allows), gets more difficult at higher bandwidth amounts (Takes ~16 minutes to probe minimum on a high speed link), and assumes all latency is from throughput saturation, which isn't necessarily true.

If someone comes up with a viable implementation, I'd love it see it work, because it would be useful. But like many ideas, it's simple to think of but difficult to implement.

So please provide your input on how to implement at the redmine feature request:
https://redmine.pfsense.org/issues/7904

Offline w0w

  • Sr. Member
  • ****
  • Posts: 534
  • Karma: +30/-6
  • kernel panic attack
    • View Profile
Re: Floating bandwidth value for shaper (bufferbloat checker)
« Reply #14 on: October 02, 2017, 12:28:24 am »
This just sounds like it will be never coded by Netgate until they have ALREADY working source for FreeBSD. Thats sounds bad, but there is no way to affect this, only invest money or doing it by yourself and bring them working solution to implement.
Some interesting thing is TEACUP experiments http://caia.swin.edu.au/tools/teacup/
http://caia.swin.edu.au/reports/161107A/CAIA-TR-161107A.pdf
Personally I am ready to invest some money into QoS monitor for FreeBSD (or anything with the same function) and also I am ready to pay for GUI version of dummynet FQ-Codel, the question remains, that Netgate just do not have resources for low priority tasks and new features, that are not needed by most of their customers. 
« Last Edit: October 02, 2017, 12:34:13 am by w0w »