Netgate m1n1wall

Author Topic: Simple, effective traffic shaping (50)  (Read 9421 times)

0 Members and 1 Guest are viewing this topic.

Offline dusan

  • Full Member
  • ***
  • Posts: 198
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #15 on: November 07, 2006, 02:56:32 pm »
Yes. You can upload at nearly 512 kb/s and download at nearly 10 mb/s. But you cannot do both the jobs so fast at the same time. When you're downloading, downlink is used (of course), but uplink is used too. Similarly when you're uploading, not only uplink but also downlink is used.

So, given your queues are set as noted, you can only download at 7.2 mb/s and upload at 100 kb/s, simultaneously.



Btw, that calculation was merely idealistic, based on traffic patterns I've observed for MTU = 1500 bytes, which are

(0.015, 1, 0.04, 0.006) for bittorrent download, and
(1, 0.024, 0.008, 0.04) for rsync upload.

Please refer to my recent posts in "Traffic Shaping" box for what the figures mean and how to calculate.

For other MTU and other services (FTP, HTTP,...), traffic patterns would be different.

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #16 on: November 08, 2006, 12:10:31 pm »
I understand that I cannot do both at once to the absolute limit, because ACK packets are required.

However, Traffic Shaping should allow me to set upload speed to unlimited, and make sure ACK packets are prioritised and so get as much bandwidth as required to support whatever download speed I am getting. This does not appear to be happening.

For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets. Thus, I should be able to set an FTP upload off with no rate cap, and see it get at least 300k/sec with only a minimal (say, 50Kb/sec to account for ACKs etc) effect on downloads. This does not happen, downloads drop to around 250Kb/sec.

At the very least, the FTP upload should be the thing that is being slowed down by traffic shaping, and downloads should remain fast. That would be enough for me. However, not even that is possible.

Offline dusan

  • Full Member
  • ***
  • Posts: 198
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #17 on: November 08, 2006, 04:04:57 pm »
For NNTP downloads, at a full 9Mb/sec down I see approximately 100Kb/sec of ACK packets.


I've done a quick-and-dirty NNTP test and seen a notably different pattern.

Test config:

+ pfsense + pftop. All SSH packets go through qWANdef/qLANdef (I run SSH through an IPsec tunnel).
+ MS Windows 2000 + Outlook Express. Packets go through qOthersUpH, qOthersDownH, qWANack and qLANack.
+ sampling period: 10 second
+ number of samples: 50.


Observed data (on average):

qOthersUpH  267.48 B/s
qOthersDownH  10963.42 B/s
qWANack  318.78 B/s
qLANack  88.30 B/s

That observation corresponds to a traffic pattern of (0.024, 1, 0.029, 0.008). Thus, if I "extrapolate" to 10 mb/s downlink + 500 kb/s uplink, the figures would be:

qOthersDownH  8929 kb/s
qLANack  71 kb/s
total       9000 kb/s

qOthersUpH  214 kb/s
qWANack    259 kb/s
total        473 kb/s

and there remains very little bandwidth for other uploads.

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #18 on: November 08, 2006, 04:30:35 pm »
Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me... maybe the queue graphs are wrong?

Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.

At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.

For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.

I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading. Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.

Offline dusan

  • Full Member
  • ***
  • Posts: 198
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #19 on: November 08, 2006, 08:01:49 pm »
Dusan: I'm not sure why you get different results to me. I am going by what pfSense is telling me... maybe the queue graphs are wrong?

Also, what data is going through qOthersUpH? You don't make it clear if you are uploading anything. Unless you are reading many very small articles, the amount of data being uploaded, excluding ACKs, should be very small, probably less than 1k/sec for NNTP. That is what I see, almost no upload except ACKs.

Yes, I downloaded a large amount of articles. The packet size in qOthersDownH was about 800-900 bytes.

Sorry for not making clear that I didn't upload anything through qOthersUpH/qOthersDownH (except NNTP and a very little amount of ICMP and DNS). For NNTP, I tried qOthersUpL/qOthersDownL before, but there're no differences.


At the moment I am running Bittorrent, uploading at 11kB/sec and downloading at around 550kB/sec (note bytes). pfSense shows 150kb/sec for qwanacks (expected as BT uses many more connections than NNTP) and 117kb/sec for qP2Pup (88kb is data upload, presumably rest is BT protocol overhead). qlanacks is 6-8kb/sec and qP2Pdown is 5-6mb/sec. All other queues empty.

For that, the total upload is around 270kb/sec, of a total 512kb theoretical, lets say 480kb after overheads. pfSense traffic graph confirms this. There should be bandwidth left over.

Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.


I have found that the basic traffic shaping in m0n0wall seems to work just as well as pfSense, in terms of response times when browsing and ability to download while uploading.

m0n0wall can classify packets based on packet size. pfSense can't. So with pfSense one must find it hard to shape, say, bittorrent upload and download.


Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.

That's new to me. I've been believing that linkshare' m2 overrides bandwidth setting...

It seems you are satisfied with your config. Congratulation. :-)

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #20 on: November 09, 2006, 05:34:56 am »
Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.

Sorry, I still don't follow you... While I was writing my previous post, BT jumped up to a full 10Mb for a while (couple of minutes). During that time, I was still uploading at 11k/sec, and the total upload bandwidth in use was about 300kb/sec. So, it is possible in real world conditions.

Are you trying to say that my link should be saturated at 6mb/169kb total (including acks etc)? If that were so, I don't think it would be possible to download at 10mb/sec. I am currently downloading from NNTP with 24 connections at 9.4mb/sec, pfSense showing 280kb/sec for qwanacks and 1.8kb/sec for qwandef. Strange thing is, if I now try to upload at 120kb/sec download speed will drop. The total outgoing would only be 400kb/sec, and traffic shaping is supposed to ensure that ACKs and NNTP control traffic (in qwandef) get priority and so downloads remain unaffected.

Simply prioritising ACKs and non-P2P traffic actually works much better than the default pfSense config from the traffic shaping wizard. By tweaking qwanacks up to 65% bandwidth and 20% m2, download speed is maintained at 11kB upload speed.

That's new to me. I've been believing that linkshare' m2 overrides bandwidth setting...

Maybe it does, frankly I don't know... It was the only parameter not 0 by default, so I tweaked it. Having said that, it didn't seem to make much difference, only bandwidth did.

I'm still not happy with traffic shaping, and am still offering a bounty to anyone who can get it to work properly. To that end, let me restate a primary goal:

Allow unlimited uploading in, say, FTP or BT while still being able to download at over 8.5Mb/sec and without dropped connections for HTTP/NNTP/SMTP/POP3 etc (a few BT drops are OK).

Offline dusan

  • Full Member
  • ***
  • Posts: 198
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #21 on: November 09, 2006, 12:05:12 pm »
Based on my P2P download-only and upload-only patterns, I've calculated that your uplink is saturated at qP2Pdown=6 mb/s and qP2Pup=169 kb/s. (There are many other solutions, for example 497 kb/s upload-only or 9.1 mb/s download-only.) Thus for bittorrent, your upload-and-download pattern seems similar to mine.

Sorry, I still don't follow you... While I was writing my previous post, BT jumped up to a full 10Mb for a while (couple of minutes). During that time, I was still uploading at 11k/sec, and the total upload bandwidth in use was about 300kb/sec. So, it is possible in real world conditions.

Are you trying to say that my link should be saturated at 6mb/169kb total (including acks etc)? If that were so, I don't think it would be possible to download at 10mb/sec.

Only uplink is saturated, and 169kb/s was only qP2Pup, not including qWANack. This was only one among infinitely many possible combinations based on a download-only and a BT upload-only pattern. This wasn't accurate, as you didn't show your own patterns (upload-and-download data don't help very much). If you are interested in calculation, try to disable BT download to construct your own upload-only pattern, then disable BT upload for download-only one.

I am currently downloading from NNTP with 24 connections at 9.4mb/sec, pfSense showing 280kb/sec for qwanacks...

9.4 mb/s and 280 kb/s makes sense to me now. In a previous post, your figures said 9 mb/s and 100 kb/s that's why I made a quick test which didn't confirm them.



Now, let's back to the main problem:

...and 1.8kb/sec for qwandef. Strange thing is, if I now try to upload at 120kb/sec download speed will drop. The total outgoing would only be 400kb/sec, and traffic shaping is supposed to ensure that ACKs and NNTP control traffic (in qwandef) get priority and so downloads remain unaffected.

As you said, and I agree, that 1.8 kb/s are NNTP control packets, which are very important to maintain your download speed. If for some reason, it drops to say, 0.1 kb/s then your 9.4 mb/s download proportionally drops, ie. to 522 kb/s. Control packets have to be priotized, you're absolutely right.

Unfortunately, they do not get any priority over other packets sent through qWANdef, as packets in the same queue are sent in first-in-first-out order.

Maybe for NNTP outbound, instead of using qWANdef/qLANdef, you should try qWANack/qLANdef and see if it looks any better. I've never tried this before.
« Last Edit: November 09, 2006, 12:54:18 pm by dusan »

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #22 on: November 09, 2006, 05:33:57 pm »
*snip*

Thanks for that, we are on the same page now I think...

I'm afraid I can't make download only stats because BT clients limit download speed when you try to upload at less than a certain amount (usually 5kB/sec). I will try seeding (upload only) though, see what I can get.

Maybe for NNTP outbound, instead of using qWANdef/qLANdef, you should try qWANack/qLANdef and see if it looks any better. I've never tried this before.

Ah, so you think the speed problems could be down to the control info being squashed by the massive upload? That sounds like a good theory. All uploads are either ACKs or in qP2Pup, so should be lower priority than the control commands which are in qwandef. Maybe qwandef needs more bandwidth allocated to it? The default from the wizard seems to be totally useless.

Many thanks for your assistance so far. It looks like it might actually be possible to make traffic shaping work with your ideas on configuring it. I wish someone like you had been around earlier to help develop the wizard or write the shaper documentation.

Offline sullrich

  • Hero Member
  • *****
  • Posts: 5110
  • Karma: +3/-0
    • View Profile
    • pfSense
Re: Simple, effective traffic shaping (50)
« Reply #23 on: November 09, 2006, 08:19:14 pm »
Many thanks for your assistance so far. It looks like it might actually be possible to make traffic shaping work with your ideas on configuring it. I wish someone like you had been around earlier to help develop the wizard or write the shaper documentation.

Bleh!   Do you know how much time the authors of the Traffic shaper has spent?  It took 3 iterations to even come up with what we have now....
« Last Edit: November 09, 2006, 08:22:49 pm by sullrich »

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #24 on: November 10, 2006, 05:49:51 am »
Well, I'm not trying to be harsh... I am a computer programmer (C/assembler/Java mostly) so I know how hard it can be. Actually, it reminds me of writing drivers back in the Amiga days. It becomes very hard to write something good because there is so much variation, so many different configurations to consider. I think that is probably what has happened here.

There are some really weird things in the TF config that the wizard puts out. For example, making everything default to 1% bandwidth. Maybe it works okay for VOIP or whatever it was tested with, but if you look at the forums a lot of people want it for controlling P2P (especially on shared connections, which mine is).

Even people like Netgear seem to have trouble implementing TF in a way that works well for everyone. At least with pfSense, the ability to configure it is there. At first I was sceptical about the new shaper code, since it was performing worse than the old ALTQ stuff in m0n0wall. However, it looks like a solution is possible...

If I get time, I'll try writing up some notes for the wiki. Dusan, do you have a PayPal address?

Offline hoba

  • Administrator
  • Hero Member
  • *****
  • Posts: 5837
  • Karma: +3/-0
  • What was the problem to this solution again?
    • View Profile
    • pfSense
Re: Simple, effective traffic shaping (50)
« Reply #25 on: November 10, 2006, 06:26:07 am »
...since it was performing worse than the old ALTQ stuff in m0n0wall. However, it looks like a solution is possible...

m0n0wall never used ALTQ. It uses dummynet which is completely different.

Offline dusan

  • Full Member
  • ***
  • Posts: 198
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #26 on: November 10, 2006, 12:45:05 pm »
Mojo-chan, an (unverified) idea is not worth to pay for. Rather, I'm curious about how you  solve the problem. For me, pfSense is an excellent work. It does it job so well that I couldn't reproduce your problem conditions. (I've only experienced bittorrent download drop, but that's an other problem.)

Perhaps your bounty would be possibly good for the current "Traffic Shaper Changes" project that seems to meet your original requirements. But that's up to you. :-)

Hope to see your notes on wiki soon.

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #27 on: November 10, 2006, 02:15:43 pm »
Dusan, I'd be happy to pay you something for your hints. Although you did not provide a full solution like I asked, you at least gave me enough hints to make it work properly. Today, after much fiddling based on your ideas and methods, I managed to have an unlimited BT upload and download at over 8Mb/sec at the same time. Upload speed hovered around 300-330Kb/sec, with ACKs taking the total to around 480Kb/sec which is frankly amazing.

It's not perfect - you at at least a few high speed connections and not too many uploads (so, for example, eMule and a single HTTP download still isn't right) but generally it is excellent. Web browsing is also very responsive.

Offline mojo-chan

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +0/-0
    • View Profile
Re: Simple, effective traffic shaping (50)
« Reply #28 on: November 10, 2006, 02:55:49 pm »
PS. I added a page to the wiki, last link on the main page. Please take a look.