I've made a bit of progress. It's not strictly pfSense related, but I'm hoping we can use this to bridge the gap.
I borrowed an Adtran Netvanta 1335 from work. It's basically a router with some Layer 3 switching capabilities. There's 24 10/100 ports and 2 gig ports.
Right off the bat, this old POS looks to be hardware limited to ~120 Mbit/sec even on the gig ports. I knew they were running out of gas (which is why we're replacing them at work) but I thought it was a CPU/ # of firewall sessions problem. I guess it's all of the above.
Also, I have no IPv6 enabled. I'm not even sure it's supported on this platform. No IPv6 = IPTV on this system.
Anyway, I fixed the upload problem. Once I got basic connectivity established, I was pulling 120 down, and only 10 up. Which is what Atlantis and I were seeing on pfSense.
After I got a QoS policy in place, upload improved to match the download rates. I was getting 120 both ways. I did verify that the gig ports were auto-negotiating at the correct rate and not accumulating errors.
So here's what I'm hoping for. Adtran configs are very similar to Cisco. It's my hope that some of the more knowledgeable folks will read what I did with this Adtran, and then chime in with how we might be able to implement a similar config on pfSense.
I'll explain the relevant parts of the config, and I'll upload the entire thing as an attachment. The only changes I've made is to remove the password hashes. Everything else is line for line identical to my running config. Please don't critique it too hard. It's just something I banged together in a few minutes for testing purposes.
So, to get this working:
Create interface VLAN 2, and set it to DHCP.
*Put interface gigabit-switchport 0/1 into VLAN trunking mode. Verify that VLAN 2 obtains an IP address and you can ping out.
*Turn up interface gigabit-switchport 0/2 and let it go on the default VLAN. Add the necessary policies to allow outbound NAT. Verify access.
Create access list GF-dhcp
*Set the ACL to match both TCP and UDP port 67. Probably only needs UDP. Whatever.
Create access list GF-default
*Set this as a permit IP any <-> any
Create QoS policy GF-QoS
* On the first policy term, match against the GF-dhcp ACL
* When packets match the ACL, set the VLAN priority / 802.1p / CoS bit 2
* On the second policy term, match against GF-default
* This is the catch-all rule, which applies VLAN priority / 802.1p / CoS bit 3
* I wanted to do a ACL and QoS term for IGMP, but I couldn't figure out how to enable that. Maybe later.
Apply the QoS policy in the outbound direction on VLAN 2.
All traffic exiting VLAN 2 towards the internet will have the .1p / CoS bits set, and upload speeds should see a dramatic improvement.
Anyone want to take a crack at interpreting this into a pfSense config?