OK, so I've enabled the fast forwarding on the hardware firewall and also tested with the following:
- Enable device polling = Enabled and Disabled
- Disable hardware checksum offload = Enabled and Disabled
- Disable hardware TCP segmentation offload = Enabled and Disabled
- Disable hardware large receive offload = Enabled and Disabled
- kern.ipc.nmbclusters="131072" and hw.igb.num_queues="1" in the '/boot/loader.conf' file
The maximum I was able to get has been 483 Mbits/sec.
I also tried updating to the 2.1 development build to see if perhaps some driver related issue had been resolve between builds. No improvement sadly.
To test my sanity I put the system back to a MikroTik 5.14 install and created firewall rules to pass traffic between two interfaces. Immediately throughput jumps to a massive 861 Mbits/sec @ 20% max CPU.
The discrepancy between those throughput and utilization numbers are insane. I really, and genuinely love pfSense (see posts on our website about pfSense for proof - https://www.xuridisa.com/blog
), but unless it can get up to these same levels on the exact same hardware you really have to start asking questions about efficiency, performance and scalability.
Anyone have any other suggestions that might be the silver bullet?