The situation: Schools. Low budget. Over-worked tech staff in need of better tools. Old Cisco 2821 routers are maxing out at 400Mb/s between VLANs/subnets. L3 switches are way out of our budget and don't offer good tools for traffic monitoring, something we sorely need. Looking at pfSense on fairly modest hardware (Supermicro SYS-5015A-EHF-D525, Atom D525, 4GB of RAM, Intel NIs on board, and two additional Intel NIs on a PCI-E card).
These boxes seem to be hitting a performance wall at about 640 Mb/s. Running IPerf between VLANs/subnets, using all kinds and number of workstations yields an aggregate throughput no higher than about 640 Mb/s. Although this isn't horrible, I was expecting more than 1 Gb/s (using LAGG) because of the throughput figures I've read in several vendors literature (ie. here
My configuration is simple: LAN, OPT1, and OPT2 are all on different VLANs on top of a 3xLAGG. We're using Squidguard on top of Squid configured as a transparent proxy. Nothing but accept all rules on LAN, OPT1, and OPT2 for testing purposes. We are using aggregate links between switches, but the bulk of our testing is between workstations on a single switch.
One thing that concerns me is that I'm seeing the following printed out on the console after running IPerf tests:
interrupt storm detected on "irq258" throttling interrupt source
irq258 corresponds to the em2 interface (RX). Nothing else seems to be using it according to vmstat -i
. To my untrained eye, it looks like the D525 just can't keep up.
I'm going to mess with the configuration a bit (disable Squid and Squidguard) to see if I can get better numbers. We also have a much higher powered box that we can do testing on as well. I'll update this thread if I discover anything interesting.
In the meantime, I'd be open to suggestions as to how to boost performance above 1 Gb/s.