pfSense English Support > IPsec

ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.

(1/2) > >>

So I've been posting over the past few weeks about wanting to put together pfSense boxes on either end of two FIOS gigabit services and connect them via ipsec vpn.

I bought 2 Dell 7010 I7-3770 machines and put in Intel 4 port NIC's.  Installed pfSense 2.4.1 and configured for AES-NI operation.

The results are both good and bad.  Not exactly sure what's happening here, but I have a lot of data to share.  Hopefully we can figure something out.

Here are the iperf results of the WAN interfaces across the internet:

Very happy with that speed.  Pretty much full subscription rate.

Here is a traffic graph of the previous iperf run: 

Notice the traffic is only showing up on WAN, no LAN or IPsec.  This is wan port to wan port across the internet.

Next up is iperf results of the LAN interfaces across the ipsec vpn:

Here is the traffic graph of the vpn iperf run: 

You see traffic showing up on WAN and IPsec.  I'm very happy with these results 872 mbps.  On my previous non AES-NI setup I was only getting 250 mbps.

But here is where all the joy ends.  When I iperf two computers connected to networks on either side of the tunnel the results drop down hugely.  Not sure why.  If I iperf from machine to local WAN interface I get 900+ mbps so I know I have a fast enough computer and it's getting packets to the WAN quickly (both computers on both sides can iperf to their respective WAN interface at 900+ mbps.  But when I iperf between the two computers it drops all the way down to 274 mbps.  I can't for the life of me figure out what's going on.  Here it is:

So a little more information.

The first two iperf tests were done from shell's on the firewalls.  iperf commands are very simple:  iperf -B -c no other switches used  -B binds to a particular interface which is how I force it to use the ipsec or wan ports.

On the computers I open command prompts and do very simple iperf -c commands.

Windows smb file copies are 35-38MBps : I was shooting for 70-90MBps

So any input or ideas are greatly appreciated and hopefully I can somehow improve these speeds otherwise I succeeded and failed at the same time.

Many thanks,


I was looking at some other websites and came across a iperf syntax that I tried.  The result is windows pc at home to windows pc at work (across the vpn)

iperf command line was: iperf -c -u -b 1000m

Results are pretty telling:  I'm not sure what these swithes do (-u says use UDP not TCP and I'm not understanding -b much at all) but I'm getting full line speed.  Hopefully this can tell us something which in turn I can tune on my firewalls.  If I lower the -b to 900 800 700 the speed starts to decrease.

Client connecting to, UDP port 5001
Sending 1470 byte datagrams, IPG target: 11.76 us (kalman adjust)
UDP buffer size:  208 KByte (default)
[  3] local port 58746 connected with port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.11 GBytes   953 Mbits/sec
[  3] Sent 810345 datagrams

I just did the same test except -b 3000m from lan interface to lan interface (on each router), and got 1.5gbps throughput.  What's going on here.  How do I unleash this beast? (no graph) bad command line, it never sent any data across the network.

After reading a few more iperf thread I tried using the -P option which will open multiple streams to send data. So from my computer at work I did iperf -c -P 3 (101 being my NAS on the other side of the vpn), and it fully saturated the line, 890 mbps.

So what's that telling me? My windows file copies are single stream and 280+ mbps is the most I'm going to get out of one stream? (as one post suggests). Are their copy programs that will do multiple streams? I've been searching and haven't come across anything.

My eventual need would be to be able to move data from the computer at work to the NAS on the other side of the vpn at line speeds. iperf just showed I can do it from machine to NAS, now I just have to find a program that can make it happen.


At this point I've been having a conversation with myself on this topic but I'm determined to provide some valuable information to someone who will inevitably come across the same dilemma that I have.

So the past few nights I've been doing a lot of reading.  WAN Accelerators, alternate protocols etc.  Tonight I came across an article about transferring data across ipsec tunnels.  One of the items the author mentioned was different speeds using different protocols.  One of the protocols was http.   Hmm.  My NAS at home has a http front end and I remembered that it did some form of file transfer.  I gave it a shot, uploading a 17.7 gig rar archive in 3 minutes and 11 seconds.  Here's the tail end of the transfer:  As you can see, it achieved full line rate 100+ MBps

I see there are a number of windows programs out there allowing for http transfer.  Hopefully I can find a command line version or better yet some that might actually map a drive or at least allow me to send files to my NAS.  That would be super.  This could be just what I'm looking for to finally saturate my ipsec vpn for file transfer.  Sure beats a four thousand dollar WAN Accelerator.


I too have Gigabit FiOS and have a site to site connection to another ISP which only has gigabit in the download direction the upload is much lower around 40 Mbps. Up until now I have been using OpenVPN because years ago it seemed to handle being behind a NAT much better. Both my machines are i5 with AES-NI support. I can't seem to get over 160 Mbps throughput so I am in the process of converting the link to IPSec to see if there is a speed increase. When I[m finish I will report my results back here. I don't expect to get the full line-rate but if I can get 50% of the link speed I will be happy.


[0] Message Index

[#] Next page

Go to full version