pfSense Support Subscription

Author Topic: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.  (Read 217 times)

0 Members and 1 Guest are viewing this topic.

Offline roveer

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +1/-0
    • View Profile
So I've been posting over the past few weeks about wanting to put together pfSense boxes on either end of two FIOS gigabit services and connect them via ipsec vpn.

I bought 2 Dell 7010 I7-3770 machines and put in Intel 4 port NIC's.  Installed pfSense 2.4.1 and configured for AES-NI operation.

The results are both good and bad.  Not exactly sure what's happening here, but I have a lot of data to share.  Hopefully we can figure something out.

Here are the iperf results of the WAN interfaces across the internet:



Very happy with that speed.  Pretty much full subscription rate.

Here is a traffic graph of the previous iperf run: 



Notice the traffic is only showing up on WAN, no LAN or IPsec.  This is wan port to wan port across the internet.

Next up is iperf results of the LAN interfaces across the ipsec vpn:



Here is the traffic graph of the vpn iperf run: 



You see traffic showing up on WAN and IPsec.  I'm very happy with these results 872 mbps.  On my previous non AES-NI setup I was only getting 250 mbps.

But here is where all the joy ends.  When I iperf two computers connected to networks on either side of the tunnel the results drop down hugely.  Not sure why.  If I iperf from machine to local WAN interface I get 900+ mbps so I know I have a fast enough computer and it's getting packets to the WAN quickly (both computers on both sides can iperf to their respective WAN interface at 900+ mbps.  But when I iperf between the two computers it drops all the way down to 274 mbps.  I can't for the life of me figure out what's going on.  Here it is:



So a little more information.

The first two iperf tests were done from shell's on the firewalls.  iperf commands are very simple:  iperf -B 192.168.0.1 -c 172.16.1.1 no other switches used  -B binds to a particular interface which is how I force it to use the ipsec or wan ports.

On the computers I open command prompts and do very simple iperf -c 172.16.1.117 commands.

Windows smb file copies are 35-38MBps : I was shooting for 70-90MBps



So any input or ideas are greatly appreciated and hopefully I can somehow improve these speeds otherwise I succeeded and failed at the same time.

Many thanks,

Roveer







« Last Edit: November 30, 2017, 09:18:05 pm by roveer »

Offline roveer

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +1/-0
    • View Profile
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #1 on: November 30, 2017, 09:21:45 pm »
I was looking at some other websites and came across a iperf syntax that I tried.  The result is windows pc at home to windows pc at work (across the vpn)

iperf command line was: iperf -c 172.16.1.117 -u -b 1000m

Results are pretty telling:  I'm not sure what these swithes do (-u says use UDP not TCP and I'm not understanding -b much at all) but I'm getting full line speed.  Hopefully this can tell us something which in turn I can tune on my firewalls.  If I lower the -b to 900 800 700 the speed starts to decrease.

------------------------------------------------------------
Client connecting to 172.16.1.117, UDP port 5001
Sending 1470 byte datagrams, IPG target: 11.76 us (kalman adjust)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.55 port 58746 connected with 172.16.1.117 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.11 GBytes   953 Mbits/sec
[  3] Sent 810345 datagrams

I just did the same test except -b 3000m from lan interface to lan interface (on each router), and got 1.5gbps throughput.  What's going on here.  How do I unleash this beast? (no graph) bad command line, it never sent any data across the network.

« Last Edit: December 01, 2017, 07:18:40 pm by roveer »

Offline roveer

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +1/-0
    • View Profile
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #2 on: December 01, 2017, 04:56:44 pm »
After reading a few more iperf thread I tried using the -P option which will open multiple streams to send data. So from my computer at work I did iperf -c 192.168.0.101 -P 3 (101 being my NAS on the other side of the vpn), and it fully saturated the line, 890 mbps.

So what's that telling me? My windows file copies are single stream and 280+ mbps is the most I'm going to get out of one stream? (as one post suggests). Are their copy programs that will do multiple streams? I've been searching and haven't come across anything.

My eventual need would be to be able to move data from the computer at work to the NAS on the other side of the vpn at line speeds. iperf just showed I can do it from machine to NAS, now I just have to find a program that can make it happen.

Roveer

Offline roveer

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +1/-0
    • View Profile
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #3 on: December 05, 2017, 07:18:39 pm »
At this point I've been having a conversation with myself on this topic but I'm determined to provide some valuable information to someone who will inevitably come across the same dilemma that I have.

So the past few nights I've been doing a lot of reading.  WAN Accelerators, alternate protocols etc.  Tonight I came across an article about transferring data across ipsec tunnels.  One of the items the author mentioned was different speeds using different protocols.  One of the protocols was http.   Hmm.  My NAS at home has a http front end and I remembered that it did some form of file transfer.  I gave it a shot, uploading a 17.7 gig rar archive in 3 minutes and 11 seconds.  Here's the tail end of the transfer:  As you can see, it achieved full line rate 100+ MBps



I see there are a number of windows programs out there allowing for http transfer.  Hopefully I can find a command line version or better yet some that might actually map a drive or at least allow me to send files to my NAS.  That would be super.  This could be just what I'm looking for to finally saturate my ipsec vpn for file transfer.  Sure beats a four thousand dollar WAN Accelerator.

Roveer

Offline mikeisfly

  • Sr. Member
  • ****
  • Posts: 472
  • Karma: +7/-0
    • View Profile
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #4 on: December 06, 2017, 05:53:54 am »
I too have Gigabit FiOS and have a site to site connection to another ISP which only has gigabit in the download direction the upload is much lower around 40 Mbps. Up until now I have been using OpenVPN because years ago it seemed to handle being behind a NAT much better. Both my machines are i5 with AES-NI support. I can't seem to get over 160 Mbps throughput so I am in the process of converting the link to IPSec to see if there is a speed increase. When I[m finish I will report my results back here. I don't expect to get the full line-rate but if I can get 50% of the link speed I will be happy.

Offline Jonb

  • Sr. Member
  • ****
  • Posts: 456
  • Karma: +0/-0
    • View Profile
    • Blue Sky Systems
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #5 on: December 06, 2017, 02:23:25 pm »
The issue you have here is SMB in general and how windows works with tcp coms. With TCP you have a window size which in principle is how much data to send before waiting for ACK.

So there is a formula which is based on latency there is a Max transmission speed to you achieve. So iperf will tell you what the line and your kit can do but this will not translate to smb throughput.

HTTP FTP etc does not exhibit these issues and if you want to transfer files use another method than smb.
Hosted desktops and servers with support without complication.
www.blueskysystems.co.uk

Offline mikeisfly

  • Sr. Member
  • ****
  • Posts: 472
  • Karma: +7/-0
    • View Profile
Re: ipsec over FIOS gigabit with AES-NI - Glory and flames, set me straight.
« Reply #6 on: December 07, 2017, 08:30:53 pm »
I tried to get SMB to work over IPsec but I was able to get anything higher than 500 Kb/s. So I switched back to Openvpn.

The issue you have here is SMB in general and how windows works with tcp coms. With TCP you have a window size which in principle is how much data to send before waiting for ACK.

So there is a formula which is based on latency there is a Max transmission speed to you achieve. So iperf will tell you what the line and your kit can do but this will not translate to smb throughput.

HTTP FTP etc does not exhibit these issues and if you want to transfer files use another method than smb.

Using Openvpn taking the information you gave me I tried various other protocols like SFTP and FTP and you were correct sir, I was able to get over 400Mb/s. I can see TCP ramp up to over 400 almost 500 Mbps but for what ever reason TCP backs off than starts to ramp up again. I know that is how TCP works but it seems to back off hard when it should probably keep ramping up until seq packets are dropped or come out of order and then have a gradual back off not a hard back off (See pic). I have my send and receive buffers set to 2MiB on both sides of the VPN Tunnel. I will report back if I can get more consistent speeds. I will post the results in the OpenVPN area and provide a link to it here if anyone is interested.