Netgate m1n1wall

Author Topic: OpenVPN performance - Huge Improvement  (Read 29190 times)

0 Members and 2 Guests are viewing this topic.

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
OpenVPN performance - Huge Improvement
« on: March 23, 2012, 11:40:26 am »
Hi all,

I'm fighting with bad openVPN performance for a while now. See this post over at the openVPN forum: https://forums.openvpn.net/post21446.html

During my testing, I noticed a huge performance jump when setting net.inet.ip.fastforwarding = 1, followed by a reboot.

This setting can be found under System|Advanced|System Tunables and defaults to 0.
« Last Edit: March 24, 2012, 05:31:02 am by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Updated: OpenVPN performance
« Reply #1 on: March 24, 2012, 05:17:42 am »
I've done some iperf testing to validate my assertions that the setting vastly improves openVPN Server performance on pfSense.

Network and Test Layout:

Code: [Select]
IPerf:    [Client]-----------------------------------------------------------------------------------[Server]
VPN:      [OpenVPN Client]---------------------------------------------------------------------------[OpenVPN Server]---------------------[Internet]
Physical: [Win32 Netbook]-----802.11n lite WLAN---[AP]-----100baseT-----[GigE Switch]------GigE------[pfSense 2.0.1]-------GigE-----------[CISCO WAN]

The pfSense box runs in a ESX 5 VM (2 vCPU, 512MB RAM, E1000 NICs) on a Core2Duo @ 2.66GHz/8GB RAM machine.

Test Results:

Code: [Select]
Test Run A - Without openVPN, with net.inet.ip.fastforwarding = 0 (default):
---------------------------------------------------------------------------

Test #1 client command line: iperf -c <firewall WLAN IP>
results: Interval=0.0-10.0sec, Transfer=29.8MBytes, Bandwidth=24.9Mbits/s

Test #2 client command line: iperf -l 32768 -c <firewall WLAN IP>
results: Interval=0.0-10.0sec, Transfer=39.8MBytes, Bandwidth=33.3Mbits/s

Test #3 client command line: iperf -w 32768 -l 65535 -c <firewall WLAN IP>
results: Interval=0.0-10.0sec, Transfer=36.6MBytes, Bandwith=30.7Mbits/s


Code: [Select]
Test Run B - With openVPN, with net.inet.ip.fastforwarding = 0 (default):
-----------------------------------------------------------------------

Test #1 client command line: iperf -c <OpenVPN Server IP>
Results: Interval=0.0-10.4sec, Transfer=944KBytes, Bandwidth=743Kbits/s

Test #2 client command line: iperf -l 32768 -c <openVPN Server IP>
Results: Interval=0.0-10.5sec, Transfer=1.88MBytes, Bandwidth=1.50Mbits/s

Test #3 client command line: iperf -w 32768 -l 65535 -c <openVPN Server IP>
Results: Interval=0.0-10.9sec, Transfer=1.94MBytes, Bandwidth=1.50Mbits/s


Code: [Select]
Test Run C - Without OpenVPN, with net.inet.ip.fastforwarding = 1:
------------------------------------------------------------------

Test #1 client command line: iperf -c <firewall WLAN IP>
results: Interval=0.0-10.0sec, Transfer=31.4MBytes, Bandwidth=26.3Mbits/sec

Test#2 client command line: iperf -l 32768 -c <firewall WLAN IP>
results: Interval=0.0-10.0sec, Transfer=39.5MBytes, Bandwidth=33.1Mbits/sec

Test#3: omitted.


Code: [Select]
Test Run D - With OpenVPN, with net.inet.ip.fastforwarding = 1:
-----------------------------------------------------------------

Test #1 client command line: iperf -c <OpenVPN Server IP>
results: Interval=0.0-10.0sec, Transfer=16.0MBytes, Bandwidth=13.4Mbits/sec

Test #2 client command line: iperf -l 32768 -c <OpenVPN Server IP>
results: Interval=0.0-10.0sec, Transfer=17.1MBytes, Bandwidth=14.0Mbits/sec

Test #3: omitted.

Compare bandwidth: Test B.2=1.50Mbits/s vs. Test D.2=14.0Mbits/s. That's an almost tenfold increase in bandwidth!
Please note, that for all test runs via openVPN, traffic shaping w/ bandwidth limiting to 14600kbps is in place.


Here's the CPU load. Time of reboot (to activate the setting) is marked with the red arrow. The reduced load is clearly visible in the graph:



Also, the time to forward packets is notably reduced:


« Last Edit: March 24, 2012, 03:10:03 pm by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Offline heper

  • Hero Member
  • *****
  • Posts: 673
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #2 on: March 24, 2012, 12:42:11 pm »
Is there a downside to this "tuneup" ?
Is it safe to enable this everywhere  without the risk of having to take a 10h drive ;)

Any advice from pfsense devs on this ?

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #3 on: March 24, 2012, 03:01:01 pm »
Quote
net.inet.ip.fastforwarding

When fast IP forwarding is enabled, IP packets are forwarded directly to the appropriate network interface with direct processing to completion, which greatly improves the throughput. All packets for local IP addresses, non-unicast, or with IP options are handled by the normal IP input processing path. All features of the normal (slow) IP forwarding path are supported including firewall (through pfil(9) hooks) checking, except ipsec(4) tunnel brokering. The IP fastforwarding path does not generate ICMP redirect or source quench messages. Compared to normal IP forwarding this can give a speedup of 40 to 60% in packet forwarding performance.

type: boolean, default: off

Source: http://segfault.in/2010/10/freebsd-net-inet-ip-sysctls-explained

Fastforwarding is purely an optimization path--if the packet requires features not supported by the fast path then it's processed normally. If this happens, a small performance hit is incurred.


« Last Edit: March 24, 2012, 03:05:51 pm by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Online cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6332
  • Karma: +0/-0
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: OpenVPN performance - Huge Improvement
« Reply #4 on: March 24, 2012, 03:06:39 pm »
fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #5 on: March 24, 2012, 03:18:33 pm »
ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.
Tech stuff on my blog: http://niston.wordpress.com

Online cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6332
  • Karma: +0/-0
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: OpenVPN performance - Huge Improvement
« Reply #6 on: March 24, 2012, 03:22:05 pm »
ICMP redirects shouldn't be used or allowed under most circumstances, for an attacker can use them to hijack your traffic and funnel it through their machine.

Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.

Offline biggsy

  • Sr. Member
  • ****
  • Posts: 380
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #7 on: March 24, 2012, 06:13:53 pm »
I tried this to see if it had any impact on OpenVPN to my own network and there was a very noticeable improvement. 

It also seemed that RDP to one of my Windows VMs was also a lot faster so I tried re-running the iperf test I reported here:  http://forum.pfsense.org/index.php/topic,45590.0.html  Same physical and virtual hosts were used and no changes to any configs - other than enabling fastforwarding.

Code: [Select]
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

------------------------------------------------------------
Client connecting to 192.168.11.2, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[128] local 192.168.111.7 port 54316 connected with 192.168.11.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[128]  0.0-10.0 sec   867 MBytes   727 Mbits/sec

That really surprised me!  No TCP window size tweak required to achieve similar performance to that described in the other thread.

fastforwarding cannot be used with IPsec and does not generate ICMP redirects, the former is primarily why that's off by default. I think there may be other caveats that I'm not recalling offhand.

I'm not concerned about IPsec but I would really like to know if there are other potential risks or problems with using fastforwarding?



Offline Helix26404

  • Jr. Member
  • **
  • Posts: 57
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #8 on: March 24, 2012, 10:14:49 pm »
Wow, upon initial testing, I am showing the same level of performance improvement. Will be interested to know if anyone finds a downside/trade-off to this.

In the mean time, thanks so much!

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #9 on: March 25, 2012, 02:22:12 am »
Quote
Not true. To accomplish that you have to be able to intercept traffic, and if you're at that point you can accomplish the same end result in multiple other ways.

Not true?

That's like saying having locks on your server racks is no good, because once an attacker stands in front of your server rack, they can simply smash it with a jackhammer, flood it with a firehose, set fire to it with gasoline or blow it into pieces with explosives :)

Another attack from http://insecure.org/sploits/arp.games.html:

Quote
ICMP redirects present a rather potent DoS.  Unlike ARP cache entries, those
host routes won't expire with time.  And of course no access to local
network is required, attack can be launched from anywhere.  So if the target
system does accept ICMP redirects (and packets can actually reach it) that
system can be stopped from talking to any particular address on the net
(well, not all, but those that aren't on the same subnet with the target).
Nameservers would be an obvious target.

Works remote if no filtering in place.

Besides, if you have to rely on ICMP redirect then your routing is flawed.
OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it.
« Last Edit: March 25, 2012, 10:33:55 am by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #10 on: March 25, 2012, 10:38:26 am »
From http://lists.freebsd.org/pipermail/freebsd-net/2004-January/002534.html:

Quote
* ip_fastforward gets its speed from processing the forwarded packet to
 * completion (if_output on the other side) without any queues or netisr's.
 * The receiving interface DMAs the packet into memory, the upper half of
 * driver calls ip_fastforward, we do our routing table lookup and directly
 * send it off to the outgoing interface which DMAs the packet to the
 * network card. The only part of the packet we touch with the CPU is the
 * IP header (unless there are complex firewall rules touching other parts
 * of the packet, but that is up to you). We are essentially limited by bus
 * bandwidth and how fast the network card/driver can set up receives and
 * transmits.
 *
 * We handle basic errors, ip header errors, checksum errors,
 * destination unreachable, fragmentation and fragmentation needed and
 * report them via icmp to the sender.
 *
 * Else if something is not pure IPv4 unicast forwarding we fall back to
 * the normal ip_input processing path. We should only be called from
 * interfaces connected to the outside world.
 *
 * Firewalling is fully supported including divert, ipfw fwd and ipfilter
 * ipnat and address rewrite.
 *
 * IPSEC is not supported if this host is a tunnel broker. IPSEC is
 * supported for connections to/from local host.
 *
 * We try to do the least expensive (in CPU ops) checks and operations
 * first to catch junk with as little overhead as possible.
 *
 * We take full advantage of hardware support for ip checksum and
 * fragmentation offloading.
 *
 * We don't do ICMP redirect in the fast forwarding path. I have had my own
 * cases where two core routers with Zebra routing suite would send millions
 * ICMP redirects to connected hosts if the router to dest was not the default
 * gateway. In one case it was filling the routing table of a host with close
 * 300'000 cloned redirect entries until it ran out of kernel memory. However
 * the networking code proved very robust and it didn't crash or went ill
 * otherwise.

So, only IPSEC tunnel brokering does not work. IPSEC from/to host does work.
Not sure for traffic shaping. I can confirm that the limiter still works, though.
« Last Edit: March 25, 2012, 10:43:18 am by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Offline dhatz

  • Hero Member
  • *****
  • Posts: 1002
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #11 on: March 25, 2012, 01:51:42 pm »
Somehow I'd have imagined that ip_fastforward would have a meaningful impact at the hundreds of Mbit/s or Gbit/s speed range.

It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense ...

Online cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6332
  • Karma: +0/-0
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: OpenVPN performance - Huge Improvement
« Reply #12 on: March 26, 2012, 04:29:20 pm »
Quote
And of course no access to local
network is required, attack can be launched from anywhere.

Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.

We don't accept ICMP redirects by default btw, we will send them though.


Besides, if you have to rely on ICMP redirect then your routing is flawed.

Yes, should never have a network design that has multiple ingress/egress points on the same subnet.

OTOH, I think CARP makes legitimate use of ICMP redirects. Not sure though, never really looked into it.

No, no use of ICMP redirects with CARP.

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #13 on: March 28, 2012, 04:09:55 am »
Don't believe everything you read on the Internet. ICMP redirects have to have the first 8 bytes of the original payload within them, good luck blind spoofing that much less getting the timing right.

Back in the days (~2001) there used to be this NZ guy on Undernet, going by the nickname Venomous.. he pulled a prank once by hijacking a shoutcast radio stream using ICMP redirection - he had to, because "his" box was not local to the stream server's subnet and thus, no ARP spoofing :) Sure, what might have worked back then won't necessarily work today. But my point is, even if an ICMP redirect is supposed to contain 64 bits of the original datagram - whether that's going to be validated in any form or not, is entirely up to the receiving IP stack. No?

As for the sanity of networking stacks, past experience (teardrop, anyone?) shows that we shouldn't put too much trust into the various implementations, especially the closed source variants. I am arguing that, under normal operating conditions, a well designed network will have no legitimate use for ICMP redirect packets. So I'm just 1) advocating a "better safe than sorry" approach by filtering icmp redirects entirely and 2) suggesting that it doesn't (for a well designed network) matter too much that the fastforwarding path doesn't generate ICMP redirects.

As for the impact of ICMP source quench not being generated, the only time I've ever seen icmp source quench was back in exactly these days, when some freak hiding behind a fake shell provider with an OC-3 would hit our 128kbps frame relay with a multi-gigabit DDoS, knocking our entire ISP offline for a few hours. Needless to add that the entire flood of source quench messages the poor zyxel sdsl routers spat and choked out was -of course- totally pointless, if not absolutely counter-productive, as all the source addresses of the bad packets chocking our routers well were naturally spoofed. Oh well. Dude finally made it into the news when the FBI would shut them down for extortion.

I must admit though, that's a quite er.. historic file I linked there :)

Yet, there is a PDF which details some of the other issues regarding icmp redirection to be found at: www.cymru.com/gillsr/documents/icmp-redirects-are-bad.pdf

« Last Edit: March 28, 2012, 05:55:11 am by gridrun »
Tech stuff on my blog: http://niston.wordpress.com

Offline gridrun

  • Jr. Member
  • **
  • Posts: 62
  • Karma: +0/-0
    • View Profile
Re: OpenVPN performance - Huge Improvement
« Reply #14 on: March 28, 2012, 04:29:09 am »
It's very surprising that you're only achieving a throughput of 1.5Mbps with OpenVPN under pfsense ...

That's exactly what I thought!
I'd still be curious to find out the root cause of the slowness :)

Maybe it has to do with the fact that my pfSense -or rather the openVPN within- is not running on bare metal?

Note that for all tests without openVPN, pfSense itself performs well as expected, in respect to the bandwidth of the WLAN. The "physical" endpoints involved remain the same for all the tests (both with ovpn and without): the win32 netbook and this (virtualized) pfSense firewall. The measured bandwidth difference is however ... dramatic.

Note also that whatever the cause may be, it doesn't happen/manifest on the fastforwarding path.

Another thing to note is, that if a client maxxed out @ 1.5Mbps, other clients could still get their 1.5Mbps. So the limiting effect was not cumulative.
« Last Edit: March 29, 2012, 07:04:38 am by gridrun »
Tech stuff on my blog: http://niston.wordpress.com