Best Multi Site to Site VPN Protocol

[Stevej]

Hi All,

We are a small VSP offering custom VoIP Solutions which end users connect to using (currently) an IPSEC VPN. Currently we have a Draytek 2955 in the Datacentre rack and deploy variou Draytek routers to customers (dependant on acces tecnology). Our 2955 is sitting on a 1Gb Ethernet connection although the router will only handle 70 presently.

Things are going well and we are expanding fast and we are looking at upgrading our core router to increase capacity and improve redundancy. We're considering this unit (or a pair of) http://www.applianceshop.eu/index.php/firewalls/opnsense/opnsense-quadcore-rack-edition-pfsense-appliance.html/ onto which to base pfsense 2.0.1.

Currently we terminate VPN's between customer sites and our network using IPSEC VPN and allocate customers an internal IP range from an internally assigned pool. In the early days we tried terminating IPSEC from a Draytek to a pfsense 1.2.3 box we had some success but randomly found the tunnel would refuse to pass data. What i am considering is deploying the hardware as suggested and migrating the current users (around 15 sites) onto the cluster using IPSEC (as i dont want to replace the CPE), and then deploy further customers over OpenVPN using pfsense at both ends.

I notice from research and trialing that in order to terminate multiple site to site VPNs onto a central pfsense box we need to configure multiple pfsense "Servers" each one on a different port. This isnt a major problem, but i am a little concerend about the use of resources in this scenario, any help would be appreciated.

Just to fill you in, the Hosted PBX's reside on a VMWare cluster and sit within /30 subnets for security meaning that a VPN to a customer site will only be able to see the PBX. On the existing Draytek i am able to add in individual IPs as routes so i can allow customer site subnets to connect to other servers outside their PBX /30 subnet, this is something i would like to replicate in pfsense.

Apologies for the lenghtly email, i suppose what i am trying to ask in short is, should i be using ipsec or openvpn to maximise efficiency and throughput (i know will need to use some ipsec). Any thoughts and advice would be appreciated.

Many thanks

Steve

[podilarius]

IMO, openVPN is more stable than IPsec (racoon).
Good choice on the HW ... those look like nice boxes.

[jimp]

OpenVPN is best for that by far. Also you might be interested to know that you can get some models of SNOM and Yealink (and probably other) handsets that have built-in OpenVPN support.

[dhatz]

Can OpenVPN under pfSense/FreeBSD scale to handle the possibly 100s of tunnels needed by a VSP ?

Also, on a related topic, are there any  real-life data about the performance limits of OpenVPN under FreeBSD ?

[jimp]

I've seen boxes with hundreds of clients connected, heard of boxes with even more. I'm not aware of any real limits. FreeBSD and some other commands might get a little slow if you have a bunch of separate static key tunnels once you get into several thousand interfaces, but really most of what you'd hit would be encryption overhead on the CPU.

[dhatz]

I've seen boxes with hundreds of clients connected, heard of boxes with even more.

Those figures would be for OpenVPN running on pfsense 2.0x  ? (sorry to be pedantic)

By the way, it would be nice to re-open the "How far have you scaled your pfSense box" thread ...

[jimp]

Yeah, on 2.x. We don't see many things still running 1.2.x but they're out there. (Just saw one today with 1.2.3 and an uptime of ~540 days...)

Typically the number of connections/interfaces isn't the limiting factor in a VPN deployment, but the total throughput is the killer.

[dhatz]

jimp,

I was basically trying to find out whether in real-life there is a difference between doing all processing in kernel (IPsec) vs user-space and having IP packets traversing the network stack back and forth (OpenVPN).

I had found some numbers for OpenVPN performance under Linux, but I know from experience that FreeBSD can behave quite differently sometimes ...

[Stevej]

Im right in my thinking that each Site to Site would require a separate VPN Server instance on the pfSense box running on a different port? We have an OpenVPN AS behind our current firewall which we use to terminate connections to customers using Snom Handsets which work well. We'd like to get them terminating on pfSense, but havent had much luck as yet.

Can i assume that multiple server "instances" as described in the pfSense documentation doesnt cause too much of an overhead?

Cheers

[podilarius]

I don't have time to look for it, but I think there is a road warrior setup using a single OpenVPN server to multiple clients at the same time. search the forums and docs.

[jimp]

You can do 1:1 or 1:many for site to site with OpenVPN. I have a few setups that have one server instance with many client sites. It does take a little doing (setting up iroutes and such) but it's easier to manage on the server side.

There are instructions on the doc wiki.

[dhatz]

I've seen boxes with hundreds of clients connected, heard of boxes with even more. I'm not aware of any real limits.

I was thinking about performance issues such as those described in http://forums.openvpn.net/topic9934.html

Apparently he fixed it with ip.fastforwarding=1

[dhatz]

According this discussion at the Openvpn forums, several users reported that a single OpenVPN server under Linux is limited to ~160Mbps:

http://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux
http://forums.openvpn.net/topic8723.html

Post subject: Gigabit Server but cannot exceed more than 160mbps

I have a gigabit server running openvpn proxy server serving a 200 road warrior users. Normal test using wget and http, traffic can achieve 700mbps without openvpn. However with more than 100 clients connecting to it, total bandwidth tops at 160Mbps. This is running in Centos 5.6. Server specs: 8GB RAM, AMD Quad Cores, SSD Drive.

From what I can observed is that once it reaches the 160Mbit, every users will be fighting for the available bandwidth within the max speed. MTU used is default value.

Really appreciate if anyone can tell me what are the best optimum settings (linux and openvpn) to utilize gigabit (server) transfer to road warriors (internet). even reaching 300Mbps is sufficient enough.

Post subject: Re: Gigabit Server but cannot exceed more than 160mbps

this is a known problem and thus far, there is very little that can be done about it. Take a look at
http://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux
which is a result of my initial investigations: it is possible to optimize a single point-to-point link, but it is NOT possible to achieve more than 160 Mbps using a "regular" client/server setup. Since then some students have performed additional tests for me, but the conclusions are still the same.

The technical reason behind this is that encryption/decryption routines are not very efficient or fast when encrypting/decrypting packets of only 1500 bytes. Unless someone can come up with a more efficient way to do this then I'm afraid that this is about the maximum speed you'll get out of a VPN setup. Perhaps an SSL accelerator card might help here, but I've not been able to test that. As an SSL accelerator card costs only ~ $50 I'd say that's worth the effort.

Do note that commercial VPNs suffer the same problem: if you want gigabit IPsec speed you pay TOP dollar.

Preliminary result of the tests and hope that others will be able to use this as well.
This is what I have done to bypass the limitations on bandwidth saturation using Gigabit, the tests were conducted on Centos.
1. Allocated 4 additional IP aliases to the main ETH0
2. Create openvpn1.conf to openvpn5.conf, each with a different IP addresses but listening on the same port.
3. Starts OpenVPN and now having 5 addresses in 5 diff tun devices, listening to user connections
4. Each tun/config files is configured with a max of 60 users (/30).
5. All tun devices is routed back to the internet using ETH0.

With this output, it seems that the bottleneck is tun device itself. Each tun can handle only 160mbit transfer rate. Though the tests is not conducted in lab environment, the real world production showed that this can be a workaround to those facing the same issues. It probably useful for the road warrior setup than the LAN setup. It probably be another 10 years for mobile users to achieve gigabit connectivity. At least my users are now happy with the performance.

I hope that OpenVPN team can run the tests more thoroughly and update us on this.

I think the bottleneck is not the tun device but the CPU. A single openvpn server can only use a single core of your processor thereby limiting how much data it can process. AFAIK openvpn is still not multithreaded. When you created multiple vpn servers listening on different IPs you are distributing the work on all the cores of your CPU. I already read some posts just like janjust that they have achieve gigabit speeds on a single tun device with encryption turned off.

[pkwong]

I'm a fan of OpenVPN, but using a pfsense box as a VPN concentrator while convenient seems to be somewhat counter-intuitive from a resource utilization perspective.  I've done something very similar for a client, a completely tuned voip encrypted network Where all the clients can communicate with the PBX or Softswitch, but nobody else.  It's really a great way to make things work.

I've used IPSEC, but the reality for me is I only deploy it on IPV6.  (It's an extension).   So it makes life much easier.  Using an OpenVPN server (dedicated) also allows you to tune specifically for VOIP (A HUGE PLUS) and when configured properly, tunnels re-establish very quickly.  The PFsense implementation of OpenVPN is great, but there is so much missing vs. a dedicated openvpn server that you can do (magic wise).. like real-time tuning of traffic / bandwidth-optimization using scripts.  It's all doable in pfsense, but it becomes a completely customized system at that point.

Cheers.

Percy
http://swimminginthought.com

[dhatz]

I'm a fan of OpenVPN, but using a pfsense box as a VPN concentrator while convenient seems to be somewhat counter-intuitive from a resource utilization perspective.

For serving a relatively small number (e.g. a few dozen) of OpenVPN clients, wouldn't it in fact more efficient from a resource utilization perspective to use pfsense as VPN concentrator ?

Using an OpenVPN server (dedicated) also allows you to tune specifically for VOIP (A HUGE PLUS) and when configured properly, tunnels re-establish very quickly.  The PFsense implementation of OpenVPN is great, but there is so much missing vs. a dedicated openvpn server that you can do (magic wise).. like real-time tuning of traffic / bandwidth-optimization using scripts.  It's all doable in pfsense, but it becomes a completely customized system at that point.

Can you be more specific about traffic-shaping optimization with scripts ? iirc pfsense can do traffic shaping of traffic inside the tunnel with the regular traffic shaping subsystem.