So to start, I have gotten this setup working but it feels like I'm working around some kind of bug or limitation in an unclean way (as described below). I'd also like comments on the setup in general - please tell me if I'm going about this in an overly complex way (and give me an alternate if so).
I have a remote endpoint running an OpenVPN server that I want my pfSense setup to connect to. I have servers on the LAN network that I want to use the WAN as their default route, but want select processes on these servers to use the OpenVPN connection as a default route. Additionally, some of these server processes require ports to be forwarded from the remote (other side of the OpenVPN connection) interface in. Other computers on the network should also optionally be able to use the OpenVPN outbound (without the default outbound WAN gateway).
To do this, I decided to set up a VLAN on the LAN interface with an alternate subnet that would route out this alternate gateway.
pfSense 2.0.1 on an Alix 2d3 (only vr0 and vr1 in use)
WAN interface: vr0, 126.96.36.199 (made up address)
LAN interface: vr1, 192.168.0.1/24 (DHCP server running)
RVPN1 interface: ovpnc1 (openvpn client interface), 192.168.10.2/24
RVPN1LOCAL interface: VLAN 101 on vr1, 192.168.5.1/24 (DHCP server running)
Remote VPN server (linux):
eth0: 188.8.131.52 (made up address)
MASQUERADE out eth0
DNAT on 184.108.40.206 port 45500 to 192.168.10.2
Local server1 on network (linux):
eth0: 192.168.0.10 (static)
eth0.101: 192.168.5.10 (static)
Manual Outbound Nat:
WAN, source:192.168.0.1/24 (auto created rule)
WAN, source:127.0.0.1/8, ports:1024-65535 (auto created rule)
RVPN1, TCP, src:*, src port:*, dst:RVPN1 address, dst port:45500, nat IP: 192.168.5.10, nat port: 45500
RVPN1: interface RVPN1, gateway: 192.168.10.1
LAN: default (source lan net, out anywhere)
WAN: default (nothing inbound)
RVPN1: TCP, src:*, src port: *, dest: 192.168.5.10, dest port: 45500
ICMP, src: RVPN1 subnet, dest: RVPN1 address
RVPN1LOCAL: src: RVPN1LOCAL subnet Gateway RVPN1 (192.168.10.1)
The weird problem:
The initial trial worked well. I was able to get processes bound to the VLAN IP (192.168.5.10) on server1 to go out the OpenVPN. Other processes went out through WAN. What I found didn't work was when I tested the forwarded port on the remote OpenVPN server's external IP address. A simple telnet to the port would result in the sender's connection being stuck in SYN_SENT and server1's connection in SYN_RECV. After some time, the connection would time out.
Some tcpdumps later, I saw confirmation of what I thought was happening - the SYN packet came in on the OpenVPN remote, was DNAT'd to RVPN1 ip, DNAT'd again to server1's eth0.101 IP and received by server1 with the correct remote address (that of the connection originator). server1 then sent out a SYN/ACK packet which was then seen to exit out of WAN (vr0 on the pfsense box) rather than the expected interface of RVPN1. Logging on the firewall rules showed that the only rule hit was the one on RVPN1 (the policy route rule on RVPN1LOCAL was never hit). The states table likewise showed the connection in an incomplete state (I cant recall exactly what).
My workaround solution:
2 floating firewall rules:
Interface: RVPN1LOCAL, direction: in, TCP, source: RVPN1LOCAL subnet, tcp flags: SYN/ACK out of SYN/ACK, state type: none, gateway: RVPN1
Interface: RVPN1LOCAL, direction: out, TCP, dest: RVPN1LOCAL subnet, tcp flags: SYN out of SYN/ACK, state type: keep state
The first is an explicit match on the SYN/ACK flag, routing it out of RVPN1. Setting state type to none seems to be necessary otherwise the rule is never matched. I'm guessing this is because the connection tracking logic 'consumes' it somehow.
After adding the first rule, connections were established OK, but when the connection was torn down by the remote end the FIN would be received by server1 but the FIN/ACK outbound would be rejected by the default rule due to it not being associated with a known connection. The 2nd rule resolves that behavior.
The big question i think is most obvious: Shouldn't the connection tracking keep track of what interface the SYN comes in on and send the SYN/ACK out the same one? It *does* seem to do this correctly once the connection is fully established. Is this the manifestation of a bug (seems unlikely..) or some consequence of how pfsense configures things?
The other question is as mentioned above: Is there a cleaner way to do this while keeping pfsense as the VPN client (I'd rather not have individual servers each running openvpn clients)?
Thanks in advance