pfSense Support Subscription

Author Topic: Routing Problem in Test Network  (Read 314 times)

0 Members and 1 Guest are viewing this topic.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Routing Problem in Test Network
« on: October 11, 2017, 03:56:23 pm »
I've been playing with a small vagrant test network, trying to get a VPN set up.
The network looks like:

Client:     192.168.57.2/24 (WAN)
pfSense:  192.168.57.3/24 (WAN)
               192.168.58.2/24 (LAN)
Internal:  192.168.58.3/24 (LAN)


I've set up pfSense's openvpn using the wizard, mostly sticking to the defaults.

IPv4 Tunnel Network : 192.168.80.0/24
IPv4 Local network(s): 192.168.58.0/24
Custom Options: push "route 192.168.58.0 255.255.255.0"


Afterwards, I set up an account, exported the configuration file, and connected to pfsense using the client.

openvpn --config configFile.ovpn --auth-user-pass auth.txt
Wed Oct 11 16:26:52 2017 WARNING: file 'auth.txt' is group or others accessible
Wed Oct 11 16:26:52 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
Wed Oct 11 16:26:52 2017 library versions: OpenSSL 1.0.2l  25 May 2017, LZO 2.08
Wed Oct 11 16:26:52 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:52 2017 UDP link local (bound): [AF_INET][undef]:1194
Wed Oct 11 16:26:52 2017 UDP link remote: [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:52 2017 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed Oct 11 16:26:52 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
Wed Oct 11 16:26:53 2017 TUN/TAP device tun0 opened
Wed Oct 11 16:26:53 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Wed Oct 11 16:26:53 2017 /sbin/ip link set dev tun0 up mtu 1500
Wed Oct 11 16:26:53 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
RTNETLINK answers: File exists
Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2
Wed Oct 11 16:26:53 2017 Initialization Sequence Completed


In another tab, I attempted to connect to the internal machine. What I found was that I could reach the LAN interface of the pfsense machine from the Client machine,  but not the internal machine. Looking into the logs, I found that this happened.

Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 UDPv4 READ [133] from [AF_INET]192.168.57.2:1194: P_DATA_V1 kid=0 DATA 8f1629dd d9f14fff a811ffc6 22fe80eb dc68cf72 9e0fd397 5d4a6621 c8a7a3e[more...]
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 TLS: tls_pre_decrypt, key_id=0, IP=[AF_INET]192.168.57.2:1194
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 DECRYPT IV: 9e0fd397 5d4a6621 c8a7a3e5 e7aebda1
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 DECRYPT TO: 00000012 45000054 84ae4000 4001aaa4 c0a85002 c0a83a03 080061de 4313000[more...]
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 PID_TEST \[0] [SSL-0] [1233456789>>>>>EE] 0:17 0:18 t=1507753641[0] r=[-2,64,15,0,1] sl=[47,17,64,528]
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.80.2 -> client1/192.168.57.2:1194 via 192.168.80.2
Oct 11 20:27:21   openvpn   57629   client1/192.168.57.2:1194 GET INST BY VIRT: 192.168.58.3 [failed]


It appears that the machine is having problems routing from the pfsense machine to the internal machine.
However, I checked that both the pfsense machine and the internal machine could ping each other within the network.
I'm at a loss of how to fix this problem. Can anyone help?

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #1 on: October 11, 2017, 04:18:51 pm »
Adding the route fails on the client as the log shows:
Code: [Select]
Wed Oct 11 16:26:53 2017 ERROR: Linux route add command failed: external program exited with error status: 2
The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #2 on: October 11, 2017, 04:27:26 pm »
The "push route" command in the custom options is not needed, since this is already set by the entry in "Local Network(s)". So delete it from the custom options field. Maybe this solve your issue.

I made the change you suggested. It removed the add route error, thanks!

Wed Oct 11 17:21:53 2017 WARNING: file 'auth.txt' is group or others accessible
Wed Oct 11 17:21:53 2017 OpenVPN 2.4.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 30 2017
Wed Oct 11 17:21:53 2017 library versions: OpenSSL 1.0.2l  25 May 2017, LZO 2.08
Wed Oct 11 17:21:53 2017 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.57.3:1194
Wed Oct 11 17:21:53 2017 UDP link local (bound): [AF_INET][undef]:1194
Wed Oct 11 17:21:53 2017 UDP link remote: [AF_INET]192.168.57.3:1194
Wed Oct 11 17:21:53 2017 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed Oct 11 17:21:53 2017 [server] Peer Connection Initiated with [AF_INET]192.168.57.3:1194
Wed Oct 11 17:22:04 2017 TUN/TAP device tun0 opened
Wed Oct 11 17:22:04 2017 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Wed Oct 11 17:22:04 2017 /sbin/ip link set dev tun0 up mtu 1500
Wed Oct 11 17:22:04 2017 /sbin/ip addr add dev tun0 192.168.80.2/24 broadcast 192.168.80.255
Wed Oct 11 17:22:04 2017 Initialization Sequence Completed


However, I am still failing to reach the internal machine, and the pfsense logs are still showing the similar GET INST BY VIRT [failed] error.

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #3 on: October 11, 2017, 05:11:08 pm »
Now you're missing the "route add" command on the client.
Please post your server and client config.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #4 on: October 12, 2017, 08:17:20 am »
Okay, so taking from my config.xml:

<openvpn>
      <openvpn-server>
         <vpnid>1</vpnid>
         <mode>server_tls_user</mode>
         <authmode>Local Database</authmode>
         <protocol>UDP</protocol>
         <dev_mode>tun</dev_mode>
         <ipaddr></ipaddr>
         <interface>wan</interface>
         <local_port>1194</local_port>
         <description><![CDATA[CertVPN]]></description>
         <custom_options></custom_options>
         <tls>-removed-</caref>
         <crlref></crlref>
         <certref>59de7040dec5d</certref>
         <dh_length>2048</dh_length>
         <cert_depth>1</cert_depth>
         <strictusercn></strictusercn>
         <crypto>AES-256-CBC</crypto>
         <digest>SHA1</digest>
         <engine>none</engine>
         <tunnel_network>192.168.80.0/24</tunnel_network>
         <tunnel_networkv6></tunnel_networkv6>
         <remote_network></remote_network>
         <remote_networkv6></remote_networkv6>
         <gwredir></gwredir>
         <local_network>192.168.58.0/24</local_network>
         <local_networkv6></local_networkv6>
         <maxclients>10</maxclients>
         <compression></compression>
         <passtos></passtos>
         <client2client>yes</client2client>
         <dynamic_ip>yes</dynamic_ip>
         <pool_enable>yes</pool_enable>
         <topology>subnet</topology>
         <serverbridge_dhcp></serverbridge_dhcp>
         <serverbridge_interface>none</serverbridge_interface>
         <serverbridge_dhcp_start></serverbridge_dhcp_start>
         <serverbridge_dhcp_end></serverbridge_dhcp_end>
         <netbios_enable></netbios_enable>
         <netbios_ntype>0</netbios_ntype>
         <netbios_scope></netbios_scope>
         <no_tun_ipv6></no_tun_ipv6>
         <verbosity_level>10</verbosity_level>
      </openvpn-server>
   </openvpn>
   <dnshaper></dnshaper>
   <cert>
      <refid>52db1097e7e40</refid>
      <descr><![CDATA[webConfigurator default]]></descr>
      <crt>-removed-</prv>
   </cert>
   <cert>
      <refid>59de7040dec5d</refid>
      <descr><![CDATA[server]]></descr>
      <type>server</type>
      <caref>59de7040c414a</caref>
      <crt>-removed-</crt>
      <prv>-removed-</prv>
   </cert>
   <cert>
      <refid>59de707095d08</refid>
      <descr><![CDATA[CertCert]]></descr>
      <type>user</type>
      <caref>59de7040c414a</caref>
      <crt>-removed-</crt>
      <prv>-removed-</prv>
   </cert>
   <ppps></ppps>
   <dyndnses></dyndnses>
   <gateways></gateways>
   <ovpnserver>
      <step1>
         <type>local</type>
      </step1>
      <step6>
         <certca>Cert-CA</certca>
         <keylength>2048</keylength>
         <lifetime>3650</lifetime>
         <country>US</country>
         <state>-removed-</state>
         <city>-removed-</city>
         <organization></organization>
         <email>-removed-</email>
         <uselist>on</uselist>
      </step6>
      <step9>
         <certname>server</certname>
         <keylength>2048</keylength>
         <lifetime>3650</lifetime>
         <country>US</country>
         <state>-removed-</state>
         <city>-removed-</city>
         <organization></organization>
         <email>-removed-</email>
         <uselist>on</uselist>
      </step9>
      <step10>
         <interface>wan</interface>
         <protocol>UDP</protocol>
         <localport>1194</localport>
         <descr><![CDATA[CertVPN]]></descr>
         <tlsauth>on</tlsauth>
         <gentlskey>on</gentlskey>
         <dhkey>2048</dhkey>
         <crypto>AES-256-CBC</crypto>
         <digest>SHA1</digest>
         <engine>none</engine>
         <tunnelnet>192.168.80.0/24</tunnelnet>
         <localnet>192.168.58.0/24</localnet>
         <concurrentcon>10</concurrentcon>
         <interclient>on</interclient>
         <dynip>on</dynip>
         <addrpool>on</addrpool>
         <topology>subnet</topology>
         <nbttype>0</nbttype>
         <advanced>push &quot;route 192.168.58.0 255.255.255.0&quot;</advanced>
      </step10>
      <step11>
         <ovpnrule>on</ovpnrule>
         <ovpnallow>on</ovpnallow>
      </step11>
   </ovpnserver>
   <ca>
      <refid>59de7040c414a</refid>
      <descr><![CDATA[Cert-CA]]></descr>
      <crt>-removed-</crt>
      <prv>-removed-</prv>
      <serial>2</serial>
   </ca>


Taking from configFile.ovpn:

dev tun
persist-tun
persist-key
cipher AES-256-CBC
auth SHA1
tls-client
client
resolv-retry infinite
remote 192.168.57.3 1194 udp
verify-x509-name "server" name
auth-user-pass
remote-cert-tls server
<ca>
-----BEGIN CERTIFICATE-----
-removed-
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
-removed-
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
-removed-
-----END PRIVATE KEY-----
</key>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
-removed-
-----END OpenVPN Static key V1-----
</tls-auth>
key-direction 1

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #5 on: October 12, 2017, 03:45:32 pm »
XML-File  :o
A screenshot from the server setting page is much easier to read. The whole config-file can be found in /var/etc/openvpn.

However, I cannot find any mistake in the config. Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #6 on: October 12, 2017, 04:06:52 pm »
Though the client doesn't set the routes. Maybe you're missing permissions for changing routes. However, this should be mentioned in the client log.

What do you mean the client doesn't set the routes? Is there a line missing?

When running the client VPN I am running as a user with passwordless sudo, using the ovpn generated by the openvpn exporter.

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #7 on: October 12, 2017, 04:41:29 pm »
If you have entered a network in the "Local Networks" field in the server setting, a route for this network should be pushed to the client.
After the client had connected you should see an entry for adding routes in the client log like
Code: [Select]
/sbin/ip route add 192.168.58.0/24 via 192.168.80.1
That line is missing in your log.

You may set the route manually to see if you have the necessary permissions.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #8 on: October 13, 2017, 09:17:46 am »
I checked and was able to run that command, but I was getting "RTNETLINK answers: File exists" as a response.
To check, I looked at the IP routing tables:

Kernel IP routing table
Destination       Gateway           Genmask         Flags    MSS Window  irtt Iface
default             10.0.2.2            0.0.0.0             UG       0     0          0 eth0
10.0.2.0           0.0.0.0             255.255.255.0   U         0     0          0 eth0
192.168.57.0    0.0.0.0             255.255.255.0   U         0     0          0 eth1
192.168.58.0    192.168.80.1    255.255.255.0   UG       0     0          0 tun0
192.168.80.0    0.0.0.0             255.255.255.0   U         0     0          0 tun0


(Note that due to this being on vagrant, the first gateway is used to provision the machine.)

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #9 on: October 13, 2017, 09:24:42 am »
So the necessary route is set fine as the routing table shows.
Has it been already set before you were running that command?

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #10 on: October 13, 2017, 09:50:41 am »
The route was set before running the command.

I've started and stopped the openVPN client several times on the machine, so it may have been set earlier.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 13913
  • Karma: +1286/-187
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Routing Problem in Test Network
« Reply #11 on: October 13, 2017, 09:55:26 am »
"192.168.57.0    0.0.0.0             255.255.255.0   U         0     0          0 eth1"

That route is not going down your tunnel.. So how would you expect to get their through the tunnel.. Seems your client is trying to go out eth1 to get there.  With no gateway so it thinks that network is on its eth1 interface.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- If I have helped you, applaud me is nice cheap way to say thanks!
- if you want to say bigger thanks https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.1-RC Oct 22 11:33:09 VM running on esxi 6.5 (home)

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #12 on: October 13, 2017, 12:44:58 pm »
I think, that's just the way how his OS prints networks connected directly to an interface. The line has only an U-flag, no G for a gateway.
It the same as
10.0.2.0           0.0.0.0             255.255.255.0   U         0     0          0 eth0

Offline viragomann

  • Hero Member
  • *****
  • Posts: 2395
  • Karma: +250/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #13 on: October 13, 2017, 12:56:14 pm »
Now, as the route is set as it should be, why do you think, you have a routing problem?

Try to ping pfSense internal address and see if you get a response.

Offline Alejandro.Carbonara

  • Newbie
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: Routing Problem in Test Network
« Reply #14 on: October 13, 2017, 01:06:05 pm »
As of now, I can ping the internal ip of pfsense and get a positive response.

Attempting to ping the internal machine gets me no response whatsoever.