The pfSense Store

Author Topic: Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.  (Read 176 times)

0 Members and 1 Guest are viewing this topic.

Offline fogelholk

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
I've been reading some threads with what would seem like similar problems, but didn't feel like hijacking their threads if it would result in not being the same issue.

So here is is. I have a few static routes set up with rules allowing traffic between different VPN connections.

Whenever, what seems to be consistent, my ISP or the VPN provider I connect through has a random disconnect, my static routes changes from the VPN interface to the localhost interface by checking netstat -rn which results in my roadwarrior clients not being able to utilize the static routes I've set up.

The scenario that is working in pfSense 2.3, even through random disconnects, is that my roadwarrior client (OpenVPN) can connect through VPN connections I have set up on my pfSense to my work place.
In 2.4.x (currently 2.4.2) this connection also works perfectly fine until, for example, my ISP has a random disconnect. After this my roadwarrior client can not send traffic through the VPN tunnel I have set up from pfSense to my work place.

I hope I haven't obfuscated too much for it to be readable, but if a dev wants the real information, I'd be happy to send it privately if needed. If you need more information, just ask away! I really want to solve this and keep using pfSense 2.4, and not downgrade to 2.3 or restart pfSense weekly to have my roadwarrior working.

Here's netstat when everything is working on pfSense 2.4.2:
Code: [Select]
Internet:
Destination        Gateway            Flags     Netif Expire
default            zzz.zzz.zzz.205    UGS         lo0
10.0.11.0/24       10.0.11.1          UGS         lo0
10.0.11.1          link#8             UHS         lo0
10.0.11.2          link#8             UH       ovpns1
tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
yy.yyy.y.0/21      172.22.233.131     UGS      ovpnc3
yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
127.0.0.1          link#3             UH          lo0
172.21.0.0/16      172.22.233.131     UGS      ovpnc3
172.22.0.0/16      172.22.233.131     UGS      ovpnc3
172.22.233.128/25  172.22.233.129     UGS      ovpnc3
172.22.233.129     link#10            UH       ovpnc3
172.22.233.131     link#10            UHS         lo0
192.168.11.0/24    link#1             U          igb0
192.168.11.1       link#1             UHS         lo0
xxx.xxx.51.0/25    link#2             U          igb1
xxx.xxx.51.94      link#2             UHS         lo0
sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
zzz.zzz.zzz.192/26 zzz.zzz.zzz.193    UGS      ovpnc2
zzz.zzz.zzz.193    link#9             UH       ovpnc2
zzz.zzz.zzz.205    link#9             UHS         lo0
vvv.v.vv.0/23      172.22.233.131     UGS      ovpnc3
vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1

And here's netstat when it's Not working on pfsense 2.4.2:
Code: [Select]
Internet:
Destination        Gateway            Flags     Netif Expire
default            zzz.zzz.zzz.144    UGS         lo0
10.0.11.0/24       10.0.11.1          UGS         lo0
10.0.11.1          link#9             UHS         lo0
10.0.11.2          link#9             UH       ovpns1
tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
yy.yyy.y.0/21      172.22.233.131     UGS         lo0
yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
127.0.0.1          link#3             UH          lo0
172.21.0.0/16      172.22.233.131     UGS         lo0
172.22.0.0/16      172.22.233.131     UGS         lo0
172.22.233.128/25  172.22.233.129     UGS      ovpnc3
172.22.233.129     link#11            UH       ovpnc3
172.22.233.131     link#11            UHS         lo0
192.168.11.0/24    link#1             U          igb0
192.168.11.1       link#1             UHS         lo0
xxx.xxx.51.0/25    link#2             U          igb1
xxx.xxx.51.94      link#2             UHS         lo0
sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
zzz.zzz.zzz.129    link#10            UH       ovpnc2
zzz.zzz.zzz.144    link#10            UHS         lo0
vvv.v.vv.0/23      172.22.233.131     UGS         lo0
vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1

And here's an ifconfig output, also onfuscated to hell:

Code: [Select]
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether mm:mm:mm:mm:mm:06
        hwaddr mm:mm:mm:mm:mm:06
        inet6 fe80::12c3:7bff:fe47:e006%igb0 prefixlen 64 scopeid 0x1
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether mm:mm:mm:mm:mm:07
        hwaddr mm:mm:mm:mm:mm:07
        inet6 fe80::12c3:7bff:fe47:e007%igb1 prefixlen 64 scopeid 0x2
        inet xxx.xxx.51.94 netmask 0xffffff80 broadcast xxx.xxx.51.127
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: enc
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
        syncpeer: 224.0.0.240 maxupd: 128 defer: on
        syncok: 1
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::12c3:7bff:fe47:e006%ovpns1 prefixlen 64 scopeid 0x8
        inet 10.0.11.1 --> 10.0.11.2  netmask 0xffffff00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun openvpn
        Opened by PID 18216
ovpnc2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::12c3:7bff:fe47:e006%ovpnc2 prefixlen 64 scopeid 0x9
        inet zzz.zzz.zzz.205 --> zzz.zzz.zzz.193  netmask 0xffffffc0
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun openvpn
        Opened by PID 66191
ovpnc3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::12c3:7bff:fe47:e006%ovpnc3 prefixlen 64 scopeid 0xa
        inet 172.22.233.131 --> 172.22.233.129  netmask 0xffffff80
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun openvpn
        Opened by PID 97458

And here's ps uxaww | grep openvpn, if needed:
Code: [Select]
root    18216   0.0  0.2  20352  6204  -  Ss   15:56    0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
root    66191   0.0  0.2  20352  6648  -  Ss   15:58    0:02.68 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
root    97458   0.0  0.2  20352  6652  -  Ss   15:58    0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3.conf

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21404
  • Karma: +1437/-26
    • View Profile
Re: Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.
« Reply #1 on: December 04, 2017, 10:11:20 am »
How are you setting these routes?

If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline fogelholk

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.
« Reply #2 on: December 04, 2017, 12:05:49 pm »
How are you setting these routes?

If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.

Indeed, I had static rules set up under System > Routing > Static routes, because that was the only was I was able to get it to work like I wanted (in 2.3.x) and also setting a few Firewall > Rules where certain traffic has specific gateways set under Advanced options.

I've now removed/disabled the Static Routes (under System > Routing) and added those CIDR ranges to the VPN Clients Remote Network on the pfSense.

Seems to be working for now, I'll try to simulate ISP disconnects and see if it still works from there, otherwise I'll reply to this topic again.

Thanks for the suggestion on setting the routes correctly :)
« Last Edit: December 04, 2017, 12:14:22 pm by fogelholk »

Offline fogelholk

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.
« Reply #3 on: December 11, 2017, 11:18:22 am »
The error seems to have arrived again, I had hoped moving the static routes to the correct place would have solved all routing issues I have, but something more must be tinkered with it seems.

Here's what happens (and works for a couple of days, before something around OpenVPN connections/ISP disconnects occurs):

On a road warrior I have set up all traffic that does not belong to my network (192.168.11.0/24), should use a specific gateway, which is through the VPN provider AzireVPN. This works perfectly fine for a couple of days, and after a few OpenVPN connects/disconnects it just completely stops sending traffic from the OpenVPN server on the pfsense (which the road warrior connects to) to the OpenVPN client set up on the pfsense (towards the VPN provider AzireVPN), until I do a complete restart of the pfsense.

See attached image openvpn-server-rules.png for reference.

If I change the Gateway to "Default" instead, which uses my ISPs ordinary connection, it works. The same issue occurs with the redacted line, which is a VPN connection from the pfsense to another place, which just have some specific networks routed through it (which jimp helped me move the specific routes for in the last post).

Here's the updated netstat -rn, if needed:

Code: [Select]
Internet:
Destination        Gateway            Flags     Netif Expire
default            xxx.xxx.51.1       UGS        igb1
10.0.11.1          link#8             UHS         lo0
10.0.11.2          link#8             UH       ovpns1
tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
127.0.0.1          link#3             UH          lo0
172.22.233.0/25    172.22.233.1       UGS      ovpnc3
172.22.233.1       link#10            UH       ovpnc3
172.22.233.3       link#10            UHS         lo0
192.168.11.0/24    link#1             U          igb0
192.168.11.1       link#1             UHS         lo0
xxx.xxx.51.0/25    link#2             U          igb1
xxx.xxx.51.94      link#2             UHS         lo0
sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
zzz.zzz.zzz.129    link#9             UH       ovpnc2
zzz.zzz.zzz.139    link#9             UHS         lo0
vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1

Not sure what more information is needed, but ask away if you need more! And again, this all works perfectly fine in pfSense 2.3.x, routes and gateway rules doesn't stop working after a few days.
« Last Edit: Yesterday at 02:54:58 pm by fogelholk »