Netgate SG-1000 microFirewall

Author Topic: OpenVPN dpinger behavior question  (Read 285 times)

0 Members and 1 Guest are viewing this topic.

Offline whorfin

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
OpenVPN dpinger behavior question
« on: October 23, 2017, 11:43:45 pm »
[edited to add - currently on 2.4.0]
[...and same on 2.4.1]
Greetings

I've been trying for some time to narrow down some problems where OpenVPN seems to lose connection.  If it exits, the watchdog I've got running [thanks, Packages!] works great.
If not, dpinger catches the problem but then nothing happens.

In the logs i see for example
Quote
Oct 22 16:04:33   rc.gateway_alarm   61843   >>> Gateway alarm: VPN_CLIENT_VPNV4 (Addr: redacted Alarm:1 RTT:29709ms RTTsd:11880ms Loss:22%)
which will be quickly followed by
Quote
Oct 22 16:04:33 pfSense check_reload_status: updating dyndns VPN_CLIENT_VPNV4
Oct 22 16:04:33 pfSense check_reload_status: Restarting ipsec tunnels         
Oct 22 16:04:33 pfSense check_reload_status: Restarting OpenVPN tunnels/interfaces                                                                         
Oct 22 16:04:33 pfSense check_reload_status: Reloading filter                 
Oct 22 16:04:35 pfSense php-fpm[71737]: /rc.dyndns.update: phpDynDNS (redacted.nope.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.                                                   
Oct 22 16:04:35 pfSense php-fpm[62324]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN_CLIENT_VPNV4.

That all looks great, but doesn't actually restart anything, as confirmed by the OpenVPN logs.

I see that the following is executed by the gateway alarm script:
Quote
pfSctl \
    -c "service reload dyndns VPN_CLIENT_VPNV4" \
    -c "service reload ipsecdns" \
    -c "service reload openvpn VPN_CLIENT_VPNV4" \
    -c "filter reload"

Digging in rc.openvpn shows that openvpn_resync_if_needed() is not called, as the conditions are not met [client interface is WAN, but $interface is opt8, where VPN_CLIENT_VPNV4/ovpnc2 sits]
Even if openvpn_resync_if_needed() were called, it wouldn't ever call openvpn_resync(), as
it notices that the binding interface didn't change [there's no failover].  There's a comment that says
Quote
  /* Compare the interface currently used by the VPN with the interface that should be used.
     If the VPN should stay on the same interface do not resync */       
   


Is that really all as intended?
Is it further intended that "service restart openvpn VPN_CLIENT_VPNV4" won't start openvpn
if not running?  It'd sure be great if the alarm could trigger a restart, as then no watchdog
would be needed and dpinger would be doing the job.

I'm going to experiment with the "interval" argument along with a watchdog, but again, wanted to understand if observed behavior was as intended.  It was surprising to me that "service reload openvpn" doesn't actually reload unless a bunch of failover-related conditions are met.  No doubt I missed or forgot something along the way.

Cheers
« Last Edit: November 01, 2017, 09:54:49 pm by whorfin »

Offline whorfin

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: OpenVPN dpinger behavior question
« Reply #1 on: November 04, 2017, 05:10:36 pm »
Don't all answer at once....

Well, adding inactive directive and forcing restart on ping failure did not improve the situation.
Code: [Select]
inactive 120
remap-usr1 SIGHUP

To recap.  I run an OpenVPN client on pfSense which is meant to be always on.
Every random amount of time, depending on ISP whatever, the connection stops working.
OpenVPN itself rarely notices the problem, even with the settings above [in addition to the usual keepalive].
If I am de-authed and it exits, my watchdog catches it, but otherwise the OpenVPN logs don't show a problem.
Attempting to use the network shows packets not flowing.
dpinger always notices the problem, and fires the gateway alarm.
...which does nothing, hence my posting here.
Going back through lengthy logs shows dpinger always noticing, reporting, firing alarm.  And then nothing until manual intervention.

So, I made this patch:
Code: [Select]
--- /etc/rc.openvpn     2017-11-03 19:46:16.254944000 -0700
+++ /tmp/rc.openvpn     2017-11-03 19:49:20.564839000 -0700
@@ -40,17 +40,18 @@
        if (isset($ovpn_settings['disable'])) {
                $resync_needed = false;
        } else {
-               if (!empty($interface)) {
-                       $mode_id = $mode . $ovpn_settings['vpnid'];
-                       $fpath = "{$g['varetc_path']}/openvpn/{$mode_id}.interface";
-                       if (file_exists($fpath)) {
-                               /* Compare the interface currently used by the VPN with the interface that should be used.
-                                  If the VPN should stay on the same interface, do not resync */
-                               if (trim(file_get_contents($fpath), " \t\n") == get_failover_interface($ovpn_settings['interface'])) {
-                                       $resync_needed = false;
-                               }
-                       }
-               }
+# /* We want to resync if called and the interface is not disabled */
+#              if (!empty($interface)) {
+#                      $mode_id = $mode . $ovpn_settings['vpnid'];
+#                      $fpath = "{$g['varetc_path']}/openvpn/{$mode_id}.interface";
+#                      if (file_exists($fpath)) {
+#                              /* Compare the interface currently used by the VPN with the interface that should be used.
+#                                 If the VPN should stay on the same interface, do not resync */
+#                              if (trim(file_get_contents($fpath), " \t\n") == get_failover_interface($ovpn_settings['interface'])) {
+#                                      $resync_needed = false;
+#                              }
+#                      }
+#              }
        }
        if ($resync_needed == true) {
                log_error("OpenVPN: Resync " . $mode_id . " " . $ovpn_settings['description']);
@@ -116,9 +117,10 @@

        if (is_array($config['openvpn']['openvpn-client'])) {
                foreach ($config['openvpn']['openvpn-client'] as &$client) {
-                       if ($client['interface'] == $interface || empty($interface) || (!empty($gwgroups) && in_array($client['interface'], $gwgroups))) {
+#                      /* We want to resync if called; in our case, $client['interface'] != $interface but we still want to resync */
+#                      if ($client['interface'] == $interface || empty($interface) || (!empty($gwgroups) && in_array($client['interface'], $gwgroups))) {
                                openvpn_resync_if_needed('client', $client, $interface);
-                       }
+#                      }
                }
        }
 }

Apply with path strip of 0 and default root dir.  And of course do so at your own risk.
I have no idea what the logic in here which twice tries to prevent the client from actually resyncing is for.
This patch disables the "only resync if VPN interface isn't what is set" logic in openvpn_resync_if_needed() and removes the various interface/gwgroups tests just for the "client" case.
Again, I don't know what they're there for, but for my use-case, this change now means that for the first time, my OpenVPN client has become reliable.
When dpinger sees a problem, it restarts the client.

This may definitely result in momentary connection loss, but that is preferable to connection being down until manual intervention.

Again, I'd appreciate commentary from those more knowledgeable on why those tests are in there.  I think I'm requesting a supported way to disable them...

Cheers

Offline whorfin

  • Newbie
  • *
  • Posts: 16
  • Karma: +0/-0
    • View Profile
Re: OpenVPN dpinger behavior question
« Reply #2 on: November 22, 2017, 04:33:30 pm »
no change in 2.4.2

Offline muppet

  • Newbie
  • *
  • Posts: 24
  • Karma: +1/-0
  • I'm a Muppet
    • View Profile
    • Tim H
Re: OpenVPN dpinger behavior question
« Reply #3 on: February 06, 2018, 07:49:55 pm »
I see the same on my pfSense 2.4.2_1:

Something triggers when check_reload_status is run that causes dpinger issues.

Not a biggy, but I too have noticed this.

Code: [Select]
Feb 6 20:48:20 rc.gateway_alarm 72290 >>> Gateway alarm: WAN_PPPOE (Addr:X.X.X.X Alarm:0 RTT:3906ms RTTsd:1128ms Loss:0%)
Feb 6 20:48:20 check_reload_status updating dyndns WAN_PPPOE
Feb 6 20:48:20 check_reload_status Restarting ipsec tunnels
Feb 6 20:48:20 check_reload_status Restarting OpenVPN tunnels/interfaces
Feb 6 20:48:20 check_reload_status Reloading filter
Feb 6 20:48:21 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
Feb 6 21:46:58 php-fpm /index.php: Successful login for user 'admin' from: 192.168.0.106
Feb 7 01:46:58 rc.gateway_alarm 95168 >>> Gateway alarm: WAN_PPPOE (Addr:X.X.X.X Alarm:1 RTT:134683ms RTTsd:486359ms Loss:0%)
Feb 7 01:46:58 check_reload_status updating dyndns WAN_PPPOE
Feb 7 01:46:58 check_reload_status Restarting ipsec tunnels
Feb 7 01:46:58 check_reload_status Restarting OpenVPN tunnels/interfaces
Feb 7 01:46:58 check_reload_status Reloading filter
Feb 7 01:46:59 php-fpm /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_PPPOE.
Feb 7 01:47:12 rc.gateway_alarm 97132 >>> Gateway alarm: WAN_PPPOE (Addr:X.X.X.X Alarm:1 RTT:219457ms RTTsd:556905ms Loss:11%)
Feb 7 01:47:12 check_reload_status updating dyndns WAN_PPPOE
Feb 7 01:47:12 check_reload_status Restarting ipsec tunnels
Feb 7 01:47:12 check_reload_status Restarting OpenVPN tunnels/interfaces
Feb 7 01:47:12 check_reload_status Reloading filter