pfSense Gold Subscription

Author Topic: WAN Connection flatlines ever now and then  (Read 323 times)

0 Members and 1 Guest are viewing this topic.

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
WAN Connection flatlines ever now and then
« on: July 28, 2017, 02:05:18 am »
Greetings PFSense Support,

I am really hoping someone would be able to point me in the right direction on how to troubleshoot an issue I am having with a PFsense firewall I have.

A bit of background, I have inherited this box because I have just started at the company, and very recently we switched from a DSL connection to a microwave point to point connection for better speed and throughput as well as VoIP phones. The VoIP system is handled by our service providers Mikrotik device and it also handles our Internet connectivity. Onto the issue at hand...

The WAN connection on the PFSense box will sometimes drop the connection and a reboot will resolve the issue. The WAN configuration has been configured with a static IP pointing to the Mikrotik device. The gateway settings are the settings provided by the ISP. For the most part this setup is working correctly but as mentioned on occasion the WAN graph will show no traffic passing through and rebooting the box gets everything working again, I have tried rebooting the Mikrotik device first to see if that might be the issue but it had no effect only rebooting the box brings the traffic back up. I have increased the network buffer by adding kern.ipc.nmbclusters to 262144 and it appeared to work because for a few days I had no problem but then the problem would resurface. The ISP says all is fine from their end and because this is not the their hardware they cant offer much in the way of support. The reason why I tuned the buffer was because I was getting this error in the system log:

Quote
dpinger WANGW x.x.x.x: sendto error: 55

The LAN cable checks out okay so I dont see that as the problem. We are using No-IP free edition for Dynamic DNS.

I am also seeing errors from OpenVPN clients along the lines of

Quote
write udpv4: no buffer space available code=55

So that is pointing me to a buffer issue but I don't think I can increase it anymore.

Can someone please help out a new PFSense Admin?

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #1 on: July 28, 2017, 02:24:31 am »
Forgot version number of PFSense, sorry.

Version is 2.3.4-Release. I do see 2.3.4_1 and will schedule an update after hours on Monday evening. But this problem was also happening with the version previous to 2.3.4.

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #2 on: July 28, 2017, 03:25:45 am »
So according to this graph it does not appear to be an mbuf issue? :/

Does anyone have any suggestions for me to try? Heck if I could pay for a once off support session I would gladly do it...

Offline heper

  • Hero Member
  • *****
  • Posts: 2687
  • Karma: +253/-11
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #3 on: July 28, 2017, 04:02:59 am »
these errors indicate the packet to the NIC, but the NIC was unable to send on the wire, it fills the buffer until its full.

what type of NICs are in this system?
how many states are in use?
are you gettings interface errors ?

or i could just link to the docs .....
https://doc.pfsense.org/index.php/No_buffer_space_available
;)


i'm guessing its a faulty NIC or a bad driver (realtek?). It might have worked flawless on a slower connection & started to act up once you got your speed boosted ...
« Last Edit: July 28, 2017, 04:09:49 am by heper »

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #4 on: July 31, 2017, 01:29:27 am »
@heper thank you for the response and apologies for the late reply.

1: I have no idea what NIC's are on this box but I may have found the information by using
Code: [Select]
dmesg | More. One is identified as a Sundance ST-1023 Gigabit NIC and the other is Realtek 8139 10/100 NIC which is the onboard one. Nice to have a mismatch in interface speeds... :/

2: I am not sure how to read the how many states are in use part. Going Diagnostics --> States, lets just say there a lot of states open the page goes on and on... If I were to reset the states what would happen?

3: Is there an easy way to get interface error logs only? If I go Status --> Interfaces for both NICs there are no in/out errors or collisions as of now but when the problem happens again I will be sure to check this page.

Regarding the page you have linked thats how I found myself to the tuning page. When the issue happens again I will try bouncing the NIC via the ifconfig command, I take it I would have to specify the NIC I want to bounce? So
Code: [Select]
ifconfig em3 down; ifconfig em3 up the em3 part should match the NIC I want?

It is not a virtual NIC and there are no traffic shaping rules or limiters in place. I also dont have a spare NIC if it is the NIC that is faulty but hopefully with your assistance I can find out if it is indeed the NIC or something else that maybe faulty.

Thank you for your assistance, if anyone else wants to help out please do :).

Offline heper

  • Hero Member
  • *****
  • Posts: 2687
  • Karma: +253/-11
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #5 on: July 31, 2017, 10:28:00 am »
not familiar/never heard of sundance nics. realtek is famous for being flaky on FreeBSD.
the sundance one might be "stge0' / the realtek one should be "re0'

to see the number of states in use, you could visit status-->monitoring

personally i'd try with a different set of nic (but i always got some spare ones laying around in the back of my car ....)


Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #6 on: August 01, 2017, 01:38:56 am »
@heper so this morning it went down again. I checked which interface is the WAN side and it is the Sundance at stge1 so I ran
Code: [Select]
ifconfig stge1 down; ifconfig stge1 up and after the command completed successfully the WAN was back up and running.

So would that say that the NIC is faulty/driver issue with FreeBSD?

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #7 on: September 14, 2017, 01:31:54 am »
Morning All,

Apologies for resurrecting the thread but some new information has come to light in a sense. I have not had the time to pull the network card and try a different one so it is still very possible that the card is at fault.

I have worked out that the issue is with the VPN clients, when the guys remotely log into the system with OpenVPN and are doing work after an indeterminate amount of time the connection flat lines and I use the command mentioned above to get it up and running again.

Would this still indicate a network card hardware failure? Or could it be a software configuration that might be the issue?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9046
  • Karma: +1031/-306
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #8 on: September 14, 2017, 01:36:57 am »
Network interface.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #9 on: September 14, 2017, 01:50:51 am »
:)

Simple answer like that is very reassuring. We recently took offline the LTE firewall so I will take a card from that and see what happens. Never replaced a card in PFSense before so I assume from the software side when it boots up that PFSense will automatically pick up the new card and I will just have to assign it as the WAN connection?

Thank you for the reply, much appreciated. :)

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9046
  • Karma: +1031/-306
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #10 on: September 14, 2017, 02:44:32 am »
The best thing to do is probably install the new NIC and boot from a USB memory stick installer or CD and see what the NICs enumerate as.

If your installation is fairly simple, as in no laggs, vlans, etc, it is a pretty good bet the interface reassignment will work fine.

If it is more complicated you might want to edit the configuration and change the physical interface names.

Regardless you probably want to back up the configuration, reinstall fresh, connect to LAN, and restore the configuration.

I would have a console connected through this evolution.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Anakha56

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: WAN Connection flatlines ever now and then
« Reply #11 on: September 14, 2017, 03:05:43 am »
Very simple setup over here. No VLANS configured on the firewall and it looks like it is a default PFSense install with some changes to get Internet connectivity and VPN access. I always make a backup of the config before doing anything, learnt that lesson the hard way before :). Thank you for your advice :).