Netgate Store

Author Topic: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM  (Read 1227 times)

0 Members and 1 Guest are viewing this topic.

Offline GyroK

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-1
    • View Profile
Hello pfSense team,

as there is preferred to open the forum topic before raising a bug, I am doing so. My pfSense Xenserver VM after upgrade from latest 2.3 to to 2.4.1 keeps crashing once I am transferring bigger amount of data through the IPSEC tunnel. I would like to collect some crash data, however it does not seems for me it is even able to create any crash file. Your hint where to search for them is welcome.

Tunnel is established between two pfSense VMs, one running on ESXi 5.5 using CPU without AES-NI, second one (crashing one) is running on the Xenserver 7.0 on CPU with AES-NI. I can provide all details of the configuration, for now I have solved the issue by using the OpenVPN tunnel.

Logs during issues shows loss of connectivity and simultaneous reboot:

Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense syslogd: kernel boot file is /boot/kernel/kernel
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: Copyright (c) 1992-2017 The FreeBSD Project.
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989                                                                                         , 1991, 1992, 1993, 1994
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: The Regents of the University of California. All                                                                                          rights reserved.
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: FreeBSD is a registered trademark of The FreeBSD                                                                                          Foundation.
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: FreeBSD 11.1-RELEASE-p2 #6 r313908+7eae9364d25(R                                                                                         ELENG_2_4): Sun Oct 22 17:32:35 CDT 2017
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: root@buildbot2.netgate.com:/builder/ce-241/tmp/o                                                                                         bj/builder/ce-241/tmp/FreeBSD-src/sys/pfSense amd64
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: FreeBSD clang version 4.0.0 (tags/RELEASE_400/fi                                                                                         nal 297347) (based on LLVM 4.0.0)
Oct 31 10:46:19 pfSense syslogd: sendto: Network is unreachable
Oct 31 10:46:19 pfSense kernel: VT(vga): text 80x25
etc...

Thanks,
GyroK

Offline SisterOfMercy

  • Jr. Member
  • **
  • Posts: 47
  • Karma: +6/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #1 on: October 31, 2017, 10:40:03 am »
I have the exact same issue. I think it has also happened on 2.4.0
I do not have VMs, it does this on bare metal, with a Supermicro A1SRi-2558F.

I can reproduce the problem by just copying some files through the IPSec tunnel.
Luckily I do have crashdumps, there are three attached to this post. And of course, they have also been sent via the automatic crash dump thingy.
Hi, I'm Lance Boyle, and people often wonder if I'm real.

Offline SisterOfMercy

  • Jr. Member
  • **
  • Posts: 47
  • Karma: +6/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #2 on: October 31, 2017, 11:11:56 am »
I disabled the AES-NI CPU-based crypto accelleration, rebooted. So far this seems to work.
Hi, I'm Lance Boyle, and people often wonder if I'm real.

Offline RMB

  • Newbie
  • *
  • Posts: 17
  • Karma: +1/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #3 on: October 31, 2017, 02:11:54 pm »
I have the same issue on a SG-2440 unit.
As soon the GB's are flowing through the IPSec tunnel the unit crashes within a few minutes.
Also on the SG-2440 disabling AES-NI (System/Advanced/Misc) seems to prevent the crashes.
This behavior is introduced since version 2.4.0, release 2.3.4-P1 was working fine.

Offline GyroK

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-1
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #4 on: October 31, 2017, 02:51:26 pm »
Hello pfSense team,

I did some research and found following bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219356

I followed behavior description, and it is the same bug - after changing encryption from AES-GCM to AES tunnel is stable as was in pfSense 2.3

Looks like some regression ...

Regards,

GyroK
« Last Edit: October 31, 2017, 05:33:44 pm by GyroK »

Offline RMB

  • Newbie
  • *
  • Posts: 17
  • Karma: +1/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #5 on: November 05, 2017, 01:45:34 am »
Who should be able to fix this bug?
Is it the pfsense team, or should this be fixed by the FreeBSD developers?

Offline SisterOfMercy

  • Jr. Member
  • **
  • Posts: 47
  • Karma: +6/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #6 on: November 08, 2017, 11:39:19 am »
Is it the pfsense team, or should this be fixed by the FreeBSD developers?

It has been fixed, in FreeBSD 11-STABLE, so this particular fix might get imported into pfSense. Don't know.
Doesn't seem to be much activity, so I'll dump this in the pfSense bug tracker, because, well, I think we can safely say it's a bug.
Hi, I'm Lance Boyle, and people often wonder if I'm real.

Offline RMB

  • Newbie
  • *
  • Posts: 17
  • Karma: +1/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #7 on: November 08, 2017, 12:48:25 pm »
Great, thanks!

Offline RMB

  • Newbie
  • *
  • Posts: 17
  • Karma: +1/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #8 on: December 17, 2017, 04:03:42 am »
Any news on this bug?
The problem is still there in version 2.4.2.
I have to disable AES-NI to prevent a kernel panic during load through an IPSec tunnel.

Offline SisterOfMercy

  • Jr. Member
  • **
  • Posts: 47
  • Karma: +6/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #9 on: December 28, 2017, 08:48:24 am »
Nope, but feel free to comment on the redmine bug repo:
https://redmine.pfsense.org/issues/8070

Or find someone with a support contract that can complain.  ::)
Hi, I'm Lance Boyle, and people often wonder if I'm real.

Offline Tacoma

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #10 on: March 08, 2018, 07:35:03 am »
Having what I believe is this issue since moving to 2.4.x
Here is a picture of the console with the Kernel crash.
No log available.
Reverted back to version 2.3.x and the problem has not occurred as of yet.



Offline Tacoma

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #11 on: March 08, 2018, 11:09:34 am »
One clarification on my application, using a supermicro motherboard with pfsense installed directly to hard drive.   No VM Software involved.

Offline Tacoma

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #12 on: March 14, 2018, 02:31:46 pm »
Anyone know if this release has a fix for this issue?

2.4.3-DEVELOPMENT (amd64)
built on Tue Mar 13 10:14:21 CDT 2018
FreeBSD 11.1-RELEASE-p7

I see this is a patched version of FreeBSD, and there was a reference to ipsec fixes in the release notes, but it wasn't clear if this fixed this same issue.




Offline GyroK

  • Newbie
  • *
  • Posts: 3
  • Karma: +0/-1
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #13 on: April 02, 2018, 02:29:29 pm »
Anyone know if this release has a fix for this issue?

2.4.3-DEVELOPMENT (amd64)
built on Tue Mar 13 10:14:21 CDT 2018
FreeBSD 11.1-RELEASE-p7

I see this is a patched version of FreeBSD, and there was a reference to ipsec fixes in the release notes, but it wasn't clear if this fixed this same issue.

Unfortunately, this bug is still valid with the following SW version:

2.4.3-RELEASE (amd64)
built on Mon Mar 26 18:02:04 CDT 2018
FreeBSD 11.1-RELEASE-p7


GCM mode cannot be used on the machines with AES-NI.

Regards,
GyroK

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21840
  • Karma: +1526/-26
    • View Profile
Re: pfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM
« Reply #14 on: April 06, 2018, 03:27:45 pm »
To claim it's unusable in general is untrue. The crash must be specific to a certain combination of hardware, traffic load, and/or pattern of traffic.

Loads of people are using AES-NI and AES-GCM without crashing, including just about every Netgate employee from our home firewalls.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!