The pfSense Store

Author Topic: pfSense keeps crashing (Fatal trap 12: page fault while in kernel mode)  (Read 422 times)

0 Members and 1 Guest are viewing this topic.

Offline telendt

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile
Sorry for imprecise title but I don't know where the problem comes from and I'm not able to parse the crash logs myself.
This is my first pfSense build, I bought this hardware specifically for pfSense, it runs pfSense 2.4.1-RELEASE installed directly on the SSD nothing else:
- Supermicro A2SAV-L (MBD-A2SAV-L-O with Atom E3940 on board)
- 8GB of RAM (Kingston KVR16LS11/8)
- 64GB M.2 SSD (Transcend  TS64GMTS800)

System crashes multiple times a day (I have not yet seen an uptime higher that 24h) since it first run (I switched it on about a week ago)
Here's the crash log

I've already ran memtest86 and it went fine (no memory errors detected).
The installation is pretty standard, the only things I changed from default install options were filesystem (ZFS), network interfaces order (igb0 for LAN and igb1 for WAN), enabling AES-NI for crypto and PowerD for burst mode (I played with other stuff but at this point everything else should be pretty standard).

As mentioned before, this is my first build and I have no prior experience with pfSense. I have not tried running different builds (maybe I should?).

Any suggestions how to fix the problem are welcome.
« Last Edit: November 10, 2017, 01:23:38 pm by telendt »

Offline Birke

  • Jr. Member
  • **
  • Posts: 49
  • Karma: +12/-0
    • View Profile
Re: pfSense keeps crashing
« Reply #1 on: November 08, 2017, 07:49:34 am »
i see in the crashlog that you use intel network chips with msix.
try the tipps mentioned here: https://forum.pfsense.org/index.php?topic=69486.0

Offline chidgear

  • Jr. Member
  • **
  • Posts: 57
  • Karma: +8/-0
    • View Profile
Re: pfSense keeps crashing
« Reply #2 on: November 08, 2017, 12:25:56 pm »
It might be your installation media or your hardware too (it could be defective from factory). Try an installation using another fresh downloaded copy of pfsense (dont use the copy you already downloaded), if that fails too, try doing it in a healthy mechanical HDD, if keeps crashing, try another ram, if the issue keeps going, you could say that the issue is in you motherboard or CPU.

Greetings!

Offline telendt

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile
Re: pfSense keeps crashing
« Reply #3 on: November 09, 2017, 04:23:45 pm »
i see in the crashlog that you use intel network chips with msix.
try the tipps mentioned here: https://forum.pfsense.org/index.php?topic=69486.0

I tried the following changes (saved to /boot/loader.conf.local + reboot):

Code: [Select]
kern.ipc.nmbclusters="1000000"
hw.pci.enable_msix=0

But it didn't help.

Here's the latest crashlog.

Also, this doc page[\url] says:
Quote
If the firewall does panic, and the panic message contains a backtrace that mentions things like memory allocation, mbuf, uma_zalloc_arg, or similar, then it may be crashing due to mbuf exhaustion.

But I don't see these things being mentioned :(

What I see though are those 2 mentions:

Code: [Select]
Fatal trap 12: page fault while in kernel mode
and

Code: [Select]
Fatal trap 1: privileged instruction fault while in kernel mode
CPU is not overheated (temp below 40C), RAM passed memtest86 test.

It might be your installation media or your hardware too (it could be defective from factory). Try an installation using another fresh downloaded copy of pfsense (dont use the copy you already downloaded), if that fails too, try doing it in a healthy mechanical HDD, if keeps crashing, try another ram, if the issue keeps going, you could say that the issue is in you motherboard or CPU.

Ouch... that would be a very lengthy and hard process (especially that I don't have any spare parts).

Cheers.
« Last Edit: November 09, 2017, 04:44:23 pm by telendt »

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile

Offline Birke

  • Jr. Member
  • **
  • Posts: 49
  • Karma: +12/-0
    • View Profile
Re: pfSense keeps crashing
« Reply #5 on: November 10, 2017, 02:19:28 am »
I tried the following changes (saved to /boot/loader.conf.local + reboot):

Code: [Select]
kern.ipc.nmbclusters="1000000"
hw.pci.enable_msix=0

But it didn't help.

according to the crashlog the devices still use msix.

try these options:
Code: [Select]
kern.cam.boot_delay=10000
kern.ipc.nmbclusters=1000000
hw.igb.num_queues=1
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0
these work with our igb-devices.

Offline telendt

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile
Re: pfSense keeps crashing
« Reply #6 on: November 10, 2017, 01:14:13 pm »
according to the crashlog the devices still use msix.

try these options:
Code: [Select]
kern.cam.boot_delay=10000
kern.ipc.nmbclusters=1000000
hw.igb.num_queues=1
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0
these work with our igb-devices.

I just copied your config and rebooted the machine. An hour later I had another crash. No MSIX interrupts this time but plain MSI. Still the same panic though:

Code: [Select]
Fatal trap 12: page fault while in kernel mode
https://forum.pfsense.org/index.php?topic=136939.msg749218#msg749218
Try 2.4.2

You mean 2.4.2-DEVELOPMENT (snapshot)? Yeah, I'll try that. Playing with different builds is the only thing that's left I guess (except of trying with different hardware, which I don't really have).

Offline w0w

  • Sr. Member
  • ****
  • Posts: 522
  • Karma: +29/-6
  • kernel panic attack
    • View Profile
Re: pfSense keeps crashing
« Reply #7 on: November 10, 2017, 01:43:34 pm »
Quote from: telendt
Yeah, I'll try that. Playing with different builds is the only thing that's left I guess (except of trying with different hardware, which I don't really have).
Not really only one, but definitely better to try it first. You can also try to disable TSO AND LRO on SYSTEM>ADVANCED>NETWORKING, but if it's a freebsd bug 213903 then there is no other way, only to try the new build.

Offline telendt

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile
Re: pfSense keeps crashing (Fatal trap 12: page fault while in kernel mode)
« Reply #8 on: November 21, 2017, 03:01:53 am »
So I reinstalled pfSense from 2.4.2-RC snapshot (20171117) and used a different boot drive (some spare WD Red spinning NAS drive). After some time I experienced a first panic, "Fatal trap 1: privileged instruction fault while in kernel mode".
I applied options mentioned by @Birke, and rebooted the system.
Sysctl complained that all hw.igb.* tunables are read only, but the system ran without a crash for over 30h. Not sure if these options helped of if I had luck.

To get a better use of my systems CPU I enabled PowerD (Hidaptive) and AES-NI acceleration for crypto. Some time after that I experienced another crash, similar to previous one + mentioned "Fatal trap 12: page fault while in kernel mode".

At this point I have no idea where does the problem come from. I contacted another forum member that owns this motherboard and he told me that his system runs stable without any tunable changes :(