pfSense Support Subscription

Author Topic: pfSense crashed on Alix  (Read 18028 times)

0 Members and 1 Guest are viewing this topic.

Offline xbipin

  • Hero Member
  • *****
  • Posts: 1631
  • Karma: +6/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #30 on: January 24, 2010, 11:16:04 am »
i use 22nd snapshot on alix, hasnt crashed for me till now so might be some hardware issue or something like that

Offline jlepthien

  • Hero Member
  • *****
  • Posts: 657
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #31 on: January 24, 2010, 12:50:59 pm »
I don't think it is hardware related since 1.2.3 is running fine on this box. This just happened now with 2.0-beta1...
| apple fanboy | music lover | network and security specialist | in love with cisco systems |

Offline sullrich

  • Hero Member
  • *****
  • Posts: 5110
  • Karma: +7/-2348
    • View Profile
    • pfSense
Re: pfSense crashed on Alix
« Reply #32 on: January 24, 2010, 03:57:13 pm »
We're looking into it.

Offline jlepthien

  • Hero Member
  • *****
  • Posts: 657
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #33 on: January 26, 2010, 01:08:56 am »
Thanks! Is there anyway I can tell pfSense to reboot automatically when it panics? But I guess no :-(
I think I will go back to 1.2.3 because my girlfriend hates me everytime the internet connection dies and now it is almost daily ;)

Downgrade is only working by re-flashing? I have an old 1.2.3 conf...
| apple fanboy | music lover | network and security specialist | in love with cisco systems |

Offline jlepthien

  • Hero Member
  • *****
  • Posts: 657
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #34 on: January 29, 2010, 07:02:12 am »
I can confirm now that it is definitely not a hardware issue. Now my box is running fine again with 1.2.3. I will use 2.0 again when it has RC status the earliest...

Thanks
| apple fanboy | music lover | network and security specialist | in love with cisco systems |

Offline Uxorious

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #35 on: February 18, 2010, 10:29:22 pm »
I just had what is possibly the same problem on an old Dell OptiPlex GX200 with a dual Intel gigabit card installed.

LAN IP was completely dead, and I did not have a keyboard so no backtrace:
em1: watchdog timeout -- resetting
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0xe0500a4
fault code = supervisor read, page not present
instruction pointer - 0x20:0xc0a63aa7
stack pointer = 0x28:0xe2c547c4
frame pointer = 0x28:0xe2c547f0
code segment = base 0x0, limit 0xfffff, type 0x1b
     DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (em0 taskq)
[thread pid 0 tid 64027]
Stopped at rn_match+0x17: movl 0xc(%eax),%ebx

Build used was pfSense-2.0-BETA1-20100216-2021.iso installed to the HDD.
Box had been running for almost 2 days when it happened.

Offline Uxorious

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #36 on: February 20, 2010, 12:49:43 pm »
Stopped at rn_match+0x17: movl 0xc(%eax),%ebx

It happened again some 20 hours later.
LAN dead again, but stopped at exactly the same instruction.
Since writing down the bactrace was too painful, I took a picture instead.

Offline ttlinna

  • Jr. Member
  • **
  • Posts: 37
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #37 on: February 22, 2010, 02:07:00 am »
I just had what is possibly the same problem on an old Dell OptiPlex GX200 with a dual Intel gigabit card installed.

LAN IP was completely dead, and I did not have a keyboard so no backtrace:
em1: watchdog timeout -- resetting
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0xe0500a4
fault code = supervisor read, page not present
instruction pointer - 0x20:0xc0a63aa7
stack pointer = 0x28:0xe2c547c4
frame pointer = 0x28:0xe2c547f0
code segment = base 0x0, limit 0xfffff, type 0x1b
     DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (em0 taskq)
[thread pid 0 tid 64027]
Stopped at rn_match+0x17: movl 0xc(%eax),%ebx

Build used was pfSense-2.0-BETA1-20100216-2021.iso installed to the HDD.
Box had been running for almost 2 days when it happened.

I've had multiple similar problems. Unfortunately I haven't been able to grab the log since the problems have occured in production environments. Network just stops suddenly working. It can run well for days or just for an hour or so.

I've rolled back also to 1.2.3 and it's running fine on the same hardware. I've had problems both with Alix 2d3 boards and different pc hardware.

My config includes use of limiters. Is it possible that it causes problems?

That's just my hunch, since I've got older snapshots running fine without limiters.

BR,

Tommi

Offline Uxorious

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #38 on: February 22, 2010, 02:20:03 am »
I've had multiple similar problems. Unfortunately I haven't been able to grab the log since the problems have occured in production environments. Network just stops suddenly working. It can run well for days or just for an hour or so.

My config includes use of limiters. Is it possible that it causes problems?
That's just my hunch, since I've got older snapshots running fine without limiters.

My config is fairly simple.
WAN and another WAN on OPT.
A couple NAT/FW rules inbound.
Nothing else.

Offline xbipin

  • Hero Member
  • *****
  • Posts: 1631
  • Karma: +6/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #39 on: February 22, 2010, 02:27:15 am »
my alix with 20th feb snapshot works perfect and older versions also have been running stable enough for me for as much as 15 days then its no crash but i usually endup trying newer snapshots.

Offline Uxorious

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #40 on: February 27, 2010, 12:58:58 pm »
Stopped at rn_match+0x17: movl 0xc(%eax),%ebx

It happened again some 20 hours later.
LAN dead again, but stopped at exactly the same instruction.
Since writing down the bactrace was too painful, I took a picture instead.

For the past 5 days I have been running completely stable on 1.2.3 using the same hardware and configuration (recreated configuration since downgrading is not possible).

Something bad is happening in 2.0 for sure...

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: pfSense crashed on Alix
« Reply #41 on: February 28, 2010, 02:23:22 pm »
Please tell your configuration or better send your config.xml to investigate further.

Offline computor

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #42 on: March 02, 2010, 09:10:02 pm »
I think I'm having a similar issue (trap 12s once or twice a day--more if torrenting, etc).  I thought it was a HW failure at first, but this crash has followed through 3 different boxes (a dual PIII, a single P4, and a dual Opteron blade).  I'm using nanoBSD and have upgraded several times to the latest snapshot--I'm probably a week out of date at most at the moment.  The faulting process is usually one of the NIC drivers (I don't think it's a driver problem--I've seen it on em, fxp, and bge), but once it was the openvpn process).  I have reflashed the card with a fresh nanobsd image a few times, so I don't think it's corruption.

I have a second box hooked up to the serial console doing a full dump of the serial console which at this point has captured over a dozen such crashes (and the subsequent reboots).  As such, it's quite large--I can email it to you if you want, Chris.  A least a couple include backtraces.

Will M.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11228
  • Karma: +894/-7
    • View Profile
    • Chris Buechler
Re: pfSense crashed on Alix
« Reply #43 on: March 03, 2010, 08:44:29 pm »
Please post any back traces here if you have them.

Offline computor

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: pfSense crashed on Alix
« Reply #44 on: March 05, 2010, 02:10:16 pm »
Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x8
fault code      = supervisor read, page not present
instruction pointer   = 0x20:0xc08f46ce
stack pointer           = 0x28:0xe59bb7c4
frame pointer           = 0x28:0xe59bb7f0
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, def32 1, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 11 (irq18: fxp0 ath0+)
[thread pid 11 tid 64026 ]
Stopped at      rn_match+0x4e:  cmpw    $0,0x8(%eax)
db> bt
Tracing pid 11 tid 64026 td 0xc3066b40
rn_match(c0cd4fcc,c3578400,0,0,e59bb8a4,...) at rn_match+0x4e
pfr_match_addr(c35769b0,c3b8702c,2,e59bb890,e59bb88c,...) at pfr_match_addr+0x63
pf_test_udp(e59bb98c,e59bb988,1,c3388500,c322b400,...) at pf_test_udp+0x4db
pf_test(1,c305f000,e59bbb50,0,0,...) at pf_test+0xbb5
init_pf_mutex(0,e59bbb50,c305f000,1,0,...) at init_pf_mutex+0x5e6
pfil_run_hooks(c0cfd140,e59bbba0,c305f000,1,0,...) at pfil_run_hooks+0x7e
ip_input(c322b400,246,c30ddc40,e59bbbc8,c06fd9b1,...) at ip_input+0x278
netisr_dispatch_src(1,0,c322b400,e59bbc00,c08e3ecf,...) at netisr_dispatch_src+0x89
netisr_dispatch(1,c322b400,c305f000,c305f000,c3b87012,...) at netisr_dispatch+0x20
ether_demux(c305f000,c322b400,3,0,3,...) at ether_demux+0x16f
ether_vlanencap(c305f000,c322b400,2,0,e59bbc5c,...) at ether_vlanencap+0x43f
fwohci_init(ffffffff,e59b0008,e59b0028,0,1816a67,...) at fwohci_init+0xb4fd
fwohci_init(c3135000,0,109,65fb90c2,e51f,...) at fwohci_init+0xc2f3
intr_event_execute_handlers(c3034aa0,c3061400,c0b5910d,4f6,c3061470,...) at intr_event_execute_handlers+0x14b
intr_getaffinity(c3134a60,e59bbd38,0,0,0,...) at intr_getaffinity+0x14a
fork_exit(c080dfe0,c3134a60,e59bbd38) at fork_exit+0x90
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe59bbd70, ebp = 0 ---
db> reboot


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x34
fault code      = supervisor read, page not present
instruction pointer   = 0x20:0xc094d84b
stack pointer           = 0x28:0xe6bf76a4
frame pointer           = 0x28:0xe6bf76dc
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, def32 1, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 438 (openvpn)
[thread pid 438 tid 64051 ]
Stopped at      arpresolve+0x3db:       movl    0x34(%edx),%eax
db> bt
Tracing pid 438 tid 64051 td 0xc32226c0
arpresolve(c305f000,0,c3360900,e6bf7768,e6bf7704,...) at arpresolve+0x3db
ether_output(c305f000,c3360900,e6bf7768,e6bf7760,c338b600,...) at ether_output+0xd2
pf_route(e6bf7a44,c359638c,2,c305f000,c35a0000,...) at pf_route+0x5ef
pf_test(2,c305f000,e6bf7a44,0,c34a2dc0,...) at pf_test+0x1a02
init_pf_mutex(0,e6bf7a44,c305f000,2,c34a2dc0,...) at init_pf_mutex+0x67c
pfil_run_hooks(c0cfd140,e6bf7aa8,c305f000,2,c34a2dc0,...) at pfil_run_hooks+0x7e
ip_output(c3360900,0,0,0,0,...) at ip_output+0x813
kmod_udpstat_inc(c355d338,0,c3360900,c32a7b90,0,...) at kmod_udpstat_inc+0x1023
sosend_dgram(c355d338,c32a7b90,e6bf7be8,c3360900,0,...) at sosend_dgram+0x463
sosend(c355d338,c32a7b90,e6bf7be8,0,0,...) at sosend+0x3f
kern_sendit(c32226c0,b,e6bf7c5c,0,0,...) at kern_sendit+0x107
kern_sendit(0,c32a7b90,10,e6bf7c78,1,...) at kern_sendit+0x400
sendto(c32226c0,e6bf7cf8,18,c32226c0,e6bf7d2c,...) at sendto+0x4f
syscall(e6bf7d38) at syscall+0x307
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x28388ddf, esp = 0xbfbfe50c, ebp = 0xbfbfe558 ---
db>

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x8
fault code      = supervisor read, page not present
instruction pointer   = 0x20:0xc08f46ce
stack pointer           = 0x28:0xeaa097c4
frame pointer           = 0x28:0xeaa097f0
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, def32 1, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 11 (irq17: fxp0)
[thread pid 11 tid 64030 ]
Stopped at      rn_match+0x4e:  cmpw    $0,0x8(%eax)
db> bt
Tracing pid 11 tid 64030 td 0xc41ae240
rn_match(c0cd4fcc,c45efb00,0,0,eaa098a4,...) at rn_match+0x4e
pfr_match_addr(c46439b0,c45e282c,2,eaa09890,eaa0988c,...) at pfr_match_addr+0x63
pf_test_udp(eaa0998c,eaa09988,1,c425a300,c4597200,...) at pf_test_udp+0x4db
pf_test(1,c418e000,eaa09b50,0,0,...) at pf_test+0xbb5
init_pf_mutex(0,eaa09b50,c418e000,1,0,...) at init_pf_mutex+0x5e6
pfil_run_hooks(c0cfd140,eaa09ba0,c418e000,1,0,...) at pfil_run_hooks+0x7e
ip_input(c4597200,80246,c40f1e40,eaa09bc8,c06fd9b1,...) at ip_input+0x278
netisr_dispatch_src(1,0,c4597200,eaa09c00,c08e3ecf,...) at netisr_dispatch_src+0x89
netisr_dispatch(1,c4597200,c418e000,c418e000,c45e2812,...) at netisr_dispatch+0x20
ether_demux(c418e000,c4597200,3,0,3,...) at ether_demux+0x16f
ether_vlanencap(c418e000,c4597200,2,c41ae240,eaa09c5c,...) at ether_vlanencap+0x43f
fwohci_init(ffffffff,eaa00008,eaa00028,0,135b809,...) at fwohci_init+0xb4fd
fwohci_init(c41b0000,0,109,34134c2,1ba30,...) at fwohci_init+0xc2f3
intr_event_execute_handlers(c4036aa0,c4034280,c0b5910d,4f6,c40342f0,...) at intr_event_execute_handlers+0x14b
intr_getaffinity(c4257380,eaa09d38,0,0,0,...) at intr_getaffinity+0x14a
fork_exit(c080dfe0,c4257380,eaa09d38) at fork_exit+0x90
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeaa09d70, ebp = 0 ---
db> reboot