The pfSense Store

Author Topic: Secondary Pfsense Crash after CARP Configuration  (Read 355 times)

0 Members and 1 Guest are viewing this topic.

Offline bilal0631

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Secondary Pfsense Crash after CARP Configuration
« on: September 26, 2017, 04:51:39 am »
I am stuck in an issues since last two weeks because my pfsense crashes and went into to endless loop of reboot whenever i configure CARP on seconday pfsense . Even i have tested on different hardwares and different versions old and latest of pfsense . Please if someone can guide 

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21409
  • Karma: +1437/-26
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #1 on: September 28, 2017, 10:46:00 am »
You must provide more detail about the crashes. Can you see any part of the error/backtrace/etc that happens? If you disconnect the sync cable does the reboot loop stop?

Are you using any limiters? (Limiters + pfsync is a known panic trigger for HA)
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #2 on: December 07, 2017, 04:43:51 pm »
Hi

I have the same issue i am using 2.4.2 i followed the following video.  https://www.youtube.com/watch?v=gxIPHR-eX_U it works up until enabling carp then send my pfsense in a massive boot loops.

I do have limiters so maybe that could be the issue im facing.

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #3 on: December 07, 2017, 04:44:23 pm »
You must provide more detail about the crashes. Can you see any part of the error/backtrace/etc that happens? If you disconnect the sync cable does the reboot loop stop?

Are you using any limiters? (Limiters + pfsync is a known panic trigger for HA)

Did you fix this?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #4 on: December 07, 2017, 08:33:02 pm »
No it is still an open bug.

Delete the limiters or disable pfsync.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #5 on: December 08, 2017, 04:43:40 am »
No it is still an open bug.

Delete the limiters or disable pfsync.

Can you not uncheck the toggle for Traffic Shaper Limiters configuration & Traffic Shaper configuration?

Mat

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #6 on: December 08, 2017, 11:32:59 am »
If whatever you try works, yes.

It is a problem specific to limiters. The shaper itself (altq) works fine.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #7 on: December 09, 2017, 05:29:59 am »
Thanks. I did try without it checked but still crashed. Iíll delete the limiters and try that instead.

Mat

Offline EdCazini

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #8 on: December 09, 2017, 06:46:37 am »
Hi,

I solved mine with this ... https://redmine.pfsense.org/issues/4310#note-44

Edwin

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #9 on: December 09, 2017, 11:33:00 am »
Hi again

i have deleted all my limiters and it still crashes.  here is the crash report.  HELP Please!

               Crash report begins.  Anonymous machine information:

amd64
11.1-RELEASE-p4
FreeBSD 11.1-RELEASE-p4 #5 r313908+79c92265a31(RELENG_2_4): Mon Nov 20 08:18:22 CST 2017     root@buildbot2.netgate.com:/builder/ce-242/tmp/obj/builder/ce-242/tmp/FreeBSD-src/sys/pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device: /dev/gptid/78cabbc4-dad6-11e7-b9e4-005056b3ced1
  Architecture: amd64
  Architecture Version: 1
  Dump Length: 106496
  Blocksize: 512
  Dumptime: Sat Dec  9 17:23:14 2017
  Hostname: srvtcfw01
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.1-RELEASE-p4 #5 r313908+79c92265a31(RELENG_2_4): Mon Nov 20 08:18:22 CST 2017
    root@buildbot2.netgate.com:/builder/ce-242/tmp/obj/builder/ce-242/tmp/FreeBSD-src/sys/pfSense
  Panic String: pfsync_undefer_state: unable to find deferred state
  Dump Parity: 583539232
  Bounds: 0
  Dump Status: good

Filename: /var/crash/info.last
Dump header from device: /dev/gptid/78cabbc4-dad6-11e7-b9e4-005056b3ced1
  Architecture: amd64
  Architecture Version: 1
  Dump Length: 106496
  Blocksize: 512
  Dumptime: Sat Dec  9 17:23:14 2017
  Hostname: srvtcfw01
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.1-RELEASE-p4 #5 r313908+79c92265a31(RELENG_2_4): Mon Nov 20 08:18:22 CST 2017
    root@buildbot2.netgate.com:/builder/ce-242/tmp/obj/builder/ce-242/tmp/FreeBSD-src/sys/pfSense
  Panic String: pfsync_undefer_state: unable to find deferred state
  Dump Parity: 583539232
  Bounds: 0
  Dump Status: good

Filename: /var/crash/minfree
2048

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000013213016002  7056 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x7ebf00
curthread    = 0xfffff80007d6f000: pid 15262 "openvpn"
curpcb       = 0xfffffe0096381cc0
fpcurthread  = 0xfffff80007d6f000: pid 15262 "openvpn"
idlethread   = 0xfffff8000351d000: tid 100003 "idle: cpu0"
curpmap      = 0xfffff80007d47138
tssp         = 0xffffffff82a73b90
commontssp   = 0xffffffff82a73b90
rsp0         = 0xfffffe0096381cc0
gs32p        = 0xffffffff82a7a3e8
ldt          = 0xffffffff82a7a428
tss          = 0xffffffff82a7a418
db:0:kdb.enter.default>  bt
Tracing pid 15262 tid 100109 td 0xfffff80007d6f000
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0096381250
vpanic() at vpanic+0x1a3/frame 0xfffffe00963812d0
panic() at panic+0x43/frame 0xfffffe0096381330
pfsync_update_state() at pfsync_update_state+0x3c5/frame 0xfffffe0096381380
pf_test() at pf_test+0x21cf/frame 0xfffffe00963815c0
pf_check_out() at pf_check_out+0x1d/frame 0xfffffe00963815e0
pfil_run_hooks() at pfil_run_hooks+0x7b/frame 0xfffffe0096381670
ip_output() at ip_output+0x22b/frame 0xfffffe00963817c0
ip_forward() at ip_forward+0x323/frame 0xfffffe0096381860
ip_input() at ip_input+0x75a/frame 0xfffffe00963818c0
netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe0096381910
tunwrite() at tunwrite+0x226/frame 0xfffffe0096381950
devfs_write_f() at devfs_write_f+0xe2/frame 0xfffffe00963819b0
dofilewrite() at dofilewrite+0xc8/frame 0xfffffe0096381a00
sys_writev() at sys_writev+0x8c/frame 0xfffffe0096381a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096381bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096381bf0
--- syscall (121, FreeBSD ELF64, sys_writev), rip = 0x8015a581a, rsp = 0x7fffffffde18, rbp = 0x7fffffffde50 ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
51715   308   308     0  S       nanslp   0xffffffff828b51e0 php-fpm
49963 70428   308     0  S       nanslp   0xffffffff828b51e0 sleep
71262 71220 69187     0  S       nanslp   0xffffffff828b51e0 sleep
71220     1 69187     0  S       wait     0xfffff8001c105000 sh
 9738  9585  9738 65534  Ss      sbwait   0xfffff800079fa4a4 darkstat
 9585     1  9585 65534  Ss      select   0xfffff8001c237b40 darkstat
 7039 71757  7039     0  Ss      (threaded)                  sshlockout_pf
100091                   S       piperd   0xfffff8001c39f8e8 sshlockout_pf
100169                   S       nanslp   0xffffffff828b51e0 sshlockout_pf
 6749     1  6749     0  Ss      select   0xfffff800074f9b40 bsnmpd
70428     1   308     0  S       wait     0xfffff80007ea4000 sh
69866     1 69866   136  Ss      select   0xfffff80007f20cc0 dhcpd
63516     1 63516    59  Ss      kqread   0xfffff80007744900 unbound
52767     1 52767     0  Ss      (threaded)                  dpinger
100156                   S       uwait    0xfffff8001c278080 dpinger
100165                   S       sbwait   0xfffff800079f8144 dpinger
100166                   S       nanslp   0xffffffff828b51e0 dpinger
100167                   S       nanslp   0xffffffff828b51e0 dpinger
100168                   S       accept   0xfffff800079f906c dpinger
52697     1 52697     0  Ss      (threaded)                  dpinger
100155                   S       uwait    0xfffff8001c2fbf00 dpinger
100161                   S       sbwait   0xfffff80007be1144 dpinger
100162                   S       nanslp   0xffffffff828b51e0 dpinger
100163                   S       nanslp   0xffffffff828b51e0 dpinger
100164                   S       accept   0xfffff800076bc3cc dpinger
  415     1   415     0  Ss+     ttyin    0xfffff80007504cb0 getty
  407     1   407     0  Ss+     ttyin    0xfffff800075050b0 getty
  123     1   123     0  Ss+     ttyin    0xfffff800075054b0 getty
99998     1 99998     0  Ss+     ttyin    0xfffff800075058b0 getty
99937     1 99937     0  Ss+     ttyin    0xfffff800074d78b0 getty
99644     1 99644     0  Ss+     ttyin    0xfffff800074b00b0 getty
99639     1 99639     0  Ss+     ttyin    0xfffff800074ae8b0 getty
99388     1 99388     0  Ss+     ttyin    0xfffff800074ae4b0 getty
82736     1 82213     0  S       select   0xfffff8001c0ec340 vmtoolsd
79421 78825 78825     0  S       nanslp   0xffffffff828b51e0 minicron
78825     1 78825     0  Ss      wait     0xfffff800079c3000 minicron
77900 77877 77877     0  S       nanslp   0xffffffff828b51e0 minicron
77877     1 77877     0  Ss      wait     0xfffff800079c4000 minicron
77783 77255 77255     0  S       nanslp   0xffffffff828b51e0 minicron
77255     1 77255     0  Ss      wait     0xfffff800079c5000 minicron
71757     1 71757     0  Ss      select   0xfffff80007617cc0 syslogd
32516     1 32516     0  Ss      (threaded)                  ntpd
100117                   S       select   0xfffff80007daae40 ntpd
31992     1 31992     0  Ss      nanslp   0xffffffff828b51e0 cron
31697 31441 31441     0  S       kqread   0xfffff80007d5f100 nginx
31679 31441 31441     0  S       kqread   0xfffff80007d5f200 nginx
31441     1 31441     0  Ss      pause    0xfffff8000754d630 nginx
15591     1 15591     0  Ss      bpf      0xfffff80007bdca00 filterlog
15262     1 15262     0  Rs      CPU 0                       openvpn
14091     1 14091     0  Ss      select   0xfffff80007cde3c0 openvpn
 7334     1  7334     0  Ss      select   0xfffff800074f71c0 sshd
  336     1   336     0  Ss      select   0xfffff800074f60c0 devd
  324   322   322     0  S       kqread   0xfffff80007745600 check_reload_status
  322     1   322     0  Ss      kqread   0xfffff80007745500 check_reload_status
  308     1   308     0  Ss      kqread   0xfffff80007746100 php-fpm
   58     0     0     0  DL      mdwait   0xfffff80007524800 [md0]
   25     0     0     0  DL      syncer   0xffffffff829aee00 [syncer]
   24     0     0     0  DL      vlruwt   0xfffff8000754a588 [vnlru]
   23     0     0     0  DL      (threaded)                  [bufdaemon]
100083                   D       psleep   0xffffffff829ad604 [bufdaemon]
100092                   D       sdflush  0xfffff8000752b8e8 [/ worker]
   22     0     0     0  DL      -        0xffffffff829ae2bc [bufspacedaemon]
   21     0     0     0  DL      pgzero   0xffffffff829c2a64 [pagezero]
   20     0     0     0  DL      psleep   0xffffffff829bef14 [vmdaemon]
   19     0     0     0  DL      (threaded)                  [pagedaemon]
100079                   D       psleep   0xffffffff82a72f85 [pagedaemon]
100086                   D       launds   0xffffffff829beec4 [laundry: dom0]
100087                   D       umarcl   0xffffffff829be838 [uma]
   18     0     0     0  DL      -        0xffffffff829ace14 [soaiod4]
   17     0     0     0  DL      -        0xffffffff829ace14 [soaiod3]
   16     0     0     0  DL      -        0xffffffff829ace14 [soaiod2]
   15     0     0     0  DL      -        0xffffffff829ace14 [soaiod1]
    9     0     0     0  DL      -        0xffffffff82789700 [rand_harvestq]
    8     0     0     0  DL      pftm     0xffffffff80e930b0 [pf purge]
    7     0     0     0  DL      waiting_ 0xffffffff82a61d70 [sctp_iterator]
    6     0     0     0  DL      -        0xfffff800039c4448 [fdc0]
    5     0     0     0  DL      idle     0xfffffe0000ee1000 [mpt_recovery0]
    4     0     0     0  DL      (threaded)                  [cam]
100020                   D       -        0xffffffff8265c480 [doneq0]
100074                   D       -        0xffffffff8265c2c8 [scanner]
    3     0     0     0  DL      crypto_r 0xffffffff829bd3f0 [crypto returns]
    2     0     0     0  DL      crypto_w 0xffffffff829bd298 [crypto]
   14     0     0     0  DL      (threaded)                  [geom]
100014                   D       -        0xffffffff82a39e20 [g_event]
100015                   D       -        0xffffffff82a39e28 [g_up]
100016                   D       -        0xffffffff82a39e30 [g_down]
   13     0     0     0  DL      sleep    0xffffffff82615c70 [ng_queue0]
   12     0     0     0  WL      (threaded)                  [intr]
100004                   I                                   [swi1: netisr 0]
100005                   I                                   [swi3: vm]
100006                   I                                   [swi4: clock (0)]
100008                   I                                   [swi6: task queue]
100009                   I                                   [swi6: Giant taskq]
100012                   I                                   [swi5: fast taskq]
100021                   I                                   [irq14: ata0]
100022                   I                                   [irq15: ata1]
100023                   I                                   [irq17: mpt0]
100025                   I                                   [irq256: ahci0]
100026                   I                                   [irq257: pcib3]
100027                   I                                   [irq258: vmx0]
100028                   I                                   [irq259: pcib4]
100029                   I                                   [irq260: pcib5]
100030                   I                                   [irq261: pcib6]
100031                   I                                   [irq262: pcib7]
100032                   I                                   [irq263: pcib8]
100033                   I                                   [irq264: pcib9]
100034                   I                                   [irq265: pcib10]
100035                   I                                   [irq266: pcib11]
100036                   I                                   [irq267: vmx1]
100037                   I                                   [irq268: pcib12]
100038                   I                                   [irq269: pcib13]
100039                   I                                   [irq270: pcib14]
100040                   I                                   [irq271: pcib15]
100041                   I                                   [irq272: pcib16]
100042                   I                                   [irq273: pcib17]
100043                   I                                   [irq274: pcib18]
100044                   I                                   [irq275: pcib19]
100045                   I                                   [irq276: vmx2]
100046                   I                                   [irq277: pcib20]
100047                   I                                   [irq278: pcib21]
100048                   I                                   [irq279: pcib22]
100049                   I                                   [irq280: pcib23]
100050                   I                                   [irq281: pcib24]
100051                   I                                   [irq282: pcib25]
100052                   I                                   [irq283: pcib26]
100053                   I                                   [irq284: pcib27]
100054                   I                                   [irq285: pcib28]
100055                   I                                   [irq286: pcib29]
100056                   I                                   [irq287: pcib30]
100057                   I                                   [irq288: pcib31]
100058                   I                                   [irq289: pcib32]
100059                   I                                   [irq290: pcib33]
100060                   I                                   [irq291: pcib34]
100061                   I                                   [irq1: atkbd0]
100062                   I                                   [irq12: psm0]
100067                   I                                   [swi1: pf send]
100068                   I                                   [swi1: pfsync]
   11     0     0     0  RL                                  [idle: cpu0]
    1     0     1     0  SLs     wait     0xfffff80003518588 [init]
   10     0     0     0  DL      audit_wo 0xffffffff82a68f40 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin   0xffffffff82a39e68 [swapper]
100007                   D       -        0xfffff80003507900 [kqueue_ctx taskq]
100010                   D       -        0xfffff80003507100 [aiod_kick taskq]
100011                   D       -        0xfffff80003506e00 [thread taskq]
100017                   D       -        0xfffff8000354a100 [firmware taskq]
100064                   D       -        0xfffff8000705a400 [mca taskq]
100069                   D       -        0xfffff800074b5e00 [acpi_task_0]
100070                   D       -        0xfffff800074b5e00 [acpi_task_1]
100071                   D       -        0xfffff800074b5e00 [acpi_task_2]
100073                   D       -        0xfffff80003548900 [CAM taskq]
100110                   D       -        0xfffff80007d1f300 [dummynet]
db:0:kdb.enter.default>  alltrace

Tracing command php-fpm pid 51715 tid 100131 td 0xfffff8001c0f4560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963ea830
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963ea860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963ea8d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963ea910
_sleep() at _sleep+0x223/frame 0xfffffe00963ea990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe00963eaa20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe00963eaa60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963eabf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963eabf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d8ff7a, rsp = 0x7fffffffae68, rbp = 0x7fffffffae90 ---

Tracing command sleep pid 49963 tid 100108 td 0xfffff800075d5560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009637c830
mi_switch() at mi_switch+0xe5/frame 0xfffffe009637c860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009637c8d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe009637c910
_sleep() at _sleep+0x223/frame 0xfffffe009637c990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe009637ca20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe009637ca60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009637cbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009637cbf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800904f7a, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command sleep pid 71262 tid 100103 td 0xfffff80007aa6000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096363830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096363860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963638d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096363910
_sleep() at _sleep+0x223/frame 0xfffffe0096363990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096363a20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe0096363a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096363bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096363bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800904f7a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command sh pid 71220 tid 100129 td 0xfffff8001c0f5000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963e5680
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963e56b0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963e5720
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963e5750
_sleep() at _sleep+0x248/frame 0xfffffe00963e57d0
kern_wait6() at kern_wait6+0x267/frame 0xfffffe00963e5870
sys_wait4() at sys_wait4+0x78/frame 0xfffffe00963e5a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963e5bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963e5bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b62dda, rsp = 0x7fffffffe5a8, rbp = 0x7fffffffe660 ---

Tracing command darkstat pid 9738 tid 100120 td 0xfffff80007f19560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963b8770
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963b87a0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963b8810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963b8840
_sleep() at _sleep+0x248/frame 0xfffffe00963b88c0
sbwait() at sbwait+0x54/frame 0xfffffe00963b88e0
soreceive_generic() at soreceive_generic+0x3bb/frame 0xfffffe00963b89a0
dofileread() at dofileread+0xba/frame 0xfffffe00963b89e0
sys_read() at sys_read+0xdb/frame 0xfffffe00963b8a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963b8bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963b8bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800df02da, rsp = 0x7fffffffe518, rbp = 0x7fffffffeae0 ---

Tracing command darkstat pid 9585 tid 100115 td 0xfffff80007d6c000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009639f6d0
mi_switch() at mi_switch+0xe5/frame 0xfffffe009639f700
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009639f770
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009639f7a0
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe009639f7f0
seltdwait() at seltdwait+0xc5/frame 0xfffffe009639f830
kern_select() at kern_select+0x86a/frame 0xfffffe009639fa20
sys_select() at sys_select+0x56/frame 0xfffffe009639fa60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009639fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009639fbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800df02ba, rsp = 0x7fffffffeae8, rbp = 0x7fffffffec40 ---

Tracing command sshlockout_pf pid 7039 tid 100091 td 0xfffff800075d5000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963277f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096327820
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096327890
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963278c0
_sleep() at _sleep+0x248/frame 0xfffffe0096327940
pipe_read() at pipe_read+0x26d/frame 0xfffffe00963279a0
dofileread() at dofileread+0xba/frame 0xfffffe00963279e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096327a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096327bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096327bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b952da, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---

Tracing command sshlockout_pf pid 7039 tid 100169 td 0xfffff8001c0cf560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00964a5830
mi_switch() at mi_switch+0xe5/frame 0xfffffe00964a5860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964a58d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00964a5910
_sleep() at _sleep+0x223/frame 0xfffffe00964a5990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe00964a5a20
sys_nanosleep() at sys_nanosleep+0x97/frame 0xfffffe00964a5a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00964a5bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00964a5bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2df7a, rsp = 0x7fffdfffdf38, rbp = 0x7fffdfffdf60 ---

Tracing command bsnmpd pid 6749 tid 100126 td 0xfffff80007d6f560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963d66b0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963d66e0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963d6750
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963d6790
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a3/frame 0xfffffe00963d67f0
seltdwait() at seltdwait+0x7e/frame 0xfffffe00963d6830
kern_select() at kern_select+0x86a/frame 0xfffffe00963d6a20
sys_select() at sys_select+0x56/frame 0xfffffe00963d6a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963d6bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963d6bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fa82ba, rsp = 0x7fffffffc348, rbp = 0x7fffffffc520 ---

Tracing command sh pid 70428 tid 100118 td 0xfffff80007f1a560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963ae680
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963ae6b0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963ae720
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963ae750
_sleep() at _sleep+0x248/frame 0xfffffe00963ae7d0
kern_wait6() at kern_wait6+0x267/frame 0xfffffe00963ae870
sys_wait4() at sys_wait4+0x78/frame 0xfffffe00963aea60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963aebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963aebf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b62dda, rsp = 0x7fffffffe878, rbp = 0x7fffffffe930 ---

Tracing command dhcpd pid 69866 tid 100119 td 0xfffff80007f1a000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963b36b0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963b36e0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963b3750
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963b3790
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a3/frame 0xfffffe00963b37f0
seltdwait() at seltdwait+0x7e/frame 0xfffffe00963b3830
kern_select() at kern_select+0x86a/frame 0xfffffe00963b3a20
sys_select() at sys_select+0x56/frame 0xfffffe00963b3a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963b3bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963b3bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d5f2ba, rsp = 0x7fffffffea18, rbp = 0x7fffffffea40 ---

Tracing command unbound pid 63516 tid 100158 td 0xfffff8001c274000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096482600
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096482630
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964826a0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00964826e0
_sleep() at _sleep+0x223/frame 0xfffffe0096482760
kqueue_kevent() at kqueue_kevent+0x294/frame 0xfffffe0096482940
kern_kevent() at kern_kevent+0xb5/frame 0xfffffe00964829b0
sys_kevent() at sys_kevent+0x138/frame 0xfffffe0096482a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096482bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096482bf0
--- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b75fba, rsp = 0x7fffffffea18, rbp = 0x7fffffffea60 ---

Tracing command dpinger pid 52767 tid 100156 td 0xfffff8001c275000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00964787f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096478820
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096478890
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00964788c0
_sleep() at _sleep+0x248/frame 0xfffffe0096478940
umtxq_sleep() at umtxq_sleep+0x145/frame 0xfffffe00964789a0
do_wait() at do_wait+0x3b1/frame 0xfffffe0096478a20
__umtx_op_wait() at __umtx_op_wait+0x4d/frame 0xfffffe0096478a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096478bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096478bf0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083ec7c, rsp = 0x7fffffffe918, rbp = 0x7fffffffe9a0 ---

Tracing command dpinger pid 52767 tid 100165 td 0xfffff8001c30a560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096491700
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096491730
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964917a0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00964917d0
_sleep() at _sleep+0x248/frame 0xfffffe0096491850
sbwait() at sbwait+0x54/frame 0xfffffe0096491870
soreceive_generic() at soreceive_generic+0x3bb/frame 0xfffffe0096491930
kern_recvit() at kern_recvit+0x1dc/frame 0xfffffe00964919f0
sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe0096491a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096491bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096491bf0
--- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b4166a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---

Tracing command dpinger pid 52767 tid 100166 td 0xfffff8001c30a000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096496830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096496860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964968d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096496910
_sleep() at _sleep+0x223/frame 0xfffffe0096496990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096496a20
sys_nanosleep() at sys_nanosleep+0x97/frame 0xfffffe0096496a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096496bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096496bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b30f7a, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 ---

Tracing command dpinger pid 52767 tid 100167 td 0xfffff8001c12b560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009649b830
mi_switch() at mi_switch+0xe5/frame 0xfffffe009649b860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009649b8d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe009649b910
_sleep() at _sleep+0x223/frame 0xfffffe009649b990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe009649ba20
sys_nanosleep() at sys_nanosleep+0x97/frame 0xfffffe009649ba60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009649bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009649bbf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b30f7a, rsp = 0x7fffdfbfbf18, rbp = 0x7fffdfbfbf40 ---

Tracing command dpinger pid 52767 tid 100168 td 0xfffff8001c210000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00964a07f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00964a0820
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964a0890
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00964a08c0
_sleep() at _sleep+0x248/frame 0xfffffe00964a0940
kern_accept4() at kern_accept4+0x1a2/frame 0xfffffe00964a0a00
accept1() at accept1+0xed/frame 0xfffffe00964a0a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00964a0bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00964a0bf0
--- syscall (541, FreeBSD ELF64, sys_accept4), rip = 0x800b3105a, rsp = 0x7fffdf9fae68, rbp = 0x7fffdf9faea0 ---

Tracing command dpinger pid 52697 tid 100155 td 0xfffff8001c275560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00964737f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096473820
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096473890
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00964738c0
_sleep() at _sleep+0x248/frame 0xfffffe0096473940
umtxq_sleep() at umtxq_sleep+0x145/frame 0xfffffe00964739a0
do_wait() at do_wait+0x3b1/frame 0xfffffe0096473a20
__umtx_op_wait() at __umtx_op_wait+0x4d/frame 0xfffffe0096473a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096473bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096473bf0
--- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083ec7c, rsp = 0x7fffffffe928, rbp = 0x7fffffffe9b0 ---

Tracing command dpinger pid 52697 tid 100161 td 0xfffff80007d6e000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009638b700
mi_switch() at mi_switch+0xe5/frame 0xfffffe009638b730
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009638b7a0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009638b7d0
_sleep() at _sleep+0x248/frame 0xfffffe009638b850
sbwait() at sbwait+0x54/frame 0xfffffe009638b870
soreceive_generic() at soreceive_generic+0x3bb/frame 0xfffffe009638b930
kern_recvit() at kern_recvit+0x1dc/frame 0xfffffe009638b9f0
sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe009638ba60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009638bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009638bbf0
--- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b4166a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---

Tracing command dpinger pid 52697 tid 100162 td 0xfffff80007d6d560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096390830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096390860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963908d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096390910
_sleep() at _sleep+0x223/frame 0xfffffe0096390990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096390a20
sys_nanosleep() at sys_nanosleep+0x97/frame 0xfffffe0096390a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096390bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096390bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b30f7a, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 ---

Tracing command dpinger pid 52697 tid 100163 td 0xfffff80007d6d000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096395830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096395860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963958d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096395910
_sleep() at _sleep+0x223/frame 0xfffffe0096395990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096395a20
sys_nanosleep() at sys_nanosleep+0x97/frame 0xfffffe0096395a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096395bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096395bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b30f7a, rsp = 0x7fffdfbfbf18, rbp = 0x7fffdfbfbf40 ---

Tracing command dpinger pid 52697 tid 100164 td 0xfffff80007d6c560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009639a7f0
mi_switch() at mi_switch+0xe5/frame 0xfffffe009639a820
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009639a890
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009639a8c0
_sleep() at _sleep+0x248/frame 0xfffffe009639a940
kern_accept4() at kern_accept4+0x1a2/frame 0xfffffe009639aa00
accept1() at accept1+0xed/frame 0xfffffe009639aa60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009639abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009639abf0
--- syscall (541, FreeBSD ELF64, sys_accept4), rip = 0x800b3105a, rsp = 0x7fffdf9fae68, rbp = 0x7fffdf9faea0 ---

Tracing command getty pid 415 tid 100150 td 0xfffff8001c129000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009645a760
mi_switch() at mi_switch+0xe5/frame 0xfffffe009645a790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009645a800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009645a830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe009645a880
tty_wait() at tty_wait+0x1c/frame 0xfffffe009645a8a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe009645a900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe009645a940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe009645a9a0
dofileread() at dofileread+0xba/frame 0xfffffe009645a9e0
sys_read() at sys_read+0xdb/frame 0xfffffe009645aa60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009645abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009645abf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 407 tid 100149 td 0xfffff8001c129560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096455760
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096455790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096455800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096455830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe0096455880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00964558a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe0096455900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe0096455940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00964559a0
dofileread() at dofileread+0xba/frame 0xfffffe00964559e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096455a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096455bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096455bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 123 tid 100148 td 0xfffff8001c12a000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096450760
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096450790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096450800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096450830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe0096450880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00964508a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe0096450900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe0096450940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00964509a0
dofileread() at dofileread+0xba/frame 0xfffffe00964509e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096450a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096450bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096450bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 99998 tid 100147 td 0xfffff8001c12a560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009644b760
mi_switch() at mi_switch+0xe5/frame 0xfffffe009644b790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009644b800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009644b830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe009644b880
tty_wait() at tty_wait+0x1c/frame 0xfffffe009644b8a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe009644b900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe009644b940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe009644b9a0
dofileread() at dofileread+0xba/frame 0xfffffe009644b9e0
sys_read() at sys_read+0xdb/frame 0xfffffe009644ba60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009644bbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009644bbf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 99937 tid 100146 td 0xfffff8001c12b000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096446760
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096446790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096446800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096446830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe0096446880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00964468a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe0096446900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe0096446940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00964469a0
dofileread() at dofileread+0xba/frame 0xfffffe00964469e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096446a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096446bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096446bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 99644 tid 100135 td 0xfffff8001c0f2560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963fe760
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963fe790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963fe800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963fe830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe00963fe880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00963fe8a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe00963fe900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe00963fe940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00963fe9a0
dofileread() at dofileread+0xba/frame 0xfffffe00963fe9e0
sys_read() at sys_read+0xdb/frame 0xfffffe00963fea60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963febf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963febf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 99639 tid 100088 td 0xfffff80007550560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096318760
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096318790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096318800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096318830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe0096318880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00963188a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe0096318900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe0096318940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00963189a0
dofileread() at dofileread+0xba/frame 0xfffffe00963189e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096318a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096318bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096318bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command getty pid 99388 tid 100145 td 0xfffff8001c0f6560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096441760
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096441790
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096441800
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096441830
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe0096441880
tty_wait() at tty_wait+0x1c/frame 0xfffffe00964418a0
ttydisc_read() at ttydisc_read+0x213/frame 0xfffffe0096441900
ttydev_read() at ttydev_read+0x67/frame 0xfffffe0096441940
devfs_read_f() at devfs_read_f+0xdc/frame 0xfffffe00964419a0
dofileread() at dofileread+0xba/frame 0xfffffe00964419e0
sys_read() at sys_read+0xdb/frame 0xfffffe0096441a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096441bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096441bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b852da, rsp = 0x7fffffffec88, rbp = 0x7fffffffecd0 ---

Tracing command vmtoolsd pid 82736 tid 100123 td 0xfffff80007f15000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963c7730
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963c7760
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963c77d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963c7810
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a3/frame 0xfffffe00963c7870
seltdwait() at seltdwait+0x7e/frame 0xfffffe00963c78b0
kern_poll() at kern_poll+0x402/frame 0xfffffe00963c7a40
sys_poll() at sys_poll+0x50/frame 0xfffffe00963c7a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963c7bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963c7bf0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801f6e68a, rsp = 0x7fffffffeb08, rbp = 0x7fffffffeb40 ---

Tracing command minicron pid 79421 tid 100142 td 0xfffff800079f0000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096422830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096422860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964228d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096422910
_sleep() at _sleep+0x223/frame 0xfffffe0096422990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096422a20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe0096422a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096422bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096422bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800904f7a, rsp = 0x7fffffffeba8, rbp = 0x7fffffffebf0 ---

Tracing command minicron pid 78825 tid 100141 td 0xfffff800079f0560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009641d680
mi_switch() at mi_switch+0xe5/frame 0xfffffe009641d6b0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009641d720
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009641d750
_sleep() at _sleep+0x248/frame 0xfffffe009641d7d0
kern_wait6() at kern_wait6+0x267/frame 0xfffffe009641d870
sys_wait4() at sys_wait4+0x78/frame 0xfffffe009641da60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009641dbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009641dbf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800904dda, rsp = 0x7fffffffebf8, rbp = 0x7fffffffec30 ---

Tracing command minicron pid 77900 tid 100140 td 0xfffff800079f1000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096418830
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096418860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00964188d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe0096418910
_sleep() at _sleep+0x223/frame 0xfffffe0096418990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe0096418a20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe0096418a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096418bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096418bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800904f7a, rsp = 0x7fffffffebb8, rbp = 0x7fffffffec00 ---

Tracing command minicron pid 77877 tid 100139 td 0xfffff800079f1560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096413680
mi_switch() at mi_switch+0xe5/frame 0xfffffe00964136b0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096413720
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096413750
_sleep() at _sleep+0x248/frame 0xfffffe00964137d0
kern_wait6() at kern_wait6+0x267/frame 0xfffffe0096413870
sys_wait4() at sys_wait4+0x78/frame 0xfffffe0096413a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096413bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096413bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800904dda, rsp = 0x7fffffffec08, rbp = 0x7fffffffec40 ---

Tracing command minicron pid 77783 tid 100138 td 0xfffff800079f2000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe009640e830
mi_switch() at mi_switch+0xe5/frame 0xfffffe009640e860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe009640e8d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe009640e910
_sleep() at _sleep+0x223/frame 0xfffffe009640e990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe009640ea20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe009640ea60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe009640ebf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe009640ebf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800904f7a, rsp = 0x7fffffffebd8, rbp = 0x7fffffffec20 ---

Tracing command minicron pid 77255 tid 100137 td 0xfffff800079f2560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096409680
mi_switch() at mi_switch+0xe5/frame 0xfffffe00964096b0
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096409720
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0096409750
_sleep() at _sleep+0x248/frame 0xfffffe00964097d0
kern_wait6() at kern_wait6+0x267/frame 0xfffffe0096409870
sys_wait4() at sys_wait4+0x78/frame 0xfffffe0096409a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096409bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096409bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800904dda, rsp = 0x7fffffffec28, rbp = 0x7fffffffec60 ---

Tracing command syslogd pid 71757 tid 100090 td 0xfffff800074c3560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963226d0
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096322700
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe0096322770
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963227a0
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe00963227f0
seltdwait() at seltdwait+0xc5/frame 0xfffffe0096322830
kern_select() at kern_select+0x86a/frame 0xfffffe0096322a20
sys_select() at sys_select+0x56/frame 0xfffffe0096322a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe0096322bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0096322bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b892ba, rsp = 0x7fffffffdd68, rbp = 0x7fffffffec20 ---

Tracing command ntpd pid 32516 tid 100117 td 0xfffff80007d6b000
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963a96d0
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963a9700
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963a9770
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00963a97a0
_cv_wait_sig() at _cv_wait_sig+0x16e/frame 0xfffffe00963a97f0
seltdwait() at seltdwait+0xc5/frame 0xfffffe00963a9830
kern_select() at kern_select+0x86a/frame 0xfffffe00963a9a20
sys_select() at sys_select+0x56/frame 0xfffffe00963a9a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963a9bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963a9bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8017e82ba, rsp = 0x7fffffffe038, rbp = 0x7fffffffe070 ---

Tracing command cron pid 31992 tid 100116 td 0xfffff80007d6b560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe00963a4830
mi_switch() at mi_switch+0xe5/frame 0xfffffe00963a4860
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963a48d0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963a4910
_sleep() at _sleep+0x223/frame 0xfffffe00963a4990
kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe00963a4a20
sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe00963a4a60
amd64_syscall() at amd64_syscall+0x6c4/frame 0xfffffe00963a4bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00963a4bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d2ef7a, rsp = 0x7fffffffeab8, rbp = 0x7fffffffecb0 ---

Tracing command nginx pid 31697 tid 100100 td 0xfffff80007aa7560
sched_switch() at sched_switch+0x4aa/frame 0xfffffe0096354600
mi_switch() at mi_switch+0xe5/frame 0xfffffe0096354630
sleepq_catch_signals() at sleepq_catch_signals+0x464/frame 0xfffffe00963546a0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00963546e0
_sleep() at _sleep+0x223/frame 0xfffffe0096354760
kqueue_kevent() at kqueue_kevent+0x294/frame 0xfffffe0096354940
ke

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #10 on: December 09, 2017, 12:17:05 pm »
Also been reading that the WAN has to be on the same NIC interface on backup cluster?

Im using vmware on both boxes so does that mean same vswitch?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #11 on: December 09, 2017, 12:26:45 pm »
Also be sure you remove all the calls to the limiters in the rules.

Disable state syncing on both nodes and try again. Does it still crash? If so you might be looking at a different problem.

Quote
Also been reading that the WAN has to be on the same NIC interface on backup cluster?

ALL NICs have to be the same on both nodes in the same order. If WAN is igb0 on the primary, WAN has to be igb0 on the secondary, and so on. Generally not the source of a panic however, just "unexpected" behavior.

You might want to start again - small, and get WAN+LAN working in a very basic HA pair before moving on to more advanced configurations. They're VMs. It don't cost nothin'.

Both nodes have to be able to pass multicast between each other.

Inability to do so will not result in a crash, however, but a MASTER/MASTER split brain issue.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #12 on: December 09, 2017, 12:31:52 pm »
Thanks

I think the problem i have is the interfaces arent the same so ill have to try and move stuff around to get same interface names.

so it has to be the same physical nic its not based on virtual nic?

Mat

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9257
  • Karma: +1054/-308
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #13 on: December 09, 2017, 12:50:20 pm »
An interface has a physical name (em0, re0, igb0, xn0, igb0.1000, lagg2.1001) and an internal name (wan, lan, opt1, opt2, opt3, optX).

They all have to match exactly on both nodes.

Use Status > Interfaces to verify.

This is all covered in detail here: https://portal.pfsense.org/docs/book/highavailability/index.html
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Mat1987

  • Jr. Member
  • **
  • Posts: 61
  • Karma: +0/-0
    • View Profile
Re: Secondary Pfsense Crash after CARP Configuration
« Reply #14 on: December 09, 2017, 12:52:04 pm »
An interface has a physical name (em0, re0, igb0, xn0, igb0.1000, lagg2.1001) and an internal name (wan, lan, opt1, opt2, opt3, optX).

They all have to match exactly on both nodes.

Use Status > Interfaces to verify.

This is all covered in detail here: https://portal.pfsense.org/docs/book/highavailability/index.html

Painful lol.

Internally they are all named the same but physical there not so ill have to change some bits around.

few more days of playing then.