next »

[mdpugh]

Hello,
I have read everything I can find and tried every permutation I can think of to try to fix this problem to no avail.  Nevertheless, I apologize if this has been dealt with elsewhere--just please point me in the right direction.  I am trying to set up CARP on the LAN interface only.  My ISP will only assign IP addresses by DHCP and, while I can get more than one, they are, in general, not adjacent (I'm not even sure they're in the same subnet--I'll have to check), so I cannot readily set up CARP on the WAN.  I realize that the failure of the master firewall/gateway will interrupt any open sessions/connections, but at least the client can recover quickly.  I have set this configuration up on FreeBSD and OpenBSD several times with no problems and I am truly stumped as to what I am doing wrong.  I have followed both Chapter 20 of the book and the online tutorial to the letter, but it fails at the same point each time.  Everything proceeds normally until I configure the first CARP VIP on the master (the slave CARP VIP is not yet configured).  I configure it exactly as specified, but when I click Apply Changes, the firewall goes into a kernel panic and, after flashing a multitude of messages, reboots.  When it gets to the CARP configuration section of the boot process, it repeats the cycle ad infinitum.  I have saved the dumps generated each time, so I can provide them if necessary and no one has a Eureka moment from having read only my description of the problem.  I would really appreciate any insight.
Thanks,
Mike Pugh

[mdpugh]

Here's the panic string:

Panic String: Lock carp_if not exclusively locked @ /usr/pfSensesrc/src/sys/netinet/ip_carp.c:892

[mdpugh]

Complete crash dump:

Crash report begins.  Anonymous machine information:

i386
8.1-RELEASE-p4
FreeBSD 8.1-RELEASE-p4 #0: Tue Sep 13 16:51:59 EDT 2011     root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_Dev.8

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device /dev/ad0s1b
  Architecture: i386
  Architecture Version: 1
  Dump Length: 77312B (0 MB)
  Blocksize: 512
  Dumptime: Thu Jan  5 16:26:31 2012
  Hostname: heads.compughterworx.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 8.1-RELEASE-p4 #0: Tue Sep 13 16:51:59 EDT 2011
    root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_Dev.8
  Panic String: Lock carp_if not exclusively locked @ /usr/pfSensesrc/src/sys/netinet/ip_carp.c:892

  Dump Parity: 737169472
  Bounds: 0
  Dump Status: good

Filename: /var/crash/minfree
2048

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000011701347467  7104 ustarrootwheeldb:0:kdb.enter.panic>  run lockinfo
db:1:lockinfo> show locks
shared rw carp_if (carp_if) r = 0 (0xc513e650) locked @ /usr/pfSensesrc/src/sys/netinet/ip_carp.c:877
db:1:locks>  show alllocks
Process 12 (intr) thread 0xc41ab280 (64007)
shared rw carp_if (carp_if) r = 0 (0xc513e650) locked @ /usr/pfSensesrc/src/sys/netinet/ip_carp.c:877
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.panic>  show pcpu
cpuid        = 0
dynamic pcpu    = 0x536300
curthread    = 0xc41ab280: pid 12 "swi4: clock"
curpcb       = 0xc3df9d90
fpcurthread  = none
idlethread   = 0xc41aba00: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:
db:0:kdb.enter.panic>  bt
Tracing pid 12 tid 64007 td 0xc41ab280
kdb_enter(c0ebbaf1,c0ebbaf1,c0eb84c6,c3df9bac,0,...) at kdb_enter+0x3a
panic(c0eb84c6,c0ed7976,c0ed7885,37c,c3df9c30,...) at panic+0x136
_rw_assert(c513e650,4,c0ed7885,37c,c0ed7885,...) at _rw_assert+0xb1
carp_send_ad_locked(c513e650,c0ed7885,36d,c4e8952c,c3df9ca0,...) at carp_send_ad_locked+0x45
carp_send_ad(c4e89400,0,c0ebcf08,189,c1325bd8,...) at carp_send_ad+0x37
softclock(c1325ba0,c3df9cc8,c09f56a4,c1329940,c41e8138,...) at softclock+0x24a
intr_event_execute_handlers(c41a97f8,c41e8100,c0eb6c63,533,c41e8170,...) at intr_event_execute_handlers+0x125
ithread_loop(c41a8130,c3df9d38,c0eb69c8,344,c41a97f8,...) at ithread_loop+0x9f
fork_exit(c09de490,c41a8130,c3df9d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3df9d70, ebp = 0 ---
db:0:kdb.enter.panic>  ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
37155 23066 39852     0  S       nanslp   0xc1325b64 sleep
49573 47993 49573     0  S+      ttyin    0xc4260e70 sh
47993 47663 47993     0  S+      wait     0xc4eb1d48 sh
47663     1 47663     0  Ss+     wait     0xc4eb02a8 login
 4384 39852 39852     0  S       accept   0xc4d3c1da php
 4672 39852 39852     0  S       accept   0xc4d3c1da php
18573 21333 21333     0  S       piperd   0xc49267a8 rrdtool
23066     1 39852     0  S       wait     0xc4d40550 sh
21333     1 21333     0  Ss      select   0xc48f0924 apinger
15124     1 15018 65534  S       select   0xc4f55524 dnsmasq
10563     1 10563    65  Ss      select   0xc4d49d64 dhclient
 6066     1  6066     0  Ss      select   0xc4d49624 dhclient
57505 16589 57505     0  Ss      (threaded)          sshlockout_pf
 64114                   S       nanslp   0xc1325b64 sshlockout_pf
 64100                   S       piperd   0xc4ee6310 initial thread
29043     1 29043     0  Ss      nanslp   0xc1325b64 minicron
28903     1 28903     0  Ss      nanslp   0xc1325b64 minicron
28519     1 28519     0  Ss      nanslp   0xc1325b64 minicron
25653     1 25653     0  Ss      nanslp   0xc1325b64 cron
61941     1 61941     0  Ss      select   0xc48f0ae4 ntpd
55579     1    26   123  S+      select   0xc4d48864 ntpd
49912 39356 39356     0  S       accept   0xc49c8d1e php
49702 39356 39356     0  S       accept   0xc49c8d1e php
39852 38901 39852     0  Ss      wait     0xc4d3fd48 initial thread
39356 38901 39356     0  Ss      wait     0xc49f6550 initial thread
38901     1 38817     0  S       kqread   0xc4b1cb00 lighttpd
16589     1 16589     0  Ss      select   0xc48ef7a4 syslogd
11964     1 11964    65  Ss      select   0xc4d486a4 dhclient
54321     1 54321     0  Ss      select   0xc48efda4 inetd
41557     1   255     0  S       piperd   0xc4925188 logger
41538     1   255     0  S       bpf      0xc4d42000 tcpdump
34414     1 34414     0  Ss      select   0xc48ef424 dhclient
22390     1 22390    65  Ss      select   0xc48efbe4 dhclient
17018     1 17018     0  Ss      select   0xc48a9824 dhclient
14474     1 14474    65  Ss      select   0xc48ef964 dhclient
 8809     1  8809     0  Ss      select   0xc48efba4 dhclient
 8657     1  8657     0  Ss      select   0xc48a8864 sshd
  268     1   268     0  Ss      select   0xc48ef464 devd
  257   255   255     0  S       kqread   0xc48f7d80 check_reload_status
  255     1   255     0  Ss      kqread   0xc48f7480 check_reload_status
   41     0     0     0  SL      mdwait   0xc4934800 [md0]
   25     0     0     0  SL      flowclea 0xc1492568 [flowcleaner]
   24     0     0     0  SL      sdflush  0xc14ad2e0 [softdepflush]
   23     0     0     0  SL      syncer   0xc1492354 [syncer]
   22     0     0     0  SL      vlruwt   0xc41ead48 [vnlru]
   21     0     0     0  SL      psleep   0xc1492088 [bufdaemon]
   20     0     0     0  SL      pollid   0xc13250bc [idlepoll]
   19     0     0     0  SL      pgzero   0xc14adfb4 [pagezero]
   18     0     0     0  SL      psleep   0xc14adbdc [vmdaemon]
   17     0     0     0  SL      psleep   0xc14adba4 [pagedaemon]
   16     0     0     0  SL      ccb_scan 0xc12ef9d4 [xpt_thrd]
    9     0     0     0  SL      pftm     0xc04f85b0 [pfpurge]
    8     0     0     0  SL      waiting_ 0xc14997d8 [sctp_iterator]
    7     0     0     0  SL      -        0xc434a23c [fdc0]
   15     0     0     0  SL      (threaded)          usb
 64036                   D       -        0xc4314dac [usbus1]
 64035                   D       -        0xc4314d7c [usbus1]
 64034                   D       -        0xc4314d4c [usbus1]
 64033                   D       -        0xc4314d1c [usbus1]
 64032                   D       -        0xc42ffdac [usbus0]
 64031                   D       -        0xc42ffd7c [usbus0]
 64030                   D       -        0xc42ffd4c [usbus0]
 64029                   D       -        0xc42ffd1c [usbus0]
   14     0     0     0  SL      -        0xc13259c4 [yarrow]
    6     0     0     0  SL      crypto_r 0xc14ac88c [crypto returns]
    5     0     0     0  SL      crypto_w 0xc14ac868 [crypto]
    4     0     0     0  SL      -        0xc1323264 [g_down]
    3     0     0     0  SL      -        0xc1323260 [g_up]
    2     0     0     0  SL      -        0xc1323258 [g_event]
   13     0     0     0  SL      sleep    0xc12c5e40 [ng_queue0]
   12     0     0     0  RL      (threaded)          intr
 64046                   I                           [irq12: psm0]
 64045                   I                           [irq1: atkbd0]
 64044                   I                           [irq7: ppc0]
 64043                   I                           [swi0: uart uart]
 64028                   RunQ                        [irq10: uhci0 uhci1]
 64027                   I                           [irq15: ata1]
 64026                   I                           [irq14: ata0]
 64025                   I                           [irq11: acpi0]
 64024                   I                           [swi6: task queue]
 64023                   I                           [swi6: Giant taskq]
 64021                   I                           [swi5: +]
 64016                   I                           [swi2: cambio]
 64007                   Run     CPU 0               [swi4: clock]
 64006                   I                           [swi3: vm]
 64005                   I                           [swi1: netisr 0]
   11     0     0     0  RL                          [idle: cpu0]
    1     0     1     0  SLs     wait     0xc41a9d48 [init]
   10     0     0     0  SL      audit_wo 0xc14acc00 [audit]
    0     0     0     0  RLs     (threaded)          kernel
 64041                   RunQ                        [em4 taskq]
 64040                   D       -        0xc4348c80 [em3 taskq]
 64039                   RunQ                        [em2 taskq]
 64038                   D       -        0xc42f76c0 [em1 taskq]
 64037                   RunQ                        [em0 taskq]
 64022                   D       -        0xc42ab340 [thread taskq]
 64020                   D       -        0xc42ab400 [kqueue taskq]
 64019                   D       -        0xc42ab5c0 [acpi_task_2]
 64018                   D       -        0xc42ab5c0 [acpi_task_1]
 64017                   D       -        0xc42ab5c0 [acpi_task_0]
 64012                   D       -        0xc4190540 [firmware taskq]
 64001                   D       sched    0xc1323340 [swapper]
db:0:kdb.enter.panic>  alltrace

Tracing command sleep pid 37155 tid 64102 td 0xc4ed0c80
sched_switch(c4ed0c80,0,104,191,a9caa160,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4ed0c80,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(ea61,c0a3fe50,c4ed0c80,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,ea61,...) at _sleep+0x31e
kern_nanosleep(c4ed0c80,e968cc64,e968cc6c,3c,0,...) at kern_nanosleep+0xc1
nanosleep(c4ed0c80,e968ccf8,c,c4ed0c80,c4d40aa0,...) at nanosleep+0x6f
syscall(e968cd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28165f17, esp = 0xbfbfedac, ebp = 0xbfbfede8 ---

Tracing command sh pid 49573 tid 64105 td 0xc4ed0500
sched_switch(c4ed0500,0,104,191,fbec2869,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4ed0500,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,c4ed0500,c4ed0500,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4260e70,0,c0ec465b,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c4260e70,c1324e50,c0ec3fe6,4fd,0,...) at _cv_wait_sig+0x240
tty_wait(c4260e00,c4260e70,1,e96a7b87,c12bd060,...) at tty_wait+0x71
ttydisc_read(c4260e00,e96a7c58,0,9e,0,...) at ttydisc_read+0xef
ttydev_read(c4377500,e96a7c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c491cb28,e96a7c58,c4f69980,0,c4ed0500,...) at devfs_read_f+0x7e
dofileread(e96a7c58,ffffffff,ffffffff,0,c491cb28,...) at dofileread+0x96
kern_readv(c4ed0500,0,e96a7c58,e96a7c78,1,...) at kern_readv+0x58
read(c4ed0500,e96a7cf8,c0eff73e,c0e9e08f,c4ef5aa0,...) at read+0x4f
syscall(e96a7d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281f3593, esp = 0xbfbfe96c, ebp = 0xbfbfea48 ---

Tracing command sh pid 47993 tid 64117 td 0xc4f01a00
sched_switch(c4f01a00,0,104,191,d2e1753a,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4f01a00,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4eb1d48,5c,c0ec287c,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4eb1d48,c4eb1dd0,15c,c0ec287c,0,...) at _sleep+0x354
kern_wait(c4f01a00,ffffffff,e96cbc74,2,0,...) at kern_wait+0xb76
wait4(c4f01a00,e96cbcf8,c0eff73e,c0ec285b,c4eb1d48,...) at wait4+0x3b
syscall(e96cbd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816469b, esp = 0xbfbfea7c, ebp = 0xbfbfea98 ---

Tracing command login pid 47663 tid 64099 td 0xc4d4a280
sched_switch(c4d4a280,0,104,191,cf19d5e7,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4d4a280,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4eb02a8,5c,c0ec287c,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4eb02a8,c4eb0330,15c,c0ec287c,0,...) at _sleep+0x354
kern_wait(c4d4a280,bb79,e9683c74,0,0,...) at kern_wait+0xb76
wait4(c4d4a280,e9683cf8,c,c4d4a280,c4eb02a8,...) at wait4+0x3b
syscall(e9683d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810d69b, esp = 0xbfbfed2c, ebp = 0xbfbfed48 ---

Tracing command php pid 4384 tid 64070 td 0xc48a7a00
sched_switch(c48a7a00,0,104,191,fca58e6b,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c48a7a00,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4d3c1da,58,c0e73796,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4d3c1da,c1491ef0,158,c0e73796,0,...) at _sleep+0x354
kern_accept(c48a7a00,0,c3fd2c70,c3fd2c6c,c3fd2c68,...) at kern_accept+0x152
accept(c48a7a00,c3fd2cf8,c0eff73e,c0ee58fd,c49f6000,...) at accept+0x8f
syscall(c3fd2d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (30, FreeBSD ELF32, accept), eip = 0x285cfb5f, esp = 0xbfbeca9c, ebp = 0xbfbecab8 ---

Tracing command php pid 4672 tid 64097 td 0xc4d4a780
sched_switch(c4d4a780,0,104,191,c54d802b,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4d4a780,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4d3c1da,58,c0e73796,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4d3c1da,c1491ef0,158,c0e73796,0,...) at _sleep+0x354
kern_accept(c4d4a780,0,e967dc70,e967dc6c,e967dc68,...) at kern_accept+0x152
accept(c4d4a780,e967dcf8,c0eff73e,c0ee58fd,c4eb07f8,...) at accept+0x8f
syscall(e967dd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (30, FreeBSD ELF32, accept), eip = 0x285cfb5f, esp = 0xbfbeca9c, ebp = 0xbfbecab8 ---

Tracing command rrdtool pid 18573 tid 64113 td 0xc4ecf000
sched_switch(c4ecf000,0,104,191,f96c2a42,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c4ecf000,0,c0ec0078,1a0,4c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c49267a8,4c,c0ec2458,100,0,...) at sleepq_wait_sig+0x17
_sleep(c49267a8,c4926918,14c,c0ec2458,0,...) at _sleep+0x354
pipe_read(c4f57070,e96bfc58,c4b1e900,0,c4ecf000,...) at pipe_read+0x417
dofileread(e96bfc58,ffffffff,ffffffff,0,c4f57070,...) at dofileread+0x96
kern_readv(c4ecf000,0,e96bfc58,e96bfc78,1,...) at kern_readv+0x58
read(c4ecf000,e96bfcf8,c0eff73e,c0e9e08f,c4ef4550,...) at read+0x4f
syscall(e96bfd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x282aa593, esp = 0xbfbfec4c, ebp = 0xbfbfec68 ---

Tracing command sh pid 23066 tid 64079 td 0xc4a08a00
sched_switch(c4a08a00,0,104,191,a99fa456,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4a08a00,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4d40550,5c,c0ec287c,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4d40550,c4d405d8,15c,c0ec287c,0,...) at _sleep+0x354
kern_wait(c4a08a00,ffffffff,e9647c74,2,0,...) at kern_wait+0xb76
wait4(c4a08a00,e9647cf8,c0eff73e,c0ec285b,c4d40550,...) at wait4+0x3b
syscall(e9647d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816469b, esp = 0xbfbfeabc, ebp = 0xbfbfead8 ---

Tracing command apinger pid 21333 tid 64090 td 0xc4d4ba00
sched_switch(c4d4ba00,0,104,191,6f3534f,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4d4ba00,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(e9668aa8,c09f5aca,c48f0910,0,c4d4ba00,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c48f0924,0,e9668ad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c48f0924,c48f0910,3e0,621,e9668b74,...) at _cv_timedwait_sig+0x250
seltdwait(e9668c5c,e9668c64,52f,c4d4ba00,e9668b5c,...) at seltdwait+0x8a
poll(c4d4ba00,e9668cf8,c0eff73e,c0ec29b9,c49f67f8,...) at poll+0x300
syscall(e9668d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281355cf, esp = 0xbfbfcc1c, ebp = 0xbfbfed38 ---

Tracing command dnsmasq pid 15124 tid 64111 td 0xc4ecf500
sched_switch(c4ecf500,0,104,191,15a85b08,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4ecf500,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c4f55510,0,c0eb9e59,c4ecf500,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4f55524,0,e96b9a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c4f55524,c4f55510,c0ec2289,621,c4ea1700,...) at _cv_wait_sig+0x240
seltdwait(c4ea1700,58,c4f12780,c4ecf500,3,...) at seltdwait+0xa2
kern_select(c4ecf500,4a,bfbfec4c,bfbfebcc,bfbfeb4c,0,20,bfbfeb00,bfbfebcc) at kern_select+0x504
select(c4ecf500,e96b9cf8,16,c4ecf500,c4ef4aa0,...) at select+0x66
syscall(e96b9d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281d8513, esp = 0xbfbfeafc, ebp = 0xbfbfed18 ---

Tracing command dhclient pid 10563 tid 64096 td 0xc4d4aa00
sched_switch(c4d4aa00,0,104,191,f986b126,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4d4aa00,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(e967aaa8,c09f5aca,c4d49d50,0,c4d4aa00,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c4d49d64,0,e967aad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c4d49d64,c4d49d50,5265c00,621,e967ab64,...) at _cv_timedwait_sig+0x250
seltdwait(e967ac5c,e967ac64,52f,c4d4aa00,e967ab5c,...) at seltdwait+0x8a
poll(c4d4aa00,e967acf8,c0eff73e,c0e45c96,c4eb0aa0,...) at poll+0x300
syscall(e967ad38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed3c, ebp = 0xbfbfed68 ---

Tracing command dhclient pid 6066 tid 64116 td 0xc4d4c780
sched_switch(c4d4c780,0,104,191,cb9df8a3,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4d4c780,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c4d49610,0,c0eb9e59,c4d4c780,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4d49624,0,e96c8ad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c4d49624,c4d49610,c0ec2289,621,e96c8b5c,...) at _cv_wait_sig+0x240
seltdwait(c4e8c32c,c0ec2289,52f,c4d4c780,e96c8b5c,...) at seltdwait+0xa2
poll(c4d4c780,e96c8cf8,c1329940,e96c8c9c,c4ef4000,...) at poll+0x300
syscall(e96c8d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed3c, ebp = 0xbfbfed68 ---

Tracing command sshlockout_pf pid 57505 tid 64114 td 0xc4d4cc80
sched_switch(c4d4cc80,0,104,191,8173d88f,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4d4cc80,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(1b7741,c0a3fe50,c4d4cc80,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,1b7741,...) at _sleep+0x31e
kern_nanosleep(c4d4cc80,e96c2c64,e96c2c6c,708,0,...) at kern_nanosleep+0xc1
nanosleep(c4d4cc80,e96c2cf8,c0eff73e,c0ec2d30,c4eb0000,...) at nanosleep+0x6f
syscall(e96c2d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2817bf17, esp = 0xbf9fef4c, ebp = 0xbf9fef68 ---

Tracing command sshlockout_pf pid 57505 tid 64100 td 0xc4d4a000
sched_switch(c4d4a000,0,104,191,ced64d0a,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c4d4a000,0,c0ec0078,1a0,4c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4ee6310,4c,c0ec2458,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4ee6310,c4ee6480,14c,c0ec2458,0,...) at _sleep+0x354
pipe_read(c4d38af0,e9686c58,c41a6400,0,c4d4a000,...) at pipe_read+0x417
dofileread(e9686c58,ffffffff,ffffffff,0,c4d38af0,...) at dofileread+0x96
kern_readv(c4d4a000,0,e9686c58,e9686c78,1,...) at kern_readv+0x58
read(c4d4a000,e9686cf8,c0eff73e,c0e45c96,c4eb0000,...) at read+0x4f
syscall(e9686d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x28199593, esp = 0xbfbfe8dc, ebp = 0xbfbfe8f8 ---

Tracing command minicron pid 29043 tid 64063 td 0xc4906c80
sched_switch(c4906c80,0,104,191,6b83ee98,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4906c80,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(5265c01,c0a3fe50,c4906c80,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,5265c01,...) at _sleep+0x31e
kern_nanosleep(c4906c80,c3fb1c64,c3fb1c6c,15180,0,...) at kern_nanosleep+0xc1
nanosleep(c4906c80,c3fb1cf8,c0eff73e,c0ec2756,c48d7aa0,...) at nanosleep+0x6f
syscall(c3fb1d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28165f17, esp = 0xbfbfed2c, ebp = 0xbfbfed58 ---

Tracing command minicron pid 28903 tid 64098 td 0xc4d4a500
sched_switch(c4d4a500,0,104,191,1cb79a79,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4d4a500,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(36ee81,c0a3fe50,c4d4a500,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,36ee81,...) at _sleep+0x31e
kern_nanosleep(c4d4a500,e9680c64,e9680c6c,e10,0,...) at kern_nanosleep+0xc1
nanosleep(c4d4a500,e9680cf8,c0eff73e,c0ec2756,c4eb0550,...) at nanosleep+0x6f
syscall(e9680d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28165f17, esp = 0xbfbfed3c, ebp = 0xbfbfed68 ---

Tracing command minicron pid 28519 tid 64092 td 0xc4d4b500
sched_switch(c4d4b500,0,104,191,d1b24f35,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4d4b500,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(3a981,c0a3fe50,c4d4b500,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,3a981,...) at _sleep+0x31e
kern_nanosleep(c4d4b500,e966ec64,e966ec6c,f0,0,...) at kern_nanosleep+0xc1
nanosleep(c4d4b500,e966ecf8,c0eff73e,c0ec2756,c4eb1550,...) at nanosleep+0x6f
syscall(e966ed38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28165f17, esp = 0xbfbfed3c, ebp = 0xbfbfed68 ---

Tracing command cron pid 25653 tid 64103 td 0xc4ed0a00
sched_switch(c4ed0a00,0,104,191,befe3e9,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4ed0a00,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(ea61,c0a3fe50,c4ed0a00,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c1325b64,5c,c0ebcdef,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c1325b64,0,15c,c0ebcdef,ea61,...) at _sleep+0x31e
kern_nanosleep(c4ed0a00,e968fc64,e968fc6c,3c,0,...) at kern_nanosleep+0xc1
nanosleep(c4ed0a00,e968fcf8,c0eff73e,c0ec2d30,c4d407f8,...) at nanosleep+0x6f
syscall(e968fd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28182f17, esp = 0xbfbfec5c, ebp = 0xbfbfec88 ---

Tracing command ntpd pid 61941 tid 64091 td 0xc4d4b780
sched_switch(c4d4b780,0,104,191,8ddfcc89,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4d4b780,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(e966baa8,c09f5aca,c48f0ad0,0,c4d4b780,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c48f0ae4,0,e966bad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c48f0ae4,c48f0ad0,3a99,621,e966bb5c,...) at _cv_timedwait_sig+0x250
seltdwait(e966bc5c,e966bc64,52f,c4d4b780,e966bb5c,...) at seltdwait+0x8a
poll(c4d4b780,e966bcf8,c0eff73e,c0ec2df0,c4eb17f8,...) at poll+0x300
syscall(e966bd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281305cf, esp = 0xbfbfed0c, ebp = 0xbfbfedc8 ---

Tracing command ntpd pid 55579 tid 64101 td 0xc4ed1000
sched_switch(c4ed1000,0,104,191,519d1758,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4ed1000,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(e9689aa8,c09f5aca,c4d48850,0,c4ed1000,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c4d48864,0,e9689ad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c4d48864,c4d48850,abe1,621,e9689b5c,...) at _cv_timedwait_sig+0x250
seltdwait(e9689c5c,e9689c64,52f,c4ed1000,e9689b5c,...) at seltdwait+0x8a
poll(c4ed1000,e9689cf8,c0eff73e,c0ec2d30,c4d40d48,...) at poll+0x300
syscall(e9689d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281305cf, esp = 0xbfbfec4c, ebp = 0xbfbfed08 ---

Tracing command php pid 49912 tid 64086 td 0xc4d4c500
sched_switch(c4d4c500,0,104,191,20e61678,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4d4c500,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c49c8d1e,58,c0e73796,100,0,...) at sleepq_wait_sig+0x17
_sleep(c49c8d1e,c1491ef0,158,c0e73796,0,...) at _sleep+0x354
kern_accept(c4d4c500,0,e965cc70,e965cc6c,e965cc68,...) at kern_accept+0x152
accept(c4d4c500,e965ccf8,c0eff73e,c0ee58fd,c4d3f2a8,...) at accept+0x8f
syscall(e965cd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (30, FreeBSD ELF32, accept), eip = 0x285cfb5f, esp = 0xbfbeca9c, ebp = 0xbfbecab8 ---

Tracing command php pid 49702 tid 64083 td 0xc4a08000
sched_switch(c4a08000,0,104,191,844361c,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4a08000,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c49c8d1e,58,c0e73796,100,0,...) at sleepq_wait_sig+0x17
_sleep(c49c8d1e,c1491ef0,158,c0e73796,0,...) at _sleep+0x354
kern_accept(c4a08000,0,e9653c70,e9653c6c,e9653c68,...) at kern_accept+0x152
accept(c4a08000,e9653cf8,c0eff73e,c0ee58fd,c4d3faa0,...) at accept+0x8f
syscall(e9653d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (30, FreeBSD ELF32, accept), eip = 0x285cfb5f, esp = 0xbfbeca9c, ebp = 0xbfbecab8 ---

Tracing command php pid 39852 tid 64082 td 0xc4a08280
sched_switch(c4a08280,0,104,191,da99b56f,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4a08280,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4d3fd48,5c,c0ec287c,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4d3fd48,c4d3fdd0,15c,c0ec287c,0,...) at _sleep+0x354
kern_wait(c4a08280,ffffffff,e9650c74,0,0,...) at kern_wait+0xb76
wait4(c4a08280,e9650cf8,c,c4a08280,c4d3fd48,...) at wait4+0x3b
syscall(e9650d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x285aa69b, esp = 0xbfbfcb5c, ebp = 0xbfbfcb78 ---

Tracing command php pid 39356 tid 64068 td 0xc4906000
sched_switch(c4906000,0,104,191,725d6089,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c4906000,0,c0ec0078,1a0,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c49f6550,5c,c0ec287c,100,0,...) at sleepq_wait_sig+0x17
_sleep(c49f6550,c49f65d8,15c,c0ec287c,0,...) at _sleep+0x354
kern_wait(c4906000,ffffffff,c3fccc74,0,0,...) at kern_wait+0xb76
wait4(c4906000,c3fcccf8,c,c4906000,c49f6550,...) at wait4+0x3b
syscall(c3fccd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x285aa69b, esp = 0xbfbfcb5c, ebp = 0xbfbfcb78 ---

Tracing command lighttpd pid 38901 tid 64072 td 0xc4a09c80
sched_switch(c4a09c80,0,104,191,fca39b1d,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4a09c80,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(3e9,c0a3fe50,c4a09c80,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c4b1cb00,58,c0eb65e5,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c4b1cb00,c4b1cb00,158,c0eb65e5,3e9,...) at _sleep+0x31e
kern_kevent(c4a09c80,c,0,24b5,c3fd8c58,...) at kern_kevent+0x364
kevent(c4a09c80,c3fd8cf8,c0eff73e,c0ec2d30,c49f5aa0,...) at kevent+0x19b
syscall(c3fd8d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x2836e07b, esp = 0xbfbfeadc, ebp = 0xbfbfeb18 ---

Tracing command syslogd pid 16589 tid 64076 td 0xc4a09280
sched_switch(c4a09280,0,104,191,fcace83d,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4a09280,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48ef790,0,c0eb9e59,c4a09280,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48ef7a4,0,c3fe4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48ef7a4,c48ef790,c0ec2289,621,c491c0a8,...) at _cv_wait_sig+0x240
seltdwait(c491c0a8,58,c41a6400,c4a09280,0,...) at seltdwait+0xa2
kern_select(c4a09280,f,284320a8,0,0,0,20,bfbfe250,28104bd4) at kern_select+0x504
select(c4a09280,c3fe4cf8,c0eff73e,c0ec2756,c49f5000,...) at select+0x66
syscall(c3fe4d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x282ea513, esp = 0xbfbfe24c, ebp = 0xbfbfe278 ---

Tracing command dhclient pid 11964 tid 64089 td 0xc4d4bc80
sched_switch(c4d4bc80,0,104,191,f9873a29,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4d4bc80,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(e9665aa8,c09f5aca,c4d48690,0,c4d4bc80,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c4d486a4,0,e9665ad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c4d486a4,c4d48690,5265c00,621,e9665b64,...) at _cv_timedwait_sig+0x250
seltdwait(e9665c5c,e9665c64,52f,c4d4bc80,e9665b5c,...) at seltdwait+0x8a
poll(c4d4bc80,e9665cf8,c0eff73e,c0e45c96,c49f6aa0,...) at poll+0x300
syscall(e9665d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command inetd pid 54321 tid 64084 td 0xc4908c80
sched_switch(c4908c80,0,104,191,dce4048a,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4908c80,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48efd90,0,c0eb9e59,c4908c80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48efda4,0,e9656a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48efda4,c48efd90,c0ec2289,621,c491c460,...) at _cv_wait_sig+0x240
seltdwait(c491c460,58,c41a6400,c4908c80,c14652f8,...) at seltdwait+0xa2
kern_select(c4908c80,d,bfbfdf80,0,0,0,20,bfbfdf40,bfbfe0a4) at kern_select+0x504
select(c4908c80,e9656cf8,c0eff73e,c0ec2756,c4d3f7f8,...) at select+0x66
syscall(e9656d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281aa513, esp = 0xbfbfdf3c, ebp = 0xbfbfedc8 ---

Tracing command logger pid 41557 tid 64088 td 0xc4d4c000
sched_switch(c4d4c000,0,104,191,9315e11c,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c4d4c000,0,c0ec0078,1a0,4c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c4925188,4c,c0ec2458,100,0,...) at sleepq_wait_sig+0x17
_sleep(c4925188,c49252f8,14c,c0ec2458,0,...) at _sleep+0x354
pipe_read(c491cb60,e9662c58,c41a6400,0,c4d4c000,...) at pipe_read+0x417
dofileread(e9662c58,ffffffff,ffffffff,0,c491cb60,...) at dofileread+0x96
kern_readv(c4d4c000,0,e9662c58,e9662c78,1,...) at kern_readv+0x58
read(c4d4c000,e9662cf8,c4a09280,e9662cd0,c49f6d48,...) at read+0x4f
syscall(e9662d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x28184593, esp = 0xbfbfe90c, ebp = 0xbfbfe928 ---

Tracing command tcpdump pid 41538 tid 64087 td 0xc4d4c280
sched_switch(c4d4c280,0,104,191,febcdd5,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,1a,...) at mi_switch+0x200
sleepq_switch(c4d4c280,0,c0ec0078,1a0,1a,...) at sleepq_switch+0x15f
sleepq_catch_signals(3e8,c0a3fe50,c4d4c280,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c4d42000,1a,c0ed4e90,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c4d42000,c4d42084,11a,c0ed4e90,3e8,...) at _sleep+0x31e
bpfread(c41f5000,e965fc58,0,0,8000,...) at bpfread+0x1dd
devfs_read_f(c491c7a8,e965fc58,c4b1ce00,0,c4d4c280,...) at devfs_read_f+0x7e
dofileread(e965fc58,ffffffff,ffffffff,0,c491c7a8,...) at dofileread+0x96
kern_readv(c4d4c280,9,e965fc58,e965fc78,1,...) at kern_readv+0x58
read(c4d4c280,e965fcf8,c0eff73e,c0e45c96,c4d3f000,...) at read+0x4f
syscall(e965fd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2839b593, esp = 0xbfbfeaec, ebp = 0xbfbfebe8 ---

Tracing command dhclient pid 34414 tid 64062 td 0xc4908000
sched_switch(c4908000,0,104,191,f8d952fc,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4908000,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48ef410,0,c0eb9e59,c4908000,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48ef424,0,c3fadad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48ef424,c48ef410,c0ec2289,621,c3fadb5c,...) at _cv_wait_sig+0x240
seltdwait(c4c7582c,c0ec2289,52f,c4908000,c3fadb5c,...) at seltdwait+0xa2
poll(c4908000,c3fadcf8,c1329940,c3fadc9c,c48d7d48,...) at poll+0x300
syscall(c3fadd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command dhclient pid 22390 tid 64077 td 0xc4a09000
sched_switch(c4a09000,0,104,191,f987c3c5,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4a09000,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c3fe7aa8,c09f5aca,c48efbd0,0,c4a09000,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c48efbe4,0,c3fe7ad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c48efbe4,c48efbd0,5265c00,621,c3fe7b64,...) at _cv_timedwait_sig+0x250
seltdwait(c3fe7c5c,c3fe7c64,52f,c4a09000,c3fe7b5c,...) at seltdwait+0x8a
poll(c4a09000,c3fe7cf8,c0eff73e,c0e45c96,c48d9d48,...) at poll+0x300
syscall(c3fe7d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command dhclient pid 17018 tid 64078 td 0xc4a08c80
sched_switch(c4a08c80,0,104,191,c4e341c,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4a08c80,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48a9810,0,c0eb9e59,c4a08c80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48a9824,0,c3feaad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48a9824,c48a9810,c0ec2289,621,c3feab5c,...) at _cv_wait_sig+0x240
seltdwait(c4ab892c,c0ec2289,52f,c4a08c80,c3feab5c,...) at seltdwait+0xa2
poll(c4a08c80,c3feacf8,c1329940,c3feac9c,c48d9aa0,...) at poll+0x300
syscall(c3fead38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command dhclient pid 14474 tid 64061 td 0xc4908280
sched_switch(c4908280,0,104,191,f9884d4d,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4908280,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c3fa9aa8,c09f5aca,c48ef950,0,c4908280,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c48ef964,0,c3fa9ad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c48ef964,c48ef950,2559c49,621,c3fa9b64,...) at _cv_timedwait_sig+0x250
seltdwait(c3fa9c5c,c3fa9c64,52f,c4908280,c3fa9b5c,...) at seltdwait+0x8a
poll(c4908280,c3fa9cf8,c0eff73e,c0e45c96,c48d9000,...) at poll+0x300
syscall(c3fa9d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command dhclient pid 8809 tid 64073 td 0xc4a09a00
sched_switch(c4a09a00,0,104,191,77d4175,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4a09a00,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48efb90,0,c0eb9e59,c4a09a00,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48efba4,0,c3fdbad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48efba4,c48efb90,c0ec2289,621,c3fdbb5c,...) at _cv_wait_sig+0x240
seltdwait(c4ab932c,c0ec2289,52f,c4a09a00,c3fdbb5c,...) at seltdwait+0xa2
poll(c4a09a00,c3fdbcf8,c1329940,c3fdbc9c,c49f57f8,...) at poll+0x300
syscall(c3fdbd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x281395cf, esp = 0xbfbfed7c, ebp = 0xbfbfeda8 ---

Tracing command sshd pid 8657 tid 64071 td 0xc48a7780
sched_switch(c48a7780,0,104,191,97c8bdc9,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c48a7780,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48a8850,0,c0eb9e59,c48a7780,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48a8864,0,c3fd5a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48a8864,c48a8850,c0ec2289,621,c491b0e0,...) at _cv_wait_sig+0x240
seltdwait(c491b0e0,58,c48f8580,c48a7780,c3fd5ae8,...) at seltdwait+0xa2
kern_select(c48a7780,6,284090b8,0,0,0,20,bfbfdf10,a) at kern_select+0x504
select(c48a7780,c3fd5cf8,c,c48a7780,c49f5d48,...) at select+0x66
syscall(c3fd5d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x2839e513, esp = 0xbfbfdf0c, ebp = 0xbfbfee18 ---

Tracing command devd pid 268 tid 64064 td 0xc4906a00
sched_switch(c4906a00,0,104,191,f97e6a7c,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4906a00,0,c0ec0078,1a0,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c09f5aca,c48ef450,0,c0eb9e59,c4906a00,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48ef464,0,c3fb5a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c48ef464,c48ef450,c0ec2289,621,c491b508,...) at _cv_wait_sig+0x240
seltdwait(c491b508,58,c41a6400,c4906a00,c1465a88,...) at seltdwait+0xa2
kern_select(c4906a00,c,bfbfe980,0,0,0,20,bfbfe950,400) at kern_select+0x504
select(c4906a00,c3fb5cf8,c0eff73e,c0ee58fd,c48d77f8,...) at select+0x66
syscall(c3fb5d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x8088343, esp = 0xbfbfe94c, ebp = 0xbfbfee28 ---

Tracing command check_reload_status pid 257 tid 64065 td 0xc4906780
sched_switch(c4906780,0,104,191,df2cf35e,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4906780,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0ec0078,160,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c48f7d80,58,c0eb65e5,100,0,...) at sleepq_wait_sig+0x17
_sleep(c48f7d80,c48f7d80,158,c0eb65e5,0,...) at _sleep+0x354
kern_kevent(c4906780,3,0,1,c3fb9c58,...) at kern_kevent+0x364
kevent(c4906780,c3fb9cf8,c1329940,c3fb9c9c,c48d7550,...) at kevent+0x19b
syscall(c3fb9d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x2818007b, esp = 0xbfbfec7c, ebp = 0xbfbfed98 ---

Tracing command check_reload_status pid 255 tid 64066 td 0xc4906500
sched_switch(c4906500,0,104,191,fcae8e7a,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,58,...) at mi_switch+0x200
sleepq_switch(c4906500,0,c0ec0078,1a0,58,...) at sleepq_switch+0x15f
sleepq_catch_signals(136d,c0a3fe50,c4906500,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c48f7480,58,c0eb65e5,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c48f7480,c48f7480,158,c0eb65e5,136d,...) at _sleep+0x31e
kern_kevent(c4906500,4,0,40,c3fbdc58,...) at kern_kevent+0x364
kevent(c4906500,c3fbdcf8,c0eff73e,c0ec2d30,c48d72a8,...) at kevent+0x19b
syscall(c3fbdd38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x2818007b, esp = 0xbfbfeb8c, ebp = 0xbfbfebd8 ---

Tracing command md0 pid 41 tid 64060 td 0xc4908500
sched_switch(c4908500,0,104,191,fb0292f7,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c4908500,0,c0ec0078,260,0,...) at sleepq_switch+0x15f
sleepq_wait(c4934800,4c,c0e86f33,0,0,...) at sleepq_wait+0x63
_sleep(c4934800,c4934820,24c,c0e86f33,0,...) at _sleep+0x36b
md_kthread(c4934800,c3fa5d38,c0eb69c8,344,c48d92a8,...) at md_kthread+0x125
fork_exit(c07c9d20,c4934800,c3fa5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3fa5d70, ebp = 0 ---

Tracing command flowcleaner pid 25 tid 64058 td 0xc4325780
sched_switch(c4325780,0,104,191,d266c4fd,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4325780,0,c0ec0078,283,c4325780,...) at sleepq_switch+0x15f
sleepq_timedwait(c1492568,0,c3f9dcc4,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c1492568,c1492570,4e20,627,0,...) at _cv_timedwait+0x250
flowtable_cleaner(0,c3f9dd38,c0eb69c8,344,c48d97f8,...) at flowtable_cleaner+0x1bc
fork_exit(c0ab3230,0,c3f9dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f9dd70, ebp = 0 ---

Tracing command softdepflush pid 24 tid 64057 td 0xc4325a00
sched_switch(c4325a00,0,104,191,3407b528,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,44,...) at mi_switch+0x200
sleepq_switch(c4325a00,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c14ad2e0,44,c0eea950,0,0,...) at sleepq_timedwait+0x6b
_sleep(c14ad2e0,c14ad284,44,c0eea950,3e8,...) at _sleep+0x339
softdep_flush(0,c3f9ad38,c0eb69c8,344,c41ea7f8,...) at softdep_flush+0x244
fork_exit(c0ca8a30,0,c3f9ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f9ad70, ebp = 0 ---

Tracing command syncer pid 23 tid 64056 td 0xc4325c80
sched_switch(c4325c80,0,104,191,2348f57c,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c4325c80,0,c0ec0078,283,c4325c80,...) at sleepq_switch+0x15f
sleepq_timedwait(c1492354,0,c3f97c88,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c1492354,c1492340,3e8,6d3,4e20,...) at _cv_timedwait+0x250
sched_sync(0,c3f97d38,c0eb69c8,344,c41eaaa0,...) at sched_sync+0x502
fork_exit(c0a9ae00,0,c3f97d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f97d70, ebp = 0 ---

Tracing command vnlru pid 22 tid 64055 td 0xc48a6000
sched_switch(c48a6000,0,104,191,34077fd5,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,50,...) at mi_switch+0x200
sleepq_switch(c48a6000,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c41ead48,50,c0ec9ff5,0,0,...) at sleepq_timedwait+0x6b
_sleep(c41ead48,c1492314,250,c0ec9ff5,3e8,...) at _sleep+0x339
vnlru_proc(0,c3f94d38,c0eb69c8,344,c41ead48,...) at vnlru_proc+0xe7
fork_exit(c0a9b9b0,0,c3f94d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f94d70, ebp = 0 ---

Tracing command bufdaemon pid 21 tid 64054 td 0xc48a6280
sched_switch(c48a6280,0,104,191,3416af90,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,44,...) at mi_switch+0x200
sleepq_switch(c48a6280,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c1492088,44,c0ec774c,0,0,...) at sleepq_timedwait+0x6b
_sleep(c1492088,c149208c,44,c0ec774c,3e8,...) at _sleep+0x339
buf_daemon(0,c3f91d38,c0eb69c8,344,c48ca000,...) at buf_daemon+0x138
fork_exit(c0a83300,0,c3f91d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f91d70, ebp = 0 ---

Tracing command idlepoll pid 20 tid 64053 td 0xc48a6500
sched_switch(c48a6500,0,104,191,ba984e1f,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c48a6500,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c13250bc,0,c0eba7c1,0,0,...) at sleepq_timedwait+0x6b
_sleep(c13250bc,0,0,c0eba7c1,bb8,...) at _sleep+0x339
poll_idle(0,c3f8ed38,c0eb69c8,344,c48ca2a8,...) at poll_idle+0x1c1
fork_exit(c09f9030,0,c3f8ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f8ed70, ebp = 0 ---

Tracing command pagezero pid 19 tid 64052 td 0xc48a6780
sched_switch(c48a6780,0,104,191,19cb087b,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c48a6780,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c14adfb4,0,c0ef0551,0,0,...) at sleepq_timedwait+0x6b
_sleep(c14adfb4,c14adb70,0,c0ef0551,493e0,...) at _sleep+0x339
vm_pagezero(0,c3f8bd38,c0eb69c8,344,c48ca550,...) at vm_pagezero+0xdc
fork_exit(c0ce6760,0,c3f8bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f8bd70, ebp = 0 ---

Tracing command vmdaemon pid 18 tid 64051 td 0xc48a6a00
sched_switch(c48a6a00,0,104,191,e81bf68,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,68,...) at mi_switch+0x200
sleepq_switch(c48a6a00,0,c0ec0078,260,0,...) at sleepq_switch+0x15f
sleepq_wait(c14adbdc,68,c0ec774c,0,0,...) at sleepq_wait+0x63
_sleep(c14adbdc,c14adbe0,68,c0ec774c,0,...) at _sleep+0x36b
vm_daemon(0,c3f88d38,c0eb69c8,344,c48ca7f8,...) at vm_daemon+0x59
fork_exit(c0ce0bc0,0,c3f88d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f88d70, ebp = 0 ---

Tracing command pagedaemon pid 17 tid 64050 td 0xc48a6c80
sched_switch(c48a6c80,0,104,191,40fbc5e2,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,44,...) at mi_switch+0x200
sleepq_switch(c48a6c80,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c14adba4,44,c0ec774c,0,0,...) at sleepq_timedwait+0x6b
_sleep(c14adba4,c14adb70,44,c0ec774c,1388,...) at _sleep+0x339
vm_pageout(0,c3f85d38,c0eb69c8,344,c48caaa0,...) at vm_pageout+0x2bb
fork_exit(c0ce1a40,0,c3f85d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f85d70, ebp = 0 ---

Tracing command xpt_thrd pid 16 tid 64049 td 0xc48a7000
sched_switch(c48a7000,0,104,191,c4d994a,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c48a7000,0,c0ec0078,260,0,...) at sleepq_switch+0x15f
sleepq_wait(c12ef9d4,4c,c0e40d78,0,0,...) at sleepq_wait+0x63
_sleep(c12ef9d4,c12efa18,4c,c0e40d78,0,...) at _sleep+0x36b
xpt_scanner_thread(0,c3f82d38,c0eb69c8,344,c48cad48,...) at xpt_scanner_thread+0x47
fork_exit(c0494d70,0,c3f82d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f82d70, ebp = 0 ---

Tracing command pfpurge pid 9 tid 64048 td 0xc48a7280
sched_switch(c48a7280,0,104,191,656d73f,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,5c,...) at mi_switch+0x200
sleepq_switch(c48a7280,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c04f85b0,5c,c0e4fe72,0,0,...) at sleepq_timedwait+0x6b
_sleep(c04f85b0,0,5c,c0e4fe72,3e8,...) at _sleep+0x339
pf_purge_thread(0,c3f7fd38,c0eb69c8,344,c41aa2a8,...) at pf_purge_thread+0x3d
fork_exit(c04f85b0,0,c3f7fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f7fd70, ebp = 0 ---

Tracing command sctp_iterator pid 8 tid 64047 td 0xc48a7500
sched_switch(c48a7500,0,104,191,c468d0f7,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,0,...) at mi_switch+0x200
sleepq_switch(c48a7500,0,c0ec0078,260,0,...) at sleepq_switch+0x15f
sleepq_wait(c14997d8,0,c0ed997c,0,0,...) at sleepq_wait+0x63
_sleep(c14997d8,c14997a0,0,c0ed997c,0,...) at _sleep+0x36b
sctp_iterator_thread(0,c3f7cd38,c0eb69c8,344,c41aa550,...) at sctp_iterator_thread+0x5c
fork_exit(c0b71d70,0,c3f7cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f7cd70, ebp = 0 ---

Tracing command fdc0 pid 7 tid 64042 td 0xc4324500
sched_switch(c4324500,0,104,191,34075078,...) at sched_switch+0x36a
mi_switch(104,0,c0ec0078,1eb,4c,...) at mi_switch+0x200
sleepq_switch(c4324500,0,c0ec0078,283,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c434a23c,4c,c0eadbc8,0,0,...) at sleepq_timedwait+0x6b
_sleep(c434a23c,c434a2f0,4c,c0eadbc8,3e8,...) at _sleep+0x339
fdc_thread(c434a200,c3f6cd38,c0eb69c8,344,c41aa7f8,...) at fdc_thread+0x2be
fork_exit(c0d67ee0,c434a200,c3f6cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3f6cd70, ebp = 0 ---

Tracing command usb pid 15 tid 64036 td 0xc4325500
sched_switch(c4325500,0,104,191,c639f20,...) at sched_switch+0x36a
mi_swiconfig.txt0600001253411701347467  7552 ustarrootwheeloptions   CONFIG_AUTOGENERATED
ident   pfSense8
machine   i386
cpu   I686_CPU
cpu   I586_CPU
cpu   I486_CPU
makeoptions   DEBUG=-g
options   ALTQ_NOPCC
options   PPS_SYNC
options   ALTQ_FAIRQ
options   MROUTING
options   DEVICE_POLLING
options   NULLFS
options   MSGTQL=2048
options   MSGSSZ=32
options   MSGSEG=512
options   MSGMNI=40
options   MSGMNB=8192
options   ALTQ_PRIQ
options   ALTQ_HFSC
options   ALTQ_RIO
options   ALTQ_RED
options   ALTQ_CBQ
options   ALTQ
options   TCP_SIGNATURE
options   IPSEC_NAT_T
options   IPSEC
options   NETGRAPH_PRED1
options   NETGRAPH_DEFLATE
options   NETGRAPH_CAR
options   NETGRAPH_PIPE
options   NETGRAPH_TCPMSS
options   NETGRAPH_VJC
options   NETGRAPH_UI
options   NETGRAPH_TTY
options   NETGRAPH_TEE
options   NETGRAPH_SOCKET
options   NETGRAPH_RFC1490
options   NETGRAPH_PPTPGRE
options   NETGRAPH_PPPOE
options   NETGRAPH_PPP
options   NETGRAPH_ONE2MANY
options   NETGRAPH_MPPC_ENCRYPTION
options   NETGRAPH_LMI
options   NETGRAPH_KSOCKET
options   NETGRAPH_EIFACE
options   NETGRAPH_IFACE
options   NETGRAPH_HOLE
options   NETGRAPH_FRAME_RELAY
options   NETGRAPH_ETHER
options   NETGRAPH_ECHO
options   NETGRAPH_CISCO
options   NETGRAPH_BRIDGE
options   NETGRAPH_BPF
options   NETGRAPH_ASYNC
options   NETGRAPH_VLAN
options   NETGRAPH_L2TP
options   NETGRAPH
options   IPSTEALTH
options   IPFIREWALL_VERBOSE_LIMIT
options   IPFIREWALL_VERBOSE
options   IPFIREWALL_FORWARD
options   IPFIREWALL_DEFAULT_TO_ACCEPT
options   UNIONFS
options   GEOM_BDE
options   GEOM_ELI
options   GEOM_UZIP
options   GEOM_MIRROR
options   ALT_BREAK_TO_DEBUGGER
options   BREAK_TO_DEBUGGER
options   USB_DEBUG
options   AH_SUPPORT_AR5416
options   IEEE80211_SUPPORT_MESH
options   IEEE80211_AMPDU_AGE
options   IEEE80211_DEBUG
options   AHD_REG_PRETTY_PRINT
options   AHC_REG_PRETTY_PRINT
options   ATA_STATIC_ID
options   SMP
options   INCLUDE_CONFIG_FILE
options   FLOWTABLE
options   MAC
options   AUDIT
options   HWPMC_HOOKS
options   KBD_INSTALL_CDEV
options   PRINTF_BUFR_SIZE=128
options   _KPOSIX_PRIORITY_SCHEDULING
options   P1003_1B_SEMAPHORES
options   SYSVSEM
options   SYSVMSG
options   SYSVSHM
options   STACK
options   KTRACE
options   SCSI_DELAY=5000
options   COMPAT_FREEBSD7
options   COMPAT_FREEBSD6
options   COMPAT_FREEBSD5
options   COMPAT_FREEBSD4
options   COMPAT_43TTY
options   GEOM_LABEL
options   GEOM_PART_GPT
options   PSEUDOFS
options   PROCFS
options   CD9660
options   MSDOSFS
options   NFS_ROOT
options   NFSLOCKD
options   NFSSERVER
options   NFSCLIENT
options   MD_ROOT
options   UFS_GJOURNAL
options   UFS_DIRHASH
options   UFS_ACL
options   SOFTUPDATES
options   FFS
options   SCTP
options   INET6
options   INET
options   PREEMPTION
options   SCHED_ULE
options   WITNESS_SKIPSPIN
options   WITNESS
options   INVARIANT_SUPPORT
options   INVARIANTS
options   GDB
options   DDB
options   KDB_TRACE
options   KDB
options   NATIVE
options   GEOM_PART_MBR
options   GEOM_PART_EBR_COMPAT
options   GEOM_PART_EBR
options   GEOM_PART_BSD
options   ISAPNP
device   isa
device   npx
device   mem
device   io
device   uart_ns8250
device   atpic
device   apic
device   cpufreq
device   acpi
device   eisa
device   pci
device   fdc
device   ata
device   atadisk
device   ataraid
device   atapicd
device   atapifd
device   atapist
device   ahb
device   ahc
device   ahd

[marcelloc]

Did you tried this way?

• configure a sync interface
• enable sync between firewalls
• check replicaton between boxes
• create a carp interface with a id that cound not conflict with any vrrp on your network

[cmb]

What type of NICs do you have on the box? Any unusual interface setup? Haven't heard of a CARP panic in a long time much less one so easily triggered.

[mdpugh]

{
•configure a sync interface
•enable sync between firewalls
•check replicaton between boxes
•create a carp interface with a id that cound not conflict with any vrrp on your network
}

Yes, I can vouch for all of this and that it was done in this order as the online tutorial suggests in contrast to the book.  I assume that by "check replication between boxes" you are referring to pfsync on the dedicated interfaces.  I really thought you had nailed it with the last point because I had an older OpenBSD box configured for CARP acting as a temporary gateway while I set up pfSense, but alas, I shut it down and still got the same result when I clicked Apply Changes on the Firewall: Virtual IP Address: Edit page.  There are no other redundancy protocols running anywhere on the network.

The NICs are all genuine Intel Pro/1000 MTs.  The only thing unusual about the (eventual) setup is that I have two disjoint LANs.  I want firewall A to be the master for LAN1 and slave for LAN2 and firewall B to be master for LAN2 and slave for LAN1.  I don't think this is the issue for two reasons: (a) I set this up in OpenBSD/FreeBSD before and it worked, and (b) I haven't gotten that far in the configuration process with pfSense.  I have configured the sync interfaces on both hosts and checked for state synchronization (which is working).  I have not set up configuration synchronization since both firewalls are on equal footing as far as the master/slave relationship is concerned (and if this is the equivalent of the (no-sync) option in pf, then this is how I set it up before anyway).  I have done no CARP configuration on firewall B yet, nor have I configured LAN2 on firewall A.  All I have done past the sync stage is attempt to configure firewall A as the master on LAN1, and that's when things go haywire.

[mdpugh]

I did a clean install of 2.0.1 hoping in vain this would fix this issue.  I left the default installation alone except I changed the LAN IP to 10.0.0.101/24.  I then attempted to set a CARP VIP to 10.0.0.100/24 with all other settings to default and off we go to the races.  This is about as basic of an installation as possible, and I'm still getting the same panic.  I've also tried this with the pfsync interface enabled with and without the slave being active.  It does not seem to matter what I do; the kernel panics.  I have a hard time believing it's the NICs; I have the same model in the FreeBSD machines that have worked in this configuration.  I stress again that there are no other machines transmitting redundancy protocols on this network.  This happens so readily that I am flabbergasted that I'm evidently the only person experiencing this problem. 

[marcelloc]

Do your hardware supports pfsense 64bits

Can you try to install it and see if you get same issue?

[mdpugh]

No, these are 32-bit boxes retasked for this purpose.

[mdpugh]

I tried something different hoping to gain some new insight.  I booted the firewall with all interfaces unplugged save the private LAN I use for configuration/maintenance.  I then configured CARP which did not immediately throw any errors.  Then I plugged each interface in individually and, if no panic resulted, rebooted the machine checking for panic on boot before proceeding to the next interface.  This worked until I got to the interface on which CARP was configured (no surprise there); as soon as I plugged it in--panic.  The switch connected to this interface is also connected to its sister interface on the other firewall and the rest of the LAN.  I tried (a) unplugging the other host, (b) unplugging the LAN, and (c) unplugging both.  The machine panics even when it is the only device connected to the switch.  I had hoped this would be an eye-opening experiment, but it seems all I have managed to determine is that when the troublesome interface is activated it spells trouble.  Lest anyone suspect it's the NIC, I have tried this with three different cards with the same result.  I can handle troubleshooting, but I have run out of ideas.  Maybe this experiment will light a bulb in someone else.

[cmb]

can you email me a backup of your config?  cmb at pfsense dot org

[cmb]

Your config works perfectly for me. So it is somehow hardware-specific, though I have no idea how. The couple boxes I tested with both have Intel Pro/1000 NICs which should behave exactly the same as yours.

[mdpugh]

This happens on the other host also, which has different onboard hardware (MB, chipset, etc.) and slightly different NICs (still Pro/1000, but GT, XT, and T).  The only similarity, really, is that they're both Pentium IIIs.  What is the likelihood I'd have the same issue on two heterogeneous machines?

[mdpugh]

So, I'm eating lunch at Subway pondering this latest news.  I'm pretty sure it's not a hardware issue because both machines are affected.  Still, the fact that you got it to work with no modification is quite the puzzle.  So, I ask myself, how could my installation be different from Chris's?  I mean, there are not that many options to choose from during installation.  Then it dawns on me.  I have been choosing the Developer's Kernel from the onset because these Pentium III systems aren't SMP, the uniprocessor option is gone in 2.0, I don't want to go headless, and I wasn't sure whether the DK was necessary for 2.1 (which I plan to install next).  I just reinstalled using the SMP option and now CARP is working perfectly.  I'm not sure this is what is supposed to happen, but the problem (on my end, at least) is solved.  If you hadn't asked for the config file and tested it, I have no idea how long it may have taken to figure this out.  Thank you!

[jimp]

All kernels (even Dev) are SMP on 2.0.

There is no longer any benefit to loading a uniprocessor kernel (Mentioned a little here but also in more detail by me around the forum).

I've had some issues with the dev kernel in certain setups as well but it does a lot more strict locking checking and reporting, which is what you appear to have hit here.

We have enough debug info in the stock kernel these days that the full dev kernel isn't quite as necessary on its own, but still useful in rare cases.