I'm at a loss as to where to look to solve this problem. It appears to be a bug in the underlying FreeBSD APIC code. After several hours of hair pulling frustration and troubleshooting I was able to track down that disabling the APIC made the system stable, but at the cost of about 60% of the CPU performance (it's a dual core system, no apic == only 1 core.)
+ MSI Wind Nettop 100 (Intel Atom 330 dual core CPU)
+ 1GB DDR2 533 RAM stick pulled from a laptop when the laptop was upgraded
+ 320GB Western Digital SATA hard drive
+ Lite-ON SATA DVDROM drive
+ Cheap Logitech USB Keyboard
The lockups and crashes happen almost randomly under normal circumstances, happening sometimes before I can even complete a normal installation. This happens on both the 1.2.3 stable and 2.0 BETA pfsense, as well as the 9.0-CURRENT-200912 livefs of FreeBSD (although hard lockups are more common on the latter.) Dropping to the bootloader prompt during bootup and entering 'set hint.apic.0.disabled="1"' makes the system stable on all three.
I can cause an almost immediate Fault Trap 12 using the following method:
1. Boot pfSense 1.2.3-stable
2. Configure VLANs, etc to get to console menu
3. Drop to a shell and enter the following command to put a load on the processor:
dd if=/dev/zero of=/dev/null4. After a couple of seconds the system crashes with Fault Trap 12 (or sometimes, but rarely, a panic of some other kind)
I have tried a different stick of RAM and the problems continue. (The RAM in it has worked flawlessly in a laptop for over a year.) Memtest86+ passes with no errors, and the system will successfully complete the linpack benchmark (run from the PelicanHPC LiveCD Linux Distro.) The system ran stable long enough to DBAN 2 1TB hard disks, so I tend to think this problem is related to FreeBSD in some way.
I should note that this system lacks a serial port so getting a backtrace isn't possible, unless a USB to serial converter can be used, in which case I'll go and buy one.
The motherboard is at the latest BIOS version released from MSI so a BIOS upgrade is not an option (I haven't tried a downgrade yet.)
If there's something I've missed let me know. I'm more of a Linux person; my only experience with FreeBSD is from using pfSense (which I've been using as my firewall for the past 3 years, may thanks for outstanding software!) As I'm used to troubleshooting Linux it's quite possible I've neglected to check something that should be checked in FreeBSD land.
Thanks in advance to anyone who has any ideas on what to do. I think I'll be able to make do with the system in a crippled state with the APIC disabled so I'm going to go ahead and install and run the 1.2.3 stable release like that. The highest load I can imagine the system seeing is transferring 20-30Mbps over VPN between a wireless client and a file server.
---- EDIT ----
I forgot to mention, I'm using VLANs and a Gig-E "smart" switch (Netgear GS108T) hence the reason only 1 Ethernet port.