The pfSense Store

Author Topic: pfSense 2.0.1 locking up (hard)  (Read 2218 times)

0 Members and 1 Guest are viewing this topic.

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
pfSense 2.0.1 locking up (hard)
« on: March 18, 2012, 03:43:30 pm »
I've just installed a new pfSense setup for my client using one of the SuperMicro Atom D525 motherboards.

It has 3 NIC's now. 2 on motherboard and one Intel NIC plugged in via PCIe.

It's been running flawlessly at home where I built a similar (but not completely same) model of his network.. but when I got it to the client... and we decided to add RIP to the system as it now sits between his netscreen firewall and his ExtremeNetworks core router, the unit would randomly lock up. It requires a hard reset. (local keyboard is completely unresponsive)

(or so it seems. I've turned off RIP for now since the network is pretty static)

And it's now the waiting game.

ALL this system is doing is Squid proxy. There's really no other rules for the FW/NAT except "manual NAT" and "allow all" for the firewall. The setup is pretty simple/no-frills.

Has anyone else seen this behavior at all?

(lock ups in general with things to avoid)?

 -Ben
--
Ben - O.D.T., S.P.

Online stephenw10

  • Hero Member
  • *****
  • Posts: 8111
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #1 on: March 19, 2012, 07:13:48 am »

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #2 on: March 19, 2012, 09:41:45 am »
I'll take a look.

The card that faces into the network is an Intel card (but comes up as em(x) which still applies in that FAQ)
(I think the 2 on the MB are Intel as well sooo)


Thanks!!

 -Ben
--
Ben - O.D.T., S.P.

Offline clarknova

  • Hero Member
  • *****
  • Posts: 920
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #3 on: March 19, 2012, 09:58:00 am »
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.
db

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #4 on: March 19, 2012, 11:15:58 am »
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Thanks for the note... it's good to hear from people with similar hardware.

I have the same MB at home (not for this install though, I got the D525 for the client) -- and THEN bought the D510 since I had parts for it laying around and don't need the mem speed for it.

Your proposition gives me the willies. What? Am I supposed to schedule a reboot every weekend? Turn on the BMC watchdog? (need to test that at home first) Yikes.

I do have 4GB in the unit -- but I am running pfSense-2.0.1-embedded_AM64 off an SSD... so RAM isn't exactly wild-free-and-open. But, I'll bump up the bufs later.

right not seems to have a fairly static 5574/25600 and the client is getting their full 50Mb/s BW to the internet. So overall, provided the system stays up, we're pretty happy.

As always, the open source community does some excellent stuff.

stay tuned,

 -Ben
--
Ben - O.D.T., S.P.

Offline clarknova

  • Hero Member
  • *****
  • Posts: 920
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #5 on: March 19, 2012, 11:27:39 am »
Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.
db

Offline clarknova

  • Hero Member
  • *****
  • Posts: 920
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #6 on: March 19, 2012, 11:35:21 am »
I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.
db

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #7 on: March 19, 2012, 11:36:05 am »
Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.

Hmmm...

This user has 50Mb/s and about 3,000 PC's (Mac/Windows Mix - but mostly probably 90% Mac).

I just need to watch it. So far, the CPU has been pretty sleepy according to RRD. Peaks of 25% with most daily usage under 20%.
(I'm assuming that's system total consider this is a dual-core w/HyperThreading mb)

Very nice.

The state table I've seen peak around 20,000 of 390,000. So.Also nice.

Thanks for the data. I'll keep sharing as I watch this systems first day in "production" mode. (scaaaarrryyy... hahahah)

 -Ben
--
Ben - O.D.T., S.P.

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #8 on: March 19, 2012, 11:38:55 am »
I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.

Agreed. I didn't think adding more MBUFs would be a big impact either. (thanks for the note though)

Cheers,

 -ben
--
Ben - O.D.T., S.P.

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #9 on: March 19, 2012, 12:25:42 pm »
BTW,

Does MBUF's recent over idle time or do they allocate and then (ideally) stick at some average level?

(i.e. what kind of behavior should I see if everything were running correctly?)

 -Ben
--
Ben - O.D.T., S.P.

Offline clarknova

  • Hero Member
  • *****
  • Posts: 920
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #10 on: March 19, 2012, 12:37:31 pm »
I've never seen them decrease except with a reboot. I think best case is that they level off and stop increasing at some point.
db

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #11 on: March 19, 2012, 01:10:47 pm »
K.

Good to know - I'll be watching.

 Thanks,

 -ben
--
Ben - O.D.T., S.P.

Offline mattlach

  • Jr. Member
  • **
  • Posts: 30
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #12 on: March 21, 2012, 12:50:37 pm »
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Forgive my possibly noobish question, but how do I set this setting?

Do I just add it on the:  "System -> Advanced -> System Tunables" page?

Does this setting require a reboot of the system?

If no, does changing the setting drop any current states?

Much obliged,
Matt

Offline bkamen

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #13 on: March 21, 2012, 12:55:04 pm »
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Forgive my possibly noobish question, but how do I set this setting?

Do I just add it on the:  "System -> Advanced -> System Tunables" page?

You have to look at the link above. There's a file in /boot you need to create and put those parms in.

Quote
Does this setting require a reboot of the system?
If no, does changing the setting drop any current states?

Yes. it needs a reboot. (so that should answer the 2nd part of that question)

It worked for me with no issues.. and I'm seeing a level off at about 7200 (7174) of 131072 after a day over heavy operation.

Shout if you need more,

 -Ben
--
Ben - O.D.T., S.P.

Offline clarknova

  • Hero Member
  • *****
  • Posts: 920
    • View Profile
Re: pfSense 2.0.1 locking up (hard)
« Reply #14 on: March 21, 2012, 12:56:32 pm »
Some settings can be set from the system tunables page, however this one cannot. You must add it to /boot/loader.conf.local (create the file if it does not exist) and then reboot for it to take effect. Do not add it to /boot/loader.conf, as this will be overwritten on the next system upgrade.
db