Netgate SG-1000 microFirewall

Author Topic: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached  (Read 4578 times)

0 Members and 1 Guest are viewing this topic.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« on: February 19, 2015, 10:22:57 am »
"kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached"
I've received this in syslog server, after that pfsense does not accessible via network.
https://lists.pfsense.org/pipermail/list/2015-January/007916.html Seems to be already reported.
And some similar problem here https://forum.pfsense.org/index.php?topic=84788.msg465221#msg465221

Since my hardware is quite different from what other reporters have, what exactly I must tune in freebsd settings?

My hardware is D2500CC and I believe that it have 82574L onboard NICs. Pfsense 2.2
« Last Edit: February 19, 2015, 12:16:01 pm by w0w »

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21739
  • Karma: +1503/-26
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #1 on: February 23, 2015, 11:40:22 am »
Since the error message specifically said kern.ipc.nmbjumbo9 limit reached, I'll go out on a limb and say you need to increase kern.ipc.nmbjumbo9 :-)

Or at least kern.ipc.nmbjumbop (See https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_ix.284.29_Cards for an example)
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #2 on: February 24, 2015, 01:43:27 am »
Recently I've tuned kern.ipc.nmbclusters="1000000"


$ netstat -m
2048/3772/5820 mbufs in use (current/cache/total)
1024/1512/2536/1000000 mbuf clusters in use (current/cache/total/max)
1024/1506 mbuf+clusters out of packet secondary zone in use (current/cache)
0/8/8/13291 4k (page size) jumbo clusters in use (current/cache/total/max)
1023/1940/2963/3938 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2215 16k jumbo clusters in use (current/cache/total/max)
11767K/21459K/33226K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/9/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #3 on: February 24, 2015, 01:50:25 am »
 
#sysctl kern.ipc.nmbjumbo9=11815
kern.ipc.nmbjumbo9: 11814
sysctl: kern.ipc.nmbjumbo9=11815: Invalid argument

Any number that differs from 11814 causes "Invalid argument", so I am not sure what is going on.

EDIT: Applied just for test

kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"

2047/2003/4050 mbufs in use (current/cache/total)
1023/1513/2536/1000000 mbuf clusters in use (current/cache/total/max)
1023/1507 mbuf+clusters out of packet secondary zone in use (current/cache)
0/3/3/524288 4k (page size) jumbo clusters in use (current/cache/total/max)
1023/8/1031/524288 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2215 16k jumbo clusters in use (current/cache/total/max)
11764K/3610K/15375K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/6/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

How can I define appropriate numbers for stable work?  I have only 2GB of RAM installed.


« Last Edit: February 24, 2015, 02:03:10 am by w0w »

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21739
  • Karma: +1503/-26
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #4 on: February 24, 2015, 07:27:58 am »
It's a loader tunable, so it must be set in /boot/loader.conf.local and only activates during a reboot.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #5 on: February 25, 2015, 10:14:12 am »
Which values are best for me? Near the maximum detected or twice or any mathematics based on stat else?
kern.ipc.nmbjumbop="?"
kern.ipc.nmbjumbo9="?"

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21739
  • Karma: +1503/-26
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #6 on: February 25, 2015, 11:57:11 am »
The link I posted above has a suggested value.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #7 on: February 26, 2015, 10:59:03 am »
Yes, it is, but only for nmbjumbop and Intel "ix" cards and mine is "em".
Thank you for your patience, jimp. I know that I am asking dumb questions.
Unfortunately I can not find any good guide for freebsd 10 about tuning those values and old guides are too old for 10.x version, I think. The only one good thing I've found is https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning but there is no any explanation... and it is mostly for 10 gigabit throughput, mine is 1 gigabit and hardware is not such powerful as described in article.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21739
  • Karma: +1503/-26
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #8 on: February 27, 2015, 08:29:27 am »
When in doubt, try 2x what you have now for a max.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline Harvy66

  • Hero Member
  • *****
  • Posts: 2360
  • Karma: +220/-12
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #9 on: February 27, 2015, 09:24:19 am »
fyi, 524288 9k jumbo frames is over 4GB and you only have 2GB. you're a bit over-subscribed. It won't be an issue unless your box actually attempts to use all of them, then bad things may happen when your kernel runs out of memory.

I think, anyway.

Offline w0w

  • Sr. Member
  • ****
  • Posts: 581
  • Karma: +35/-8
  • kernel panic attack
    • View Profile
Re: kernel: [zone: mbuf_jumbo_9k] kern.ipc.nmbjumbo9 limit reached
« Reply #10 on: February 28, 2015, 02:00:37 am »
Ok, thanks.
Tuned a little bit differently
kern.ipc.nmbclusters="131072"
kern.ipc.nmbjumbo9="20000"