Netgate SG-1000 microFirewall

Author Topic: igb 2.4.0 causing crashes  (Read 7417 times)

0 Members and 1 Guest are viewing this topic.

Offline maverick_slo

  • Hero Member
  • *****
  • Posts: 809
  • Karma: +37/-2
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #15 on: March 12, 2014, 10:58:40 am »
Yes, correct.
Snapshots will be soon online again as jimp posted here: https://forum.pfsense.org/index.php?topic=72763.msg401986#msg401986

Offline Renato Botelho

  • Administrator
  • Full Member
  • *****
  • Posts: 261
  • Karma: +43/-0
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #16 on: March 13, 2014, 08:39:45 am »
Yes it is in the latest ones.

I'm not getting any snapshots newer than what I'm on (Fri Mar 7 18:35:38 EST 2014).

Mar 12 snapshots are available
Renato Botelho

Offline Jason Litka

  • Hero Member
  • *****
  • Posts: 1294
  • Karma: +53/-1
    • View Profile
    • Utter Ramblings
Re: igb 2.4.0 causing crashes
« Reply #17 on: March 13, 2014, 03:42:29 pm »
So the good news is that it's not crashing any more.

The bad news is that I still seem to be hitting a pretty hard wall at ~2.1Gbit/s across 10Gb ix interfaces.
I can break anything.

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #18 on: March 13, 2014, 03:49:25 pm »
You need to do tuning for that.
It depends on traffic amount you are generating, what you are using to generate traffic etc...

Offline Jason Litka

  • Hero Member
  • *****
  • Posts: 1294
  • Karma: +53/-1
    • View Profile
    • Utter Ramblings
Re: igb 2.4.0 causing crashes
« Reply #19 on: March 13, 2014, 05:44:19 pm »
You need to do tuning for that.
It depends on traffic amount you are generating, what you are using to generate traffic etc...

I've applied the same tweaks I had done to my (now defunct) FreeNAS servers with no luck.  Those boxes had slower CPUs and were able to hit ~5-6Gbit/s between each other. Testing is with iperf.

If you have any specific tweaks in mind I'll definitely give them a go.
I can break anything.

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #20 on: March 13, 2014, 07:54:32 pm »
Start by sharing what you are doing!

Offline Klaws

  • Full Member
  • ***
  • Posts: 253
  • Karma: +14/-3
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #21 on: March 14, 2014, 04:25:38 am »
You might contemplate to check if you are CPU-bound or if something else is the issue.

Code: [Select]
top -SH usually gives an idea where the CPU time goes.

Offline Jason Litka

  • Hero Member
  • *****
  • Posts: 1294
  • Karma: +53/-1
    • View Profile
    • Utter Ramblings
Re: igb 2.4.0 causing crashes
« Reply #22 on: March 17, 2014, 09:56:08 am »
Start by sharing what you are doing!

Hardware Specs (both boxes are identical):
  • Intel E3-1245 V2 CPU (3.4GHz) w/ HT disabled
  • 16GB DDR3 ECC RAM
  • Intel 530 240GB SSD
  • (12) Intel i350 1Gbe
  • (2) Intel X520 10Gbe

Software Config:
  • iperf tests running across ix1 (have tried both SFP+ Direct Attach and Single-Mode OM3 patch with Intel SR optics directly between boxes, as well as running through a Cisco Nexus 5548UP)
  • Interface has simple any/any firewall rule
  • Snort is NOT running on these interfaces (though it is on others)

Tweaks in /boot/loader.conf.local:
  • kern.ipc.nmbclusters="262144"
  • kern.ipc.nmbjumbop="262144"
  • hw.intr_storm_threshold=10000

Setting MSIX on or off seems to make no difference and neither does setting the number of interface queues (have tried 1, 2, and 4).

Tweaks in System Tunables:
  • kern.ipc.maxsockbuf=16777216
  • net.inet.tcp.recvbuf_inc=524288
  • net.inet.tcp.recvbuf_max=16777216
  • net.inet.tcp.sendbuf_inc=16384
  • net.inet.tcp.sendbuf_max=16777216

Test Results (always +/- 2 Gbit/s, sometimes 1.8, sometimes 2.2):
  • iperf -c & -s = 2Gbit/s
  • iperf -c -d & -s = sum of both directions is 2Gbit/s (typically something like 1.8 and 0.2)
  • iperf -c -P2 & -s = sum of both threads is 2Gbit/s (typically something like 1.3 & 0.7)
  • iperf -c -P4 & -s = sum of all threads is 2Gbit/s (typically +/- 0.5 on each)

All 4 cores have an idle percentage in the 40-50% range even when running at the -P4 test.
I can break anything.

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #23 on: March 17, 2014, 12:12:36 pm »
You are sourcing traffic from the same box?

Offline Jason Litka

  • Hero Member
  • *****
  • Posts: 1294
  • Karma: +53/-1
    • View Profile
    • Utter Ramblings
Re: igb 2.4.0 causing crashes
« Reply #24 on: March 17, 2014, 04:21:15 pm »
I have two identical boxes.  For the purpose of testing throughput (before I route all the internal traffic from my servers through them) I have them connected directly to each other.
I can break anything.

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #25 on: March 17, 2014, 05:11:21 pm »
Well your result may vary here from the tool used.
Since there are many cores your program may bounce here and there so i do not think you can achieve stable results as that.

What i recommend you for ix devices is
Code: [Select]
hw.ixgbe.rx_process_limit=1024 #maybe higher or lower depends on testing
hw.ixgbe.tx_process_limit=1024

hw.ixgbe.num_queues=#ofcores you have

hw.ixgbe.txd=4096
hw.ixgbe.rxd=4096


Though these are very dependant on the workload you are trying to produce.

Also with single stream i am not sure with default parameters of iperf you can achieve 10G :).

Also remove this as well
hw.intr_storm_threshold=10000

Offline charliem

  • Sr. Member
  • ****
  • Posts: 565
  • Karma: +43/-1
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #26 on: March 17, 2014, 07:45:35 pm »
Give it another shot with new snapshots.

The panics have been resolved and let us know.

Any pointers to what the fix actually was?  I didn't see anything in redmine, or freebsd patches.  Course I haven't jumped through the hoops followed through to get access to the tools again.  Not sure it's worth it for a non-contributor, but active tester and curious code reader.

Offline adam65535

  • Sr. Member
  • ****
  • Posts: 326
  • Karma: +9/-0
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #27 on: March 17, 2014, 08:25:35 pm »
You are overthinking the fix I think.  I think the fix he is referring to is that thy reverted the drivers to the older versions.

Offline ermal

  • Hero Member
  • *****
  • Posts: 3832
  • Karma: +85/-5
    • View Profile
Re: igb 2.4.0 causing crashes
« Reply #28 on: March 17, 2014, 09:21:26 pm »
Actually the drivers are the latest found in FreeBSD.

The fix was involved in correcting the handling of the interface in FreeBSD 8 which is a bit of a mix compared to later ones.

Offline Jason Litka

  • Hero Member
  • *****
  • Posts: 1294
  • Karma: +53/-1
    • View Profile
    • Utter Ramblings
Re: igb 2.4.0 causing crashes
« Reply #29 on: March 17, 2014, 09:23:21 pm »
Well your result may vary here from the tool used.
Since there are many cores your program may bounce here and there so i do not think you can achieve stable results as that.

What i recommend you for ix devices is
Code: [Select]
hw.ixgbe.rx_process_limit=1024 #maybe higher or lower depends on testing
hw.ixgbe.tx_process_limit=1024

hw.ixgbe.num_queues=#ofcores you have

hw.ixgbe.txd=4096
hw.ixgbe.rxd=4096


Though these are very dependant on the workload you are trying to produce.

Also with single stream i am not sure with default parameters of iperf you can achieve 10G :).

Also remove this as well
hw.intr_storm_threshold=10000

Thanks, I'll give those a try tomorrow.

It's not so much the single stream performance I'm worried about.  It's more the fact that 2 or 4 threads produce the exact same throughput in aggregate but it doesn't appear that I'm CPU bound.
I can break anything.