Netgate SG-1000 microFirewall

Author Topic: netmap errors in console  (Read 264 times)

0 Members and 1 Guest are viewing this topic.

Offline whizzy

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
netmap errors in console
« on: December 11, 2017, 08:45:35 am »
Hi all,
I have finally installed Suricata and enabled inline mode. I setup the sid management as per others recommendations.
I am running PFsense version 2.4.2. Using one of the recommended NIC's, i350T2, using the igb driver.
All seems to be running well. Getting the proper Suricata alerts.

Issue is I get tons of netmap.grab_packets bad pkt listings in the GUI and system log. No errors in the Suricata logs.
Are these errors to be ignored, or is there a problem with Suricata?

Here is a sample. There were 250+ of these overnight.
Dec 11 07:04:53    kernel       093.421388 [1071] netmap_grab_packets bad pkt at 762 len 2333
Dec 11 07:04:53    kernel       093.400226 [1071] netmap_grab_packets bad pkt at 759 len 2333
Dec 11 06:30:16    kernel       016.480547 [1071] netmap_grab_packets bad pkt at 828 len 2313
Dec 11 06:30:16    kernel       016.480523 [1071] netmap_grab_packets bad pkt at 827 len 2313
Dec 11 06:30:16    kernel       016.435130 [1071] netmap_grab_packets bad pkt at 818 len 2313
Dec 11 06:30:16    kernel       016.434499 [1071] netmap_grab_packets bad pkt at 816 len 2313
Dec 11 06:30:16    kernel       016.413635 [1071] netmap_grab_packets bad pkt at 806 len 2313
Dec 11 06:13:06    kernel       986.166554 [1071] netmap_grab_packets bad pkt at 1392 len 2546
Dec 11 06:04:50    kernel       490.242882 [1071] netmap_grab_packets bad pkt at 637 len 2333
Dec 11 06:04:50    kernel       490.220744 [1071] netmap_grab_packets bad pkt at 634 len 2333
Dec 11 06:00:01    php       [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Dec 11 06:00:00    check_reload_status       Reloading filter
Dec 11 06:00:00    php       /usr/local/www/pfblockerng/pfblockerng.php: The command '/sbin/ifconfig 'igb3' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 11 06:00:00    php       [pfBlockerNG] Starting cron process.
Dec 11 05:34:50    kernel       689.947109 [1071] netmap_grab_packets bad pkt at 1356 len 2333
Dec 11 04:34:50    kernel       090.163484 [1071] netmap_grab_packets bad pkt at 930 len 2333
Dec 11 04:34:50    kernel       090.140454 [1071] netmap_grab_packets bad pkt at 927 len 2333
Dec 11 04:09:12    kernel       552.264122 [1071] netmap_grab_packets bad pkt at 708 len 2546
Dec 11 04:09:12    kernel       552.201909 [1071] netmap_grab_packets bad pkt at 701 len 2546
Dec 11 04:09:12    kernel       552.170906 [1071] netmap_grab_packets bad pkt at 695 len 2534
Dec 11 04:00:46    kernel       046.108484 [1071] netmap_grab_packets bad pkt at 171 len 3104
Dec 11 03:39:14    check_reload_status       Reloading filter

Are there any tunables I should be using? And yes, I have followed the tuning guide already. I saw in other posts this issue mentioned, but never an explanation as to what causes this other than the NIC is not supported. Which is not my issue. Should I disable inline mode?

Thanks in advance for your support for a great product.
« Last Edit: December 11, 2017, 09:25:55 am by whizzy »

Offline whizzy

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: netmap errors in console
« Reply #1 on: December 12, 2017, 08:36:32 am »
No one?

I guess the netmap errors are a mystery and Inline mode is not a useable feature. Shame.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3403
  • Karma: +895/-0
    • View Profile
Re: netmap errors in console
« Reply #2 on: December 12, 2017, 03:49:10 pm »
When you say you followed the tuning guide, that includes turning off all the hardware checksum offloading, LRO and segmentation stuff.  Have you done that?  If so, you just may have one of those NICs that is partially supported.  If it seems to work and is just spamming the console, maybe you live with it until Netmap is more mature.  This may be something you post to the FreeBSD folks as that's where the driver will get updated.

Bill

Offline whizzy

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: netmap errors in console
« Reply #3 on: December 14, 2017, 10:19:07 am »
Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.

I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.

But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.

I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.

Thanks
« Last Edit: December 14, 2017, 10:39:41 am by whizzy »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3403
  • Karma: +895/-0
    • View Profile
Re: netmap errors in console
« Reply #4 on: December 14, 2017, 10:45:40 am »
Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.

I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.

But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.

I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.

Thanks

I am not knowledgeable enough about the FreeBSD kernel to say with certainty, but my guess would be Netmap may be falsely reporting bad packets.  It's true you will get some now and then, but I would not think you would see very many at all in a switched LAN environment.

Bill

Offline whizzy

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: netmap errors in console
« Reply #5 on: December 14, 2017, 12:26:34 pm »
I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
This can't be a coincidence.
Are there any scheduled events outside of the ones that Cron performs in PFsense?
Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.

Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3403
  • Karma: +895/-0
    • View Profile
Re: netmap errors in console
« Reply #6 on: December 14, 2017, 04:01:17 pm »
I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
This can't be a coincidence.
Are there any scheduled events outside of the ones that Cron performs in PFsense?
Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.

Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.

There are some other tasks that happen on a periodic basis, but I do not know what all they are.  That would be a question better answered by a member of the pfSense developer team.

Bill

Offline whizzy

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: netmap errors in console
« Reply #7 on: December 14, 2017, 04:12:36 pm »
Where do I submit a question for the devs? GitHub?

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3403
  • Karma: +895/-0
    • View Profile
Re: netmap errors in console
« Reply #8 on: December 15, 2017, 07:33:20 pm »
Where do I submit a question for the devs? GitHub?

Probably the best place is here at the Redmine bug site:  https://redmine.pfsense.org/projects/pfsense (for reporting the Netmap bug along with supporting evidence/clues you've gathered).

If you just want to ask a question, that is better done in one of the forum threads such as General Questions.

Bill