pfSense Gold Subscription

Author Topic: Is there a limit to the number of VMXNET3 NICs I can use in VMware?  (Read 901 times)

0 Members and 1 Guest are viewing this topic.

Offline MilkmanX

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Hello, I'm currently attempting to install using the latest version (pfSense-CE-2.3.3-RELEASE-amd64.iso) on a virtual machine in VMware vCloud.

For the VM I"m using the following settings:
     Other Linux 64 bit
     2 GB disk
     512 MB RAM
     5 x VMXNET 3 Network Adapters

I'm doing a quick install with all default settings and everything appears to go fine.  However, following the install, I'm unable to ping any of the interfaces and DHCP, etc. doesn't appear to be working.  The console is up but doesn't show any errors that I can see.  I'm able to assign interfaces and assign IPs from the console without any errors but this doesn't resolve the issue either.

What's interesting is that if I only use 4 VMXNET NICs everything works perfectly.  I can also get it to work with 5 E1000 NICs.  This leads me to believe there's some kind of limit or some other issue I'm running in to?

Thanks!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 20508
  • Karma: +1260/-10
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #1 on: March 20, 2017, 09:59:19 am »
There is no limit I know of BUT ESX does wonky things with the NICs after 4. They get interleaved in the probe order.

So you end up with:

1
5
2
3
4

or

1
5
2
6
3
7
4
8

So you have to be careful to line up the detected NICs, MAC addresses, interface assignments in pfSense and so on.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline MilkmanX

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #2 on: March 22, 2017, 01:39:40 pm »
Yes, I have seen it do strange things but I made sure to double check the order and confirmed the mac addresses matched the interfaces in the UI.  As it stands now 5 VMX nics do not work but the identical set up with 5 e1000 nics do.  For now I've just switched to e1000 so I can continue working but I do think there's something strange going on here.

Offline tortue

  • Newbie
  • *
  • Posts: 21
  • Karma: +3/-0
    • View Profile
I'm currently running pfSense 2.3.4 under ESXi 6.5 using 6 vmxnet3 nics. No issues here.

Offline fwcheck

  • Newbie
  • *
  • Posts: 14
  • Karma: +0/-0
    • View Profile
As far as i know, the maximum number of adapters within a virtual esxi-machine is 10.
https://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf

You can use 10 Adapters with no problems (ESXi 5.5/6.0/6.5).

Be aware that adapters might be reordered as mentioned above. As this is nasty you can either
- add 10 Adapters at start, leave unused adapters disconnected.
- use a different way (e.g. VLAN-Trunking) with one adapter

Offline tortue

  • Newbie
  • *
  • Posts: 21
  • Karma: +3/-0
    • View Profile
- use a different way (e.g. VLAN-Trunking) with one adapter

I was considering of going down that path of VLANing it. I didn't think I changed around my stuff that often but I've had to straighten out the nics 3-4 times now after about 4 months.

Anyone aware of anything in pfSense that cannot be used or is limited in some form when using VLANs vs dedicated nics? Looking specifically for pfSense items, I'm aware of the virtualization layer pieces like threaded interrupt requests, multi queues, etc.

I would hate to switch off of dedicated nics only to find something I may want to use and have to recreate what I have already.

Offline k-bull

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #6 on: June 07, 2017, 06:15:34 pm »
Hello,

I too am running into this problem.  I'm running ESXi 6.5d. I've been trying to add a 5th and 6th NIC using vmxnet3.  But there seem to be problem with the MAC's.  PFsense has MAC's for the extra nics but when I check the ESXi,  it has no MAC addresses.  I have redone the VM a few times , with fresh installs and still the same problem.  I believe there's a compatibility issue.  Mostly likely the problem is with ESXi. Can someone please confirm this.

Offline tortue

  • Newbie
  • *
  • Posts: 21
  • Karma: +3/-0
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #7 on: June 07, 2017, 09:16:08 pm »
I'm running 6.5 build 5310538 with a pfSense VM using 6 nics with no issues here (other than the reordering which is known).

Offline cbarber2

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #8 on: July 11, 2017, 06:29:19 pm »
similar problem here. esxi 5.5, pfs 2.3.4, all interfaces are vmxnet at this point.

I'm ok with 8, total disaster when I add the 9th. In my case something gets misconfigured and most or all of the interfaces stop talking. No pings, no nothing on the lan or other configured and known working subnets. But the basic interface list on the console screen is still the same. Remove the 9th nic, and it all goes back to normal. Really weird.

I have the ordering problems, etc, others have described too. took a long time to comprehend that the mac address was the only constant linking the pfs interface to the esxi virtual adapter.

I'll try the E1000 types. Why vmxnet? That's what the appliance is built with. Ironic that some other documentation I found about this appliance says the best choice is to use the E1000.

VLANs and trunking - as a last resort. These are all virtual subnets that will never leave the virtual environment, except possibly to traverse from one esxi host to another. Seems like overkill, more moving parts, more things to keep track of.

-Chris

Offline cbarber2

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Is there a limit to the number of VMXNET3 NICs I can use in VMware?
« Reply #9 on: July 11, 2017, 07:56:49 pm »
follow-up: with 8 vmx nics, adding a ninth nic with an E1000 causes no problem.