pfSense Support Subscription

Author Topic: [How-To] Using VMXNET2/3 NICs in pfSense 2.0  (Read 30595 times)

0 Members and 1 Guest are viewing this topic.

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
[How-To] Using VMXNET2/3 NICs in pfSense 2.0
« on: March 07, 2011, 07:22:11 am »
Quote
TL;DR version:
Install Open-VM-Tools package.
Add vmxnet_load="YES" to /boot/loader.conf
Copy vmxnet.ko to /boot/kernel/
Chmod to 0555
Replace old NICs with VMXNET2

0. If you don't already have a VM running pfSense, then create a new one. Otherwise you can safely skip to step 1.
Select the "Other" option in "Guest Operating System:" and choose "FreeBSD (32-bit)" or "FreeBSD (64-bit)"
I'd recommend the 64 bit version as it won't have the 4GB limit when it comes to traffic counters on your interfaces, same might hold true for other statistic counters elsewhere in pfSense.
(A small note, using the 32 bit version won't give you any fuctional issues, the various counters will just wrap when they reach 4GB and start over).
In the "Create Network Connections" section, choose at least two NICs (one for WAN and one for LAN traffic) but don't use the "Flexible" adapter if that option is on the list of adapters.
Instead use the "E1000" adapter for both interfaces (provides better spped and lower CPU usage).
Now start the new VM and load the pfSense iso of choice (32 or 64 bit) into the VM's CD/DVD drive. Install as you would normally, the E1000 adapters should show up as em0 and em1.

1. For this to work, it requires that the pfSense firewall have internet access on its WAN interface for downloading packages.
Hit up the web interface of your pfSense installation and go to the "System -> Packages" menu. Find the "Open-VM-Tools" package in the list and install it.

2. Go to the "Diagnostics -> Edit File" menu and browse for "/boot/loader.conf". Add the following at the end of the file if its not there already:
Code: [Select]
vmxnet_load="YES"(a blank line at the end of the file might be needed for the system to read the file properly?
Anyway I've added one just to make sure, so haven't tested if this is really needed or not).
Remeber to save the file!

3. Open the console of the VM and hit "8" for the "Shell" option. Type
Code: [Select]
find /* |grep vmxnet.ko and hit enter. Verify that a copy of vmxnet.ko is located in
Code: [Select]
/boot/kernel/ if not, copy vmxnet.ko to that location from any other vmxnet.ko file present on the system.
In my case the earlier find command found a vmxnet.ko file in /usr/local/lib/vmware-tools/modules/drivers/
If you have the file located in the same place you can use this command to copy it to the proper location:
Code: [Select]
cp /usr/local/lib/vmware-tools/modules/drivers/vmxnet.ko /boot/kernel/
4. chmod the vmxnet.ko file to make it executable using this command:
Code: [Select]
chmod 0555 /boot/kernel/vmxnet.koWhen done type:
Code: [Select]
exitand hit enter.
At the pfSense console menu hit "6" (Halt system) to power off the VM.

5. Edit your VM configuration settings and remove all the NICs from the configuration. Save the changes and open the VM configuration settings again, this time add a new Ethernet Adapter. In "Adapter Type" this time choose "VMXNET2 (Enhanced)" Add as many NIC's as you need for your environment and make sure that all of them are the VMXNET2 (Enhanced) type. Save the VM configuration and power up the VM.

6. During the bootup of your pfSense firewall it'll prompt you to assign the new NICs do to "Network interface mismatch". The new interfaces should show up as vxnX (e.g. vxn0, vxn1 etc.)
Thats it and you're done.

Edit I forgot to add that this approach might also work for the 1.2 series of pfSense, but I haven't tested it.

Tags: ESX ESXi Virtual Machine VMware Tools
« Last Edit: August 28, 2011, 08:42:52 am by pfSense.User.1138 »

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #1 on: March 10, 2011, 02:42:25 am »
Appereantly either the "VMXNET2 (Enhanced)" NIC or the vmxnet.ko driver or both does not support vlans and as such they don't get listed as an optional "Parent interface" when trying to configure vlans in the "Interfaces -> (assign)" menu.
I've tried both the latest vmxnet.ko driver from the Open-VM-Tools package and tried the latest vmxnet.ko driver from the official VMware-Tools on my host.
Could anyone confirm that the "VMXNET2 (Enhanced)" NIC does indeed not support vlans or is it a shortcomings in the freebsd vmxnet.ko driver ?
« Last Edit: August 28, 2011, 08:41:52 am by pfSense.User.1138 »

Offline AhnHEL

  • Sr. Member
  • ****
  • Posts: 572
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2 NICs in pfSense 2.0
« Reply #2 on: March 10, 2011, 04:34:29 am »
Doesnt seem to support ALTQ either so traffic shaping isnt possible.  E1000 seems to be the tried and true method on pfSense.
AhnHEL (Angel)
NYC

3 pfSense sites: 2.1 RELEASE (amd64)
Dell 745 SFF E4400 @ 2.0Ghz, 2GB RAM, 20/5 Mbps
Dell 755 SFF E6550 @ 2.3Ghz, 2GB RAM, 20/5 Mbps
White Box i5 3570k @ 4.4Ghz, 16GB RAM, 114/6 Mbps, SSD
OpenVPN (Site to Site, Road Warrior), IPSec Mobile, UPnP Gaming, Traffic Shaping, Snort, Suricata

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #3 on: March 10, 2011, 05:27:15 am »
So basicly using the VMXNET2 NIC, you'll get better throughput and lower cpu utilization at the expense of features.
If just pfSense/FreeBSD had a working driver for the VMXNET3 NIC :-(
Don't think the VMXNET2 NIC or driver will ever get those features, afaik VMware have stopped development of that NIC entirely.
« Last Edit: August 28, 2011, 08:41:34 am by pfSense.User.1138 »

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #4 on: April 21, 2011, 08:05:15 am »
It seems that the Open-VM-Tools package is currently broken.
As a workaround it is possible to use the regular vendor supplied VMware Tools following these instructions:

1. Install prerequisites in the VM console:
When running on a 64 bit installation, run:
Code: [Select]
pkg_add -r compat6x-amd64When running on a 32 bit installation, run:
Code: [Select]
pkg_add -r compat6x-i386(Note for 32 bit: not tested but this should be it)

2. Install Perl:
Code: [Select]
pkg_add -r perl
3. Link the newly installed libraries so the VMware Tools installer can find it:
Code: [Select]
ln -s /usr/local/lib/compat/libm.so.4 /lib
ln -s /usr/local/lib/compat/libc.so.6 /lib
ln -s /usr/local/lib/compat/libthr.so.2 /lib

4. Install VMware Tools:
Select the "Install/Upgrade VMware Tools" option on the VM in the vSphere Client.
Back in the VM console type:
Code: [Select]
cd /mnt
mkdir cdrom
mount t cd9660 /dev/acd0 /mnt/cdrom
cd /tmp
tar zxvf /mnt/cdrom/vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
(Perl script, this is why Perl is installed)
Accept all defaults (press enter 8 times)

5. To clean up:
Code: [Select]
cd ..
rm -r vmware-tools-distrib

All credits go to OnHEL.
Original post linked.
« Last Edit: August 28, 2011, 08:41:21 am by pfSense.User.1138 »

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #5 on: August 28, 2011, 08:27:13 am »
Since ESXi 5.0 it is now possible to use VMXNET3 NIC's in pfSense using the vendor surplied VM-Tools.
Unfortunately the VMXNET3 NIC also seems to be missing the VLAN tagging and ALTQ features on FreeBSD.
I know the VMXNET3 NIC does support VLAN tagging in other OS's, odds is it must be an issue with the FreeBSD driver.
« Last Edit: August 28, 2011, 08:41:03 am by pfSense.User.1138 »

Offline photonman

  • Jr. Member
  • **
  • Posts: 79
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #6 on: September 18, 2011, 12:33:16 am »
Since ESXi 5.0 it is now possible to use VMXNET3 NIC's in pfSense using the vendor surplied VM-Tools.
Unfortunately the VMXNET3 NIC also seems to be missing the VLAN tagging and ALTQ features on FreeBSD.
I know the VMXNET3 NIC does support VLAN tagging in other OS's, odds is it must be an issue with the FreeBSD driver.

I am getting ready to build a ESXi 5 pfSense vm.  If I do not need those missing features, is the VMXNET3 going to perform better then the E1000?

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #7 on: September 21, 2011, 09:10:34 am »
I am getting ready to build a ESXi 5 pfSense vm.  If I do not need those missing features, is the VMXNET3 going to perform better then the E1000?
It sure is.

Offline photonman

  • Jr. Member
  • **
  • Posts: 79
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #8 on: September 29, 2011, 10:03:35 am »
I downloaded the packages manually from ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/  but now I am stumped as to how to load them in pfsense?

[UPDATE]...I got it loaded... used

pkg_add -rv ftp://ftp.freebsd.org/pubFreeBSD/ports/i386/packages-8-stable/Latest/

and assigning the interfaces was a tricky thing because it gave the valid interfaces as interface name + MAC address so for example:

vmx3f000:12:34:56:78:90:bb
vmx3f100:12:34:56:78:90:hh

You have to realize the interface names are vmx3f0 and vmx3f1

THANK YOU so much for the great instructions as I could have never done this without...

Notice that these files exist in /boot/modules and not in /boot/kernel like vmxnet.ko so is that a problem???

vmmemctl.ko
vmblock.ko
vmxnet3.ko

but I get this with kldstat which is good but no vmhgfs which could be a change in ESXi 5 vmtools?

$ kldstat
Id Refs Address    Size     Name
 1   10 0xc0400000 11796f4  kernel
 2    1 0xc157a000 5684     vmblock.ko
 3    1 0xc1580000 3404     vmmemctl.ko
 4    1 0xc1584000 4988     vmxnet.ko
 5    1 0xc1589000 8608     vmxnet3.ko

but then when I add the vmxnet 3 nics to the vm and reboot I get this:

$ ps ax|grep vmware
19319  ??  S      0:00.00 sh -c ps ax|grep vmware
19733  ??  S      0:00.00 grep vmware

which means the tools are not running?

and $ find /* |grep vmware-guestd

gives me nothing???

so I reinstalled the package from the shell and now I get this so perhaps vmware-guestd is now vmtoolsd and in VMware client it actually says VMware Tools: Running (Current)

$ ps ax|grep vmware
12758  ??  S      0:00.00 sh -c ps ax|grep vmware
12846  ??  R      0:00.00 grep vmware
15633  ??  S      0:00.11 /usr/local/lib/vmware-tools/sbin/vmtoolsd

but as soon as I reboot, the VMware says VMware Tools:Not Running (Current) and grep vmware does not show vmtoolsd?

I am hoping that I am way ahead of the pack and the current documentation cannot explain ESXi 5 vmtools on a pfSense 2.0 machine.



« Last Edit: September 29, 2011, 04:41:15 pm by photonman »

Offline biggsy

  • Sr. Member
  • ****
  • Posts: 381
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #9 on: October 02, 2011, 09:50:18 pm »
Quote
and assigning the interfaces was a tricky thing because it gave the valid interfaces as interface name + MAC address so for example:

I honestly can't remember where I first came across this script.  It may have been on this forum.  It certainly helps in figuring out which MAC address belongs to which interface when assigning them in pfSense.  

EDIT:
Info - http://www.virtuallyghetto.com/2011/05/how-to-query-for-macs-on-internal.html
Download - http://vghetto.svn.sourceforge.net/viewvc/vghetto/other/vswitchInfo.sh?view=log

Biggsy



« Last Edit: October 02, 2011, 10:14:01 pm by biggsy »

Offline wagebox

  • Newbie
  • *
  • Posts: 11
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #10 on: October 16, 2011, 02:35:28 pm »
Hello everyone,

I have managed to get the vmxnet3 working under pfsense 2.0 runing on shiny new esxi 5.0 host (build 474610).
I have installed the native vmware tools using guide in this thread. vmxnet3 interfaces are working fine so far, but I have come across one weird issue. According to vmware configuration maximums, it is possible to assign up to 10 virtual nics per vm. However, as soon as I assign four or more vmxnet3 adapters to pfsense vm, network gets cut off. Link is up, I can ping hosts from pfsense, but it is not possible to ping or connect to pfsense, it is not accepting any incomming connections or routing traffic. I have tried to reset pfsense to factory defaults, but that did not help. If I remove just one vmxnet3 adapter, everything is back to normal.
This issue is not manifested when using e1000 adapters.
I am by no means freebsd guru so I cannot do in depth troubleshooting, but if someone is interested I can provide additional data when instructed.

In the meantime, back to e1000 adapters :)

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #11 on: December 31, 2011, 01:50:34 am »
However, as soon as I assign four or more vmxnet3 adapters to pfsense vm, network gets cut off. Link is up, I can ping hosts from pfsense, but it is not possible to ping or connect to pfsense, it is not accepting any incomming connections or routing traffic. I have tried to reset pfsense to factory defaults, but that did not help. If I remove just one vmxnet3 adapter, everything is back to normal.
That is strange. I use six vmxnet3 adapters with no problem at all.

Offline pfSense.User.1138

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0 (pfSense 2.0.1 Note)
« Reply #12 on: December 31, 2011, 02:11:17 am »
I've just updated from 2.0 Release to 2.0.1 Release and found the vendor supplied VMware Tools failed to start on boot after the upgrade.
But fortunately its a quick and easy fix.

On the console open the shell ( press 8 )
Then type:
Code: [Select]
vmware-config-tools.plWhen vmware-config-tools.pl is done running type:
Code: [Select]
exitTo leave the shell again.

VMware Tools status for the pfSense VM in your vSphere Client should now be "Running".
As an extra check, try reboot the VM to verify that VMware Tools is running when pfSense is back up.

Offline zachsaw

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #13 on: January 24, 2012, 05:20:08 pm »
Hi guys,

I've managed to install vendor supplied vmware tools in pfsense 2.0.1.
However, upon changing e1000 to vmxnet3, my WebGUI slows down to a halt. All other traffic runs at full speed except for the WebGUI.

I also notice however that if I run "Assign Interfaces" from shell and go through the prompts again, WebGUI will become responsive. At least until the next reboot.

This sounds as though the vmxnet3.ko driver must be reloaded after VMWare Tools memory manager has been loaded.

I guess alternatively, if I could just do the last part of "Assign Interfaces" where it reloads interfaces, it would work as well.

So my question is, where can I find the script that does "Assign Interfaces"?

Or better yet, does anyone know what it does to reload interfaces after writing to configuration file?


EDIT: Found it -- /etc/rc.reload_interfaces
« Last Edit: January 24, 2012, 06:20:59 pm by zachsaw »

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 2893
  • Karma: +0/-0
    • View Profile
Re: [How-To] Using VMXNET2/3 NICs in pfSense 2.0
« Reply #14 on: April 20, 2012, 12:40:27 am »
Ok I was using the vmxnet3 drivers without any issue -- I thought!

But had not had need in a while to get onto work vpn..  Last night got a call, and I could not get in..  Spent this evening troubleshooting it, and nothing was jumping out at me.. 

Even enabled oakley.log on the box -- nothing jumped out why it was just freaking hanging..  Its a BS custom client from the mother company in DE..  Logs are useless in the thing, etc.  Anyway I went back to the e1000 and bam connected right in..  No issues at all.

If I get a chance I will try and figure it out - maybe this weekend, but for me had to go back to the e1000 drivers.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Have I helped you, want to say thanks?  Donate to pfsense the cost of a beer http://pfsense.org/donate.html