The pfSense Store

Author Topic: Very slow traffic from other VM's through pfSense on XenServer  (Read 29676 times)

0 Members and 1 Guest are viewing this topic.

Offline mortenchristensen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Very slow traffic from other VM's through pfSense on XenServer
« on: December 23, 2014, 02:02:42 pm »
I have 2 XenServers, one with XenServer 6.2 and one with Xenserver Creedence beta 3.

Both have a pfSense 2.2 RC as router/firewall and a couple of Ubuntu Linux VM's and a windows-VM.

Traffic through both the physical xenserver-box and the virtual pfSense firewall goes at expected speeds.

But traffic from the other VM's on the same xenserver through the pfSense out on wan/internet goes very, very slow.
It goes so bad they cannot update themselve with apt-get.

When I try with iperf from a linux VM through the pfSense's WAN the speed is 3,82 KBits/sec.
The VM's and pfSense are connected with an internal single-server network (as OPT1), and tests to iperf server run on pfSense from a linux VM shows gigabit-speed.

One of the pfSense' has xen-tools installed. The other has not. I cannot se improvements with the tools installed.

One of the XenServers can get several public IP'numbers. On that I now have installed VM's with both an IPCop firewall and a Zentyal firewall.
When one of those new firewall-VMs' is default gateway for the ordinary VM's on the XenServer, their wan/internet-speed is normal.


Anobody with experience on XenServer as hypervisor, that can give me in a direction to experiment in to get traffic from VM's on the same Xenserver through pfSense up at useful performance ?

Offline cmb

  • Hero Member
  • *****
  • Posts: 11230
  • Karma: +884/-7
    • View Profile
    • Chris Buechler
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #1 on: December 24, 2014, 01:18:18 am »
Try disabling hardware checksum offloading under System>Advanced, Networking. TSO and LRO should also be disabled, though they likely already are since that's the default for those.

Which type of NIC is showing up in the VM? re0, em0, xn0?

Offline mortenchristensen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #2 on: December 24, 2014, 04:59:48 am »
Sorry.

Tried to disable hardware checksum offloading. The other 2 were disabled by default.

Did not improve the problem.

NIC's in the pfSense VM are nx0 to nx3

Offline mortenchristensen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #3 on: December 27, 2014, 05:18:46 pm »
New test with a pfSense 2.1.
Here internet-traffic from other VM's on the same Xenserver is normal.

The problem seems to be new in pfSense 2.2.

Offline cmb

  • Hero Member
  • *****
  • Posts: 11230
  • Karma: +884/-7
    • View Profile
    • Chris Buechler
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #4 on: December 27, 2014, 09:55:01 pm »
2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

Offline mortenchristensen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #5 on: December 28, 2014, 04:26:09 am »
> 2.1x wouldn't have xn NICs, it's specific to that. Can you force it to e1000 NICs on 2.2 and see?

On my 2.1.5 the nic's are called re. Can you give me some hints on, where abd how to change the driver ?

Offline phadm

  • Newbie
  • *
  • Posts: 2
  • Karma: +2/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #6 on: January 14, 2015, 08:52:58 am »
Hi,

i have the same problem with RC 2.2 (XenServer 6.2, SP1016, different platforms and nics) . The problem is the offload engine. If you route traffic between virtual hosts, you get tcp retransmissions, only a few sessions survive....

You have to disable the offload function at the VIF at the XenServer.
First identify the uuid of the VIF's:

xe vm-vif-list uuid=VMUUID

And disable the offload settings:

xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

shutdown / start the VM.

And now the disadvantage, whitout offload engine the TCP throughput falls on GBIT level over the vswitch. With offload I reach over 371 MBps with fetch, download the xencenter.iso from dom0 via http, whitout 98 MBps.

So who has a better solution, bring it on !!

 
« Last Edit: January 14, 2015, 09:30:46 am by phadm »

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7856
  • Karma: +922/-247
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #7 on: January 24, 2015, 02:44:09 am »
This all worked for me on the test stack I use which is now all 2.2-RELEASE.  I don't really care about performance much in this application, but before I did this it was useless.  Thanks much.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!

Offline mortenchristensen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #8 on: January 25, 2015, 01:35:45 pm »
Quote
First identify the uuid of the VIF's:
xe vm-vif-list uuid=VMUUID

And disable the offload settings:
xe vif-param-set uuid=VIFUUID other-config:ethtool-gso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-ufo="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tso="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-sg="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

shutdown / start the VM

Used this on both a XenServer 6.5 and a 6.2 later upgraded to 6.5. On both it has given other VM's internet-access again.

Run the xe commands on a Xenserver Private Network, so I hope the speed degrade will only occur on traffic that involves that net.
I think, both the pfSense VM and the other VM's need to be restartet to get useful speed.

Offline apollo13

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #9 on: January 27, 2015, 06:06:08 am »
You have to disable the offload function at the VIF at the XenServer.
First identify the uuid of the VIF's:

Which VIF? Local or WAN or both?

Thanks,
Florian

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7856
  • Karma: +922/-247
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #10 on: January 27, 2015, 06:18:12 am »
I did it on all.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!

Offline jpenninkhof

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #11 on: January 29, 2015, 06:32:13 am »
This helped me too. I only did this for my LAN port.

In my setup it seemed to be sufficient to execute:
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"
« Last Edit: January 29, 2015, 07:19:26 am by jpenninkhof »

Offline apollo13

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #12 on: January 30, 2015, 02:58:27 am »
This helped me too. I only did this for my LAN port.

In my setup it seemed to be sufficient to execute:
xe vif-param-set uuid=VIFUUID other-config:ethtool-tx="off"
xe vif-param-set uuid=VIFUUID other-config:ethtool-rx="off"

I can confirm that the LAN port should be enough. On a related note, did someone install the XenServer Tools in the VM?

Offline corotte

  • Jr. Member
  • **
  • Posts: 42
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #13 on: January 31, 2015, 01:45:22 pm »
Hi,

updated my XenServer 6.2 to 6.5 a few day ago with my VM pfsense 2.1.5 with no issue

updated pfsense to 2.2 WITH XENTOOLS (xe-guest-utilties 6.0.2_3) and got the same issue !

installed xentool using that method http://blog.feld.me/posts/2014/07/pfsense-on-citrix-xenserver/ (Thanks feld !)

look like issue remain even with Xentools :/

anyone can confirm ?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7856
  • Karma: +922/-247
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #14 on: January 31, 2015, 01:49:09 pm »
Yes.  It's broken.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help!