The pfSense Store

Author Topic: Very slow traffic from other VM's through pfSense on XenServer  (Read 33346 times)

0 Members and 1 Guest are viewing this topic.

Offline corotte

  • Jr. Member
  • **
  • Posts: 42
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #15 on: January 31, 2015, 02:07:52 pm »
damn !

but a quesiton remain ... was it working well in snapshot ? was it working well with previous version of xentool ?

in this thread
https://forum.pfsense.org/index.php?topic=86827.0
it look like to be an issue with xn nic ...
maybe a previous version should work ?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8827
  • Karma: +1007/-302
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #16 on: January 31, 2015, 02:12:26 pm »
No.

Just disable the tx/rx like in the above until FreeBSD and/or Citrix fixes it.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline corotte

  • Jr. Member
  • **
  • Posts: 42
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #17 on: January 31, 2015, 02:57:30 pm »
Ok

didi the above fix and it finally work.

Thanks folks !

Offline dsiminiuk

  • Newbie
  • *
  • Posts: 8
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #18 on: February 03, 2015, 09:43:55 pm »
My Internet speed normally is 20 Mb/s down and 2 Mb/s up.

I deployed pfSense 2.2-RELEASE X64 in XenServer 6.5

Without modification, the pfSense 2.2 would only muster 5 Mb/s down, and 0.06 Mb/s up. Painful.

I applied the changes to the LAN side VIF and the upload speed went back to full 2 Mb/s. The WAN speed did not improve.

I applied the changes to the WAN side VIF and the upload speed went back up to 20 Mb/s.

Eureka!




Offline Andy_

  • Newbie
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #19 on: February 05, 2015, 03:19:05 am »
It's just the tx-offload setting that needs to be changed, rx-offload is fixed-on.

I can confirm the problem and fix with Debian Wheezy/Xen 4.1.4 dom0.

ethtool -K ${dev} tx off in vif-bridge online did the trick.

The issue wasn't submitted to freebsd-bugs so far, now it is:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197344

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #20 on: February 06, 2015, 04:42:05 am »
You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

sudo ethtool -K $interface tx off

where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off

This has been documented in a different thread a couple of weeks ago. This goes for ALL drivers that remove checksumming as part of their 'optimisation'. The problem is that virtual IO drivers often use shared memory for fast communication, and since shmem is not the same as a bad write where packets might get corrupted, virtual IO developers often opt to disable checksumming since the packets wont corrupt anyway. But PF in BSD drops wrongly summed packets and there you have your problem. Disabling offloading forces software-calculated checksums (which is practually the same as 'offloading' to a 'software device' :p) and fixes this.

Solutions for this issue lie NOT with Xen, virtIO, Linux, BSD or pfSense, but with documentation and the users of pfSense.

Options that could be developed:

1. Xen/VirtIO/netfront detection: display a warning about shmem adapters not checksumming and how to act on both the GUI and the Console for pfSense

2. Have an option to make BSD's PF not drop packets with wrong checksums and recalculate them instead, or just not use checksums at all

Offline xlot

  • Jr. Member
  • **
  • Posts: 28
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #21 on: February 10, 2015, 04:44:20 pm »
Interesting - only appears to apply to virtual interfaces. 

My pfSense VM is running in xen 4.2 (Centos 6.6 dom0) and has no speed issues, but I'm using pci-passthrough to give 2 dedicated hardware NICs (off a dual-port Intel card) to pfSense for LAN/WAN  (so that DMZ/intranet are physically separate too).


Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #22 on: February 10, 2015, 06:27:36 pm »
Yes, it only has to do with virtIO and not with networking in general. Hell, it's basically a simple checksumming issue but it's only a big thing since 2.2 started supporting VirtIO and after the upgrade it automatically switches over to do that. :p

Offline bananaboy

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #23 on: February 25, 2015, 12:53:22 pm »
Thanks johnkeats for putting that up here. It really helped me sort this out.

One thing to note is disabling tx offload using ethtool -K does not persist across guest reboots or live migration because the dom-id and assigned vif changes, while xe vif-param-set other-config:ethtool-tx="off" does.

Is there any downside to using the vif-param-set option, or are the two basically equivalent?

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #24 on: February 28, 2015, 08:52:34 am »
Thanks johnkeats for putting that up here. It really helped me sort this out.

One thing to note is disabling tx offload using ethtool -K does not persist across guest reboots or live migration because the dom-id and assigned vif changes, while xe vif-param-set other-config:ethtool-tx="off" does.

Is there any downside to using the vif-param-set option, or are the two basically equivalent?

They are basically equivalent. ethtool is more for Xen using XenLight as a toolstack rather than XenServer (which is XAPI / XCP I believe, using xe instead of xl or xm). So if you want to persist on XenServer, use the xe command. On XenLight and other Xen's, stick the ethtool command in the vif-script of your choice so it changes the offloading settings once the vif gets attached.

Offline bennymundz

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #25 on: April 30, 2015, 03:48:37 pm »
You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

sudo ethtool -K $interface tx off

where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off


Sorry noob question here,

I am using a Xen implementation on a unraid distribution, when you say Dom0 side are you talking about the VIF that is spun up with the PFsense VM ? Like when i ifconfig to list my interfaces I just don't really know how to identify the interface you are referring to.

Sorry for the noob question again

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8827
  • Karma: +1007/-302
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #26 on: April 30, 2015, 04:41:02 pm »
It's all here:

https://forum.pfsense.org/index.php?topic=85797.msg475906#msg475906

I recently just rebuilt my test stack and all I did was the tx and rx on every NIC which is still probably more than is necessary but it worked.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #27 on: May 01, 2015, 01:54:44 pm »
You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

sudo ethtool -K $interface tx off

where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off


Sorry noob question here,

I am using a Xen implementation on a unraid distribution, when you say Dom0 side are you talking about the VIF that is spun up with the PFsense VM ? Like when i ifconfig to list my interfaces I just don't really know how to identify the interface you are referring to.

Sorry for the noob question again

Basically, when Xen starts a VM, the Domain ID gets appended to the VIF name. So if you start pfSense and it gets domain ID 123, the name you will see in ifconfig is something like vif123.0 for the first interface, vif123.1 for the second interface, etc. Sometimes, there are double interfaces, one with -emu on it, so you'd have vif123.0-emu as well.

So, if you are running non-enterprise Xen, you use XL or XM, and you can list your domains, like: sudo xl list. That will show you all domU's, and the ID's. Using ethtool you can then set the interface options.

You can also edit the vif-up scripts, or whatever vif-script is configured for your Xen setup, and have it do the ethtool magic when the interface is setup at domain startup.

It's all here:

https://forum.pfsense.org/index.php?topic=85797.msg475906#msg475906

I recently just rebuilt my test stack and all I did was the tx and rx on every NIC which is still probably more than is necessary but it worked.

Yes, but that usually applies to XE and not XM or XL installations :) Both are important of course, but the people using Xen sometimes don't know what they have :p so we need to know what they are using to give any useful comment :p

Offline bennymundz

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #28 on: May 04, 2015, 04:13:56 am »
You only need to disable checksum offloading on the hypervisor side of pfSense's interface.

Any interface that does DomU-DomU communication on pfSense's side produces un-checksummed packets which get dropped by PF in BSD.

sudo ethtool -K $interface tx off

where $interface is the VIF on the Xen Dom0 side is enough. Setting TX off on the bridge forces the Dom0 to calculate ALL checksums on ALL packets no matter where the come from or where they are going. This is not a smart idea since it creates a lot of calculations where they might not be needed. So if the pfSense DomU is on vif123.0 you run: sudo ethtool -K vif123.0 tx off


Thank you for taking the time to explain this, i turned the TX off on the pfsense vif and all was good. Happy days

Offline BBMitch

  • Newbie
  • *
  • Posts: 21
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #29 on: May 25, 2015, 03:45:09 pm »
Hello all...

Thanks for the information - sure helped us solve this but I have some more information that wasn't clear to me from all posted here.

This issue only seems to apply where Pf is communicating with hosts within the same xen host (dom0).

We use xenserver 6.2 fwiw. We have two xen dom0 - pf was natting for two services - one on dom0-a and one on dom0-b

pf itself was located on dom0-b
The dom0-a service worked perfectly after the update to 2.2.2 - the dom0-b service did not.

For people new to xenserver / for completeness, we used:
xe vm-list
#then find the uuid of your pf vm
xe vif-list vm-uuid={uuid of the vm from above}
#note the uuid of the vif - not the network you want to change!
#for each vif you can check the status:
xe vif-param-get uuid={uuid of vif} param-name=other-config
xe vif-param-set uuid={uuid of vif} other-config:ethtool-tx="off"

For what it's worth I was able to turn off tx on only the LAN interface (which nats for the dom0-b service).

I tried but did not need to keep offload off for the WAN interface which seems to get proper checksum as it leaves the dom0 through the physical nic.

Once complete you need to reboot the pf vm. the setting will persist across reboots.

Hope that helps someone else :-)

Mitch