Netgate SG-1000 microFirewall

Author Topic: Very slow traffic from other VM's through pfSense on XenServer  (Read 33347 times)

0 Members and 1 Guest are viewing this topic.

Offline Gr1pen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #30 on: June 02, 2015, 01:17:33 pm »
I've been running pfsense 2.2 on XenServer 6.2 for a while with the mentioned offloads disabled and it's been working great. I believe since I upgraded to XenServer 6.5 (or when I upgraded to 6.5 SP1) pfsense only works as before on one specific host in the pool. I have 3 hosts in the pool and when pfsense is running on 2 of them it is very slow, but on the 3rd host it works fine.

How come..?? ???

Offline BBMitch

  • Newbie
  • *
  • Posts: 21
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #31 on: June 02, 2015, 01:25:22 pm »
Without knowing your network I can only guess... but see if this makes sense.

What I found was that if the pfsense was routing traffic for vm's on other systems (outside the xen box itself) then things worked - the offload worked as expected as the offload is added at the nic as the data leaves the xen server.

When I was routing traffic that was contained by the virtual network on the same xen host, that's when it didn't work - until I disabled the offloads - you only need to disable on the paths which you see the performance issues in my opinion - but you have to think it through.

Cheers.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 8827
  • Karma: +1007/-302
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #32 on: June 02, 2015, 02:34:07 pm »
The stack in the diagram in my sig is all on XenServer 6.5.  Works fine as long as the checksumming is turned off.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline Gr1pen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #33 on: June 03, 2015, 07:30:10 am »
Well, this issue is when traffic flows from external machines through pfsense wan-interface to resources on the internal lan.

The host on where this works has different hardware (including different NIC's) than the other two hosts in the pool. So when I migrate or restarts pfsense on  host 1 or 2 I don't get through the firewall from the outside (ia its so slow that it dont work). But with pfsense on host 3 it works as expected.

Before it worked on all 3 hosts. Now the pfsense is not protected against host failure.

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #34 on: June 03, 2015, 08:46:35 am »
Well, this issue is when traffic flows from external machines through pfsense wan-interface to resources on the internal lan.

The host on where this works has different hardware (including different NIC's) than the other two hosts in the pool. So when I migrate or restarts pfsense on  host 1 or 2 I don't get through the firewall from the outside (ia its so slow that it dont work). But with pfsense on host 3 it works as expected.

Before it worked on all 3 hosts. Now the pfsense is not protected against host failure.

What are the eth specs when it's failing? And is it a live migration or a shutdown-boot migration?
If you want to protect against failure, it's better to use pfSense's failover options instead of hypervisor-based failover.

Offline BBMitch

  • Newbie
  • *
  • Posts: 21
  • Karma: +1/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #35 on: June 03, 2015, 10:09:44 am »
I think he was trying to do that but he perceived one pfsense to work and two others not to work.

I'll try to explain it another way... the interface (if any) which transmits traffic to machines on the same physical xen server needs to have tx check sums turned off as I noted in my post. That's the only interface affected.

If you have a pf on xen and it does not route for any hosts on the same xen box you don't see any problem.

This would affect any traffic to which check sums would be applicable (all I think?) - so it would affect carp traffic too I imagine IF your pf boxes were on the same network - if they are on different boxes the carp traffic will be fine.

Just turn off the tx check sums for all the pfsense interfaces if you don't understand what I mean - the method I described surives rebooting and only affects the pf vms you apply the changes to.

Hope that clarfies. Cheers.

Offline Gr1pen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #36 on: June 03, 2015, 02:51:33 pm »
Perhaps my explanation was not so clear. The offload settings mentioned here has been applied on all interfaces of pf from the start when I was running it on XenServer 6.2. That fixed the problem then and pf worked perfectly fine on all 3 hosts. It was like living in a Dream where the streets where paved with gold and there was free candy for everyone.

After upgrading to XS 6.5/SP1 pf only works on 1 host. It doesnt matter if I live migrate or shut down and restart on Another host. It ONLY works on "host 3".

I am only running 1 instance of pfsense and sure it may be better running 2 or more in a HA  setup, but thats not really the question here. I had a fine working setup. But not anymore. The candy is all gone and the only change is XS that has been upgraded.

In reply to johnkeates I dont know what eht spec I should look into...?

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #37 on: June 03, 2015, 04:36:49 pm »
In reply to johnkeates I dont know what eht spec I should look into...?

Use XE to get all the vif specs from the working pf hypervisor and one non-functional hypervisor, as well as ethtool parameters for both.
We're looking for other variables that might mess with the in-memory transport, because that's where VirtIO related issues seem to lie.
If you could post those 4 outputs it'd help us diagnose.

Offline Gr1pen

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #38 on: June 04, 2015, 11:15:32 am »
My bad...

I noticed tht the interfaces on 2 failing XenServer hosts was reordered for some reason. Correcting this solved my problem, hence it was not related to pfsense.

I am thankful for your effort to help out and apologize for confusing you!

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #39 on: June 04, 2015, 11:23:07 am »
My bad...

I noticed tht the interfaces on 2 failing XenServer hosts was reordered for some reason. Correcting this solved my problem, hence it was not related to pfsense.

I am thankful for your effort to help out and apologize for confusing you!

Glad you got it fixed!

Offline viniciusferrao

  • Jr. Member
  • **
  • Posts: 28
  • Karma: +3/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #40 on: July 26, 2016, 11:34:52 am »
Just to keep this updated.

This problem still happens on XenServer 7.0 with pfSense 2.3.1.

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #41 on: July 26, 2016, 12:05:10 pm »
Just to keep this updated.

This problem still happens on XenServer 7.0 with pfSense 2.3.1.

Yep, until it's fixed in upstream FreeBSD it won't get fixed, ever.

Offline gothicman02

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #42 on: March 18, 2017, 05:45:07 am »
Just to keep this updated.

This problem still happens on XenServer 7.0 with pfSense 2.3.1.

Yep, until it's fixed in upstream FreeBSD it won't get fixed, ever.

Just figured I'd update this thread on these issues.  It looks like freebsd 11 is supporting dom0 support for xen, so hopefully these issues will be fixed.  I'm just getting a virtualized setup going with support ending for 32 bit here soon so I may try 2.4 of PFSense to see how it works out of the box with xen.

Here is a link to the freebsd support, though it will be experimental at this stage:

https://wiki.freebsd.org/Xen

Offline johnkeates

  • Sr. Member
  • ****
  • Posts: 460
  • Karma: +40/-1
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #43 on: March 18, 2017, 07:33:18 am »
Just to keep this updated.

This problem still happens on XenServer 7.0 with pfSense 2.3.1.

Yep, until it's fixed in upstream FreeBSD it won't get fixed, ever.

Just figured I'd update this thread on these issues.  It looks like freebsd 11 is supporting dom0 support for xen, so hopefully these issues will be fixed.  I'm just getting a virtualized setup going with support ending for 32 bit here soon so I may try 2.4 of PFSense to see how it works out of the box with xen.

Here is a link to the freebsd support, though it will be experimental at this stage:

https://wiki.freebsd.org/Xen

I suppose that could actually fix the netback/netfront problems because it will be BSD on the other end too. Interesting.

Offline gothicman02

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Very slow traffic from other VM's through pfSense on XenServer
« Reply #44 on: March 21, 2017, 08:06:09 pm »
Just to keep this updated.

This problem still happens on XenServer 7.0 with pfSense 2.3.1.

Yep, until it's fixed in upstream FreeBSD it won't get fixed, ever.

Just figured I'd update this thread on these issues.  It looks like freebsd 11 is supporting dom0 support for xen, so hopefully these issues will be fixed.  I'm just getting a virtualized setup going with support ending for 32 bit here soon so I may try 2.4 of PFSense to see how it works out of the box with xen.

Here is a link to the freebsd support, though it will be experimental at this stage:

https://wiki.freebsd.org/Xen

I suppose that could actually fix the netback/netfront problems because it will be BSD on the other end too. Interesting.

Yes very.  Although there is still some work to do.  I got the latest 2.4 snapshot running (as of March 18th) with FreeBSD 11.0-p8 under Xenserver 7.1 with all patches, and the issues with checksum offloading still exist.  Disabling it still fixes the issue through only on the rx and tx side, but I do believe there is a slight performance drop like others have said here.  I haven't tested local file transfers yet, but I do notice a slight drop in internet bandwidth.  I'll do more testing when I got time.