pfSense Gold Subscription

Author Topic: IMPORTANT: Xen/KVM networking will not work using default hypervisor settings!  (Read 38696 times)

0 Members and 1 Guest are viewing this topic.

Offline fohdeesha

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +13/-1
    • View Profile
As far as I know, this has been fixed in FreeBSD 10.2 upstream.

sadly still open bug reports and workaround still needs to be done when using virtio drivers, as of today 10.2

Offline tootai

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Still true with 2.3.1_1

Offline nonooo

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
I can confirm this is still needed using debian Jessie & pfsense 2.3.1

so my next question is: does someone know how to configure my vm to always use the ethtool option? ( ethtool -K vif54.0 tx off)

nonooo

Offline johnkeates

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +52/-1
    • View Profile
so my next question is: does someone know how to configure my vm to always use the ethtool option? ( ethtool -K vif54.0 tx off)

Depends on your hypervisor. On XenServer with XE you use the xe command for those settings and to make them persistent. For Xen on Debian with no enterprise glue, and the xl command stack, you can edit the scripts depending on your network configuration. For example:

/etc/xen/scripts/vif-openvswitch on line 97 I added "ethtool -K ${dev} tx off". Then, that small command case looks like:

Code: [Select]
case "$command" in
    add|online)
        check_tools
        setup_virtual_bridge_port $dev
        add_to_openvswitch $dev
        ethtool -K ${dev} tx off
        ;;

    remove|offline)
        do_without_error ovs-vsctl --timeout=30 \
            -- --if-exists del-port $dev
        do_without_error ip link set $dev down
        ;;
esac

You could probably do it in xen-network-common.sh instead, on line 133, where you can also add "ethtool -K ${dev} tx off".
The add_to_bridge function there would then look like this:

Code: [Select]
# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2

    # Don't add $dev to $bridge if it's already on a bridge.
    if [ -e "/sys/class/net/${bridge}/brif/${dev}" ]; then
        ip link set dev ${dev} up || true
        return
    fi
    brctl addif ${bridge} ${dev}
    ip link set dev ${dev} up
    ethtool -K ${dev} tx off
}

Offline twx

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Can anyone confirm if this is still required with pfSense 2.3.3-RELEASE ?

Offline far.east

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Hi to All,

I just installed PFSense 2.2 on Xenserver 6.5 . So I got the same problem as described. And I want to share with my configuration fix for this problem.

Sorry if will be off topic I didn't find best place for this.  :)

Find your PFsense VM network VIF UUID's:
Code: [Select]
[root@xen ~]# xe vif-list vm-name-label="RT-OPN-01"
uuid ( RO)            : 08fa59ac-14e5-f087-39bc-5cc2888cd5f8
         vm-uuid ( RO): 0128bdba-df81-d729-ddbc-c60575e02624
          device ( RO): 1
    network-uuid ( RO): 7af0dc44-dc05-44f2-3741-883acb937747


uuid ( RO)            : 799fa8f4-561d-1b66-4359-18000c1c179f
         vm-uuid ( RO): 0128bdba-df81-d729-ddbc-c60575e02624
          device ( RO): 0
    network-uuid ( RO): 106ad80e-9522-77fd-3cc6-4b2b6fc03ecc

Then modify those VIF UUID's with this settings:
Code: [Select]
xe vif-param-set uuid=08fa59ac-14e5-f087-39bc-5cc2888cd5f8 other-config:ethtool-tx="off"
xe vif-param-set uuid=799fa8f4-561d-1b66-4359-18000c1c179f other-config:ethtool-tx="off"

And Shutdown VM and start again. Not restart PFSense from console.

org ganu kate jadi molek je... alhamdulilah

malaysian people says, it works like charms on xen7.0 with pfsense 2.3.3-RELEASE (amd64)! tq m8

Offline DrOffler

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Can anyone confirm if this is still required with pfSense 2.3.3-RELEASE ?

I've got a fresh install of  2.3.3-RELEASE-p1 (amd64), running on ovirt (kvm). My guests (only have linux guests) could ping out, but no udp or tcp.

Just selecting the "Disable hardware checksum offload" and rebooting (though I didn't check without rebooting) the pfsense instance did the trick.

I didn't need to change anything on the ovirt/kvm hypervisor - the default offload settings are in place. All guests (pfsense and the linux ones) are using virtio network drivers.

Thanks for all who have contributed to this thread - particularly johnkeates for the OP!

Offline opticalc

  • Newbie
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
i kind of had the same experience, CentOS7 KVM hypervisor, using bridges to create the VM, using virtio.  PFSense sees it as vmnet connections.  It had very slow network performance, but traffic did pass OK.  just would only do 15mbps on a 60mbps cable network conneciton.  I went to advanced-networking and disabled that hw checksum and rebooted, full 60mbps now.

Offline tzcole

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
I've put together a little script for XenServer that will use a TAG to indicate it wants CHECKSUM TX OFF.  The only missing piece is calling the script prior to VM startup.  I'll try to figure that out later.  After that piece is working, you should just be able to add the tag in XenCenter, and perform a XenServer managed reboot of the VM.   All NIC's for that VM will have the CHECKSUM TX disabled.

In the mean-time, after creating the script and adding a tag to the VM "cksum_offload_tx_off", you can run this script from Dom0.


Code: [Select]
# cat fix_cksum_offload.sh
VMUUIDS=`xe vm-list tags:contains=cksum_offload_tx_off | grep "^uuid" | awk '{print $NF}'`
for vmuuid in ${VMUUIDS}
do
    NICUUIDS=`xe vif-list vm-uuid=${vmuuid} | grep "^uuid" | awk '{print $NF}'`
    for nicuuid in ${NICUUIDS}
    do
        xe vif-param-set uuid=${nicuuid} other-config:ethtool-tx="off"
    done
done
« Last Edit: August 25, 2017, 11:07:06 am by tzcole »

Offline kripz

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Can anybody comment on the performance hit when disabling tcp offload?

Offline johnkeates

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +52/-1
    • View Profile
Can anybody comment on the performance hit when disabling tcp offload?

If you disable TCP Offload everything is fast. If turn TCP Offload on, everything is very slow.

Offline Loopa

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
This configuration really helps me alot..
Iíve always loved quotes because they express deep meaning in as few as words as possible.
 bodybuilding fitness quotes

Offline fohdeesha

  • Jr. Member
  • **
  • Posts: 63
  • Karma: +13/-1
    • View Profile
confirming this is still required on 2.4 :(

Offline ednc

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
I have just wasted a week on this issue.

It's a real pity that the pfSense Wiki has no "Xen/KVM Virtualisation" page, not even with anything more but the link to this topic.

Offline johnkeates

  • Hero Member
  • *****
  • Posts: 645
  • Karma: +52/-1
    • View Profile
I have just wasted a week on this issue.

It's a real pity that the pfSense Wiki has no "Xen/KVM Virtualisation" page, not even with anything more but the link to this topic.

It actually does:

https://doc.pfsense.org/index.php/VirtIO_Driver_Support
https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear
https://doc.pfsense.org/index.php/Virtualizing_pfSense_on_Proxmox