Netgate SG-1000 microFirewall

Author Topic: Xen PV domU - How to run pfSense in paravirtulised mode  (Read 13347 times)

0 Members and 1 Guest are viewing this topic.

Offline nexus

  • Jr. Member
  • **
  • Posts: 39
  • Karma: +1/-0
    • View Profile
Xen PV domU - How to run pfSense in paravirtulised mode
« on: June 11, 2011, 12:12:20 pm »
The main purpose of this topic is to provide a simple way and share patches to build a pfSense DomU in paravirtualised mode for Xen Hypervisor.

Why I'm doing that : For the challenge and because my personnal Xen Hypervisor Host does'nt have any Virtualisation set built in. (Atom)

Good to know before building
Xen Kernel for domU in RELENG_8_1 (FreeBSD 8.1) is not really stable, I got a lot of panic and a lot of feature where not present at this stage of freeze in Xen code.

I started in two way :
- Try to backport Xen kernel code the the RELENG_8_1 kernel, hard work and I'm not familliar with code versionning for FreeBSD so I give up.
- Build pfSense based on RELENG_8_2 (Xen kernel is more complete at this stage).

Building your own pfSense domU

1 - Before anything else, you must install a freebsd8.1 from scrach (I run it on another VM)
2 - Follow the instruction to setup your own pfSense build scratchbox ( )
3 - Make sure that the building process is working by following the instruction in the wiki
4 - Now we need to make some custom things in order to make it build for RELENG_8_2

Customisation 1 : Make the Kernel Xen Aware
Create your own KERNCONF file to add XEN support (and keeping pfsense stuff in it). For this tack I took the pfsense_wrap.8.i386 file and the XEN one from freebsd sources and mixed them.

Warning XEN is including PAE so there some device to remove (scsi stuff, and some others) because they stop when compiling complaining of memory. In my file I remove all the devices including wireless because I don't need them at the moment. Feel free to custom yourself if you need specific device with PCIpasstrough from Xen (excep the need of wireless device, I don't see others to keep in domU).

See below for my pfSense_wrap.8.XEN file.

To use your custom file, override the existing one (make a backup before) in /home/pfsense/tools/builder_script/conf/pfSense_wrap.8.i386

Next you have to modify /home/pfsense/tools/builder_scripts/conf/src.conf.embedded.8 to add the module we don't want to build (using makeoption WITHOUT_MODULES in KERNCONF seems not working fine with RELENG_8_2). So add :
Code: [Select]
WITHOUT_MODULES= aha ahb amd cxgb dpt drm hptmv ida malo mps mwl nve sound sym trm xfs
Customisation 2 : Modify some scripts to allow Kernel get compiled

Next we need to remove a line in ome/pfsense/tools/builder_scripts/ because we add an override witch is not compatible with XEN Kernel. Modify the line :

Code: [Select]
export MODULES_OVERRIDE=${MODULES_OVERRIDE:-"i2c ipmi acpi ndis ipfw ipdivert dummynet fdescfs cpufreq opensolaris zfs glxsb runfw if_stf"}

Code: [Select]
export MODULES_OVERRIDE=${MODULES_OVERRIDE:-"ipfw ipdivert dummynet fdescfs runfw if_stf"}

Customisation 3 : Modify patches to work with RELENG_8_2

You can find attached the modified diff file to work with RELENG_8_2, copy/replace them to /home/pfsense/tools/patches/RELENG_8_2

Code: [Select]

OK we are almost done now !

Customisation 4 : Set the right release to build

Use the and go to version to configure building pfSense_2_0 with FreeBSD_8_1

Exit the and go to modify /home/pfsense/tools/builder_script/pfsense-locale.conf and replace any RELENG_8_1 occurences with RELENG_8_2.

We are ready to build !

Go to the and select build nanobsd image, if I don't miss anything at this point, it should compile and build without errors.

How to install my fresh image as domU ?

Is fairly simple, locate the img you just build (normaly /tmp/builder/nanobsd.full.img) and copy it on your dom0.

Locate the kernel in building tree /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386/kernel  and copy it on your dom0

In my configuration I use lvm as disk management for my domU but it should work with any other disk method.

Dump the nanobsd.img with dd as below :
Code: [Select]
dd if=nanobsd.full.img of=/dev/myVG/pfsense_rootfs bs=16k
and setup you domU according to
Code: [Select]
kernel = '/boot/pfsense/kernel-pfsense-xen-domU-8.2'
extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
disk = ['phy:/dev/myVG/pfSense_rootfs,xvda,w']
memory = 256
bootload = '/usr/bin/pygrub'
vcpus = 1
vif = ['mac=00:16:3e:0f:12:df, bridge=br8021q,model=', 'mac=00:16:3e:45:18:2a, bridge=eth0,model=']

Your pfsense domU should start !

Some notes and todo ...

Using vlan with xen network interface need a patch :
Code: [Select]
--- sys/dev/xen/netfront/netfront.c  2010-11-24 02:03:03.000000000 +0100
+++ sys/dev/xen/netfront/netfront.c  2011-05-29 17:30:30.000000000 +0200
@@ -2067,6 +2067,10 @@
        ifp->if_capenable = ifp->if_capabilities;
+       /* Add VLAN support capabilite for xn if
+        ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU;
+        ifp->if_capenable |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU;

        ether_ifattach(ifp, np->mac);
        callout_init(&np->xn_stat_ch, CALLOUT_MPSAFE);

If you got some error like : xennet_get_responses: too many frags 6 > max 5

here is the patch
Code: [Select]
--- head/sys/dev/xen/netfront/netfront.c 2011/01/29 00:53:58 218055
+++ head/sys/dev/xen/netfront/netfront.c 2011/01/29 02:36:45 218056
@@ -1273,7 +1273,6 @@
  struct mbuf *m, *m0, *m_prev;
  grant_ref_t ref = xennet_get_rx_ref(np, *cons);
  RING_IDX ref_cons = *cons;
- int max = 5 /* MAX_TX_REQ_FRAGS + (rx->status <= RX_COPY_THRESHOLD) */;
  int frags = 1;
  int err = 0;
  u_long ret;
@@ -1416,20 +1415,10 @@
  *list = m0;
- if (unlikely(frags > max)) {
- if (net_ratelimit())
- WPRINTK("Too many frags\n");
- printf("%s: too many frags %d > max %d\n", __func__, frags,
-        max);
- err = E2BIG;
- }
  *cons += frags;
  *pages_flipped_p = pages_flipped;
- return err;
+ return (err);
 static void

If you need to use the shapping, you will get an error with the wizard like "You do not have 2 of local interfaces!".

To fix it (not tested yet) you need to add the xen network device in a /etc/inc/
Code: [Select]
function is_altq_capable($int) {
        /* Per:
                        pfSense_interface_mtu($realhwif, $mtu);
         * Only the following drivers have ALTQ support
        $capable = array("age", "alc", "ale", "an", "ath", "aue", "awi", "bce", "xn"

The wizard and interface will be ok, but you need to add support of ALTQ to xen xn device.

I don't have any patch yet but you can follow in the following man page :

Feel free to comment or add your own experiments !
Note : All the attachement have been suffixed with txt in order to be attached.

« Last Edit: June 30, 2011, 01:58:25 am by nexus »

Offline vsnine

  • Newbie
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: Xen PV domU - How to run pfSense in paravirtulised mode
« Reply #1 on: June 22, 2011, 11:29:02 pm »
How stable is this setup? Could you provide the disk image?

Offline cyruspy

  • Full Member
  • ***
  • Posts: 112
  • Karma: +0/-0
    • View Profile
Re: Xen PV domU - How to run pfSense in paravirtulised mode
« Reply #2 on: August 15, 2011, 09:18:02 am »
I would really love Xen domU support on PFSense. I've read that 2.0 final won't upgrade to 8.2, so having an official build won't be possible. I'll test this and provide some feedback when possible..

Offline lvvloten

  • Newbie
  • *
  • Posts: 9
  • Karma: +1/-0
    • View Profile
Re: Xen PV domU - How to run pfSense in paravirtulised mode
« Reply #3 on: August 15, 2011, 10:38:02 am »

Thanks for this very useful post! I am a total newbie in the field of pfSense and/or freeBSD, but your post really helped me to get started.

I want to implement pfSense in a domU on my home server, which has a simple Pentium IV processor without virtualization support. I overcame quite a few problems but now I seem to be stuck. I will start with give a short overview of some of the problems I solved, once I manage (if I manage...) to complete the procedure I will post my solutions in more detail.
Disclaimer: I hardly know what I am doing. My solutions need to be reviewed by someone a bit more exprienced in this matter before it can be called 'reliable'!

What did I encounter?
  • Even with the modified patches that you provided, I encountered quite a few patches that didn't work. I analyzed and modified those patches by hand and I managed to make all patches work without modifying any of the code itself. See attachments.
  • One patch turned out to lead to a compile error, I modified the code so it would work. (This might need some review from someone who, unlike me, would actually know what he is doing)
    Specifically: in if_rum_pr_144642.diff I changed "return 0;" to "return;". This solved the compile error.
  • Three pfPorts  did not build due to compile errors. I solved two. So far I did not yet manage to install php52-pfSense-module; however, this does not seem to be related to the issue below.

One important addition to the procedure above is that in my pfSense_wrap.8.i386 file (see attachment), I explicitly excluded everything that I could encounter that might be related to USB, SCSI or wireless support. To achieve this I used the 'nodevice' and 'nooptions'. My purpose is that no USB (or SCSI or wireless) related materials will be built - not directly into the kernel, and not as modules. However, as can be seen in the problems I encounter, this does not work as I expected.

So where does it go wrong?
I start the procedure using the /usr/home/pfsense/toold/builder_scripts/ script. I get the same result when I start the procedure from the menu, as described above.

Everything seems to run fine up to the time that it starts buidling the kernel. At "stage 3.1: making dependencies", the following error occurs:

Code: [Select]
/usr/pfSensesrc/src/sys/dev/usb/net/if_udav.c:72:21: error: usbdevs.h: No such file or directory
I have attached the full output of the command; normal (build_nano_output-normal.txt) and after uncommenting 'set -x' in the script (build_nano_output-debug.txt).

I manage to pass this stage by manually making some changes:
  • Modify if_udav.c line 72: change 'include "usbdevs.h"' to 'include <dev/usb/usbdevs.h>'
  • Manually run the following command:
Code: [Select]
cd /usr/pfSensesrc/src/sys/dev/usb && /usr/bin/awk -f /usr/pfSensesrc/src/sys/tools/usbdevs2h.awk usbdevs -h
    Although I don't have a clue what I am doing here I manage to get past the above error, only to end up in some other compile error in a USB related program.

    So my question is:
    How can I avoid that any USB related software will be built in the kernel build process? Clearly, excluding it in my pfSense_wrap.8.i386 configuration file is not the way to do it.

    Any help would be greatly appreciated!

    Kind regards,

    Offline CDuv

    • Jr. Member
    • **
    • Posts: 89
    • Karma: +1/-0
      • View Profile
    Re: Xen PV domU - How to run pfSense in paravirtulised mode
    « Reply #4 on: August 17, 2011, 08:10:59 pm »

    I'm also trying to put a pfSense VM into Xen (running on a Debian 6).

    By following the "DevelopersBootStrapAndDevIso" guide (on a VirtualBox-ed FreeBSD VM), I'm running into the following issue when running ./

    chmod: /usr/local/pfsense-fs/usr/local/lib/php/20060613/ioncube/ No such file or directory
    >>> Installation collected library information (usr/local), please wait...
    >>> chroot'ing and running /etc/rc.php_ini_setup
    >>> Copying config.xml from conf.default/ to cf/conf/
    WARNING: attempt to domain_add(netgraph) after domainfinalize()
    Loaded /boot/kernel/ng_socket.ko, id=2
    >>> Testing PHP installation in /usr/local/pfsense-fs:
    Fatal error: Call to undefined function pfSense_get_interface_addresses() in /etc/inc/ on line 49

    An error occured while testing the php installation in /usr/local/pfsense-fs

    Something went wront, check errors!

    NOTE: a lot of time you can run ./ to resolve.

    Press enter to continue.
    Clearly, the builder is lacking some PHP components but why?

    Offline nexus

    • Jr. Member
    • **
    • Posts: 39
    • Karma: +1/-0
      • View Profile
    Re: Xen PV domU - How to run pfSense in paravirtulised mode
    « Reply #5 on: August 18, 2011, 02:19:52 am »
    cd /home/pfsense/tools/pfPorts/php5/ && make all install clean
    cd /home/pfsense/tools/pfPorts/php5-pfSense-module/ && make all install clean

    Then run the build again it should fix.

    Offline cougarmaster

    • Jr. Member
    • **
    • Posts: 87
    • Karma: +0/-0
      • View Profile
    Re: Xen PV domU - How to run pfSense in paravirtulised mode
    « Reply #6 on: August 18, 2011, 11:31:26 am »
    Please make this post a sticky  ;D

    Offline nexus

    • Jr. Member
    • **
    • Posts: 39
    • Karma: +1/-0
      • View Profile
    Re: Xen PV domU - How to run pfSense in paravirtulised mode
    « Reply #7 on: August 18, 2011, 11:42:49 am »
    To get sticked I think there is some work to be done by other member. I fell alone with this topic so I give up for the moment (other things to do like my wedding for example).

    I think the best way is to use the appliance part of pfSense build tools, we also need a lot of patches from last kernel and xen devices (the xn network device for example that need some improvement if we need to manage VLAN inside pfSense or QOS with bandwidth tweaks).

    Feel free to try and Improve the receipe, if this topic is moving on, I will do my best to continue my work.