pfSense English Support > Installation and Upgrades

HOW TO: 2.4.0 ZFS Install, RAM Disk, Hot Spare, Snapshot, Resilver Root Drive

(1/8) > >>

EDIT: I'm glad to see this was made a sticky! I hope it is helpful, I'll keep this original post updated so that relevant info stays at the top and won't get lost in whatever discussion may come here in the future.

pfSense 2.4 is coming and with it comes ZFS, for some this feature is a non-event but others will find it very useful.

in pfSense, ZFS is good for:

* Generally rugged filesystem (the forum has posts about users having to reinstall due to UFS errors due to hard shutdown or unknown reasons, ZFS avoids greatly mitigates this)
* High availability systems (using software raid provides redundancy in the event of drive failure)
* Remote systems (for same reasons as above, allows you to "fix" your system remotely if a drive fails without having to physically access it
* Cheap install media (installing to USB flash is very cheap, but they are write sensitive, ZFS helps mitigate that issue) EDIT: While I've never encountered them there can be issues with USB
* Full System Backup (snapshots, not as useful on pfSense due to ease of restoring config.xml, but still has it's uses)
* System replication (if setting up many identical systems can set one up and quickly distribute install + config to other systems - snapshot)
in pfSense, ZFS is bad for:

* RAM limited systems (it uses a lot of RAM, general rule of thumb would be 1GB RAM available for ZFS only, but it's not a hard rule)
I'm not an expert of any kind in the IT or networking world, I'm a hobbyist so keep that in mind reading through this.
None of this is my original idea, it comes from various places throughout the internet.
This is probably most commonly useful for someone wishing to install to cheap media (that's why I'm doing it.)

Why flash drives? They are cheap, really cheap. I got 5x8GB for $30.
Why does ZFS help? Flash drives (the cheap ones) are write sensitive, write too many times to them and they break. Operating Systems write a lot, so they can destroy a flash drive really fast. ZFS alone doesn't stop this (RAM disks mitigate it greatly) ZFS just gives you software raid. This way if one disk fails, another is there to keep things running seamlessly.

So why raid and a hot spare? All of your disks are being written to equally in the raid pool, so if one drive fails it isn't unreasonable to think that the others will follow suit relatively soon. With a hot spare already partitioned with boot code installed, you can introduce a fresh disk with little to no writes on it to the pool.
Why bother with flash drives at all if they are so fragile? Looking through FreeNAS forums (another FreeBSD based appliance that actually recommends USB flash installs) you can see many scenarios where people have gotten years of use out of single $5 flash drives as boot drives.

This won't be complex, this guide is for non-IT people like myself. It will cover:

* The general idea of what's happening in the pfSense 2.4 ZFS Auto Install
* A few zpool and zfs settings
* Partitioning, installing bootcode to and adding a hot spare to your zpool
* Recovering from a boot drive failure
* Snapshot basics

1. - The General Idea of What's Happening in the pfSense 2.4 ZFS Auto Install -
 The auto install feature is quick and straightforward but I'll mention a few things to save you some googling.
 Pool Type: Just read the descriptions, they tell you what's going on. If you still have questions ask here. I'm using raidz2 (not because I think I need it).
 I'm betting a 2 disk mirror is best for most people unless you know you have a reason for more.
 When you install I recommend inserting one disk at a time and selecting disk info, write that serial number somewhere and assign it a number and letter, then write that number or letter on the physical disk.
 i.e., disk 1, S/N: 1234567890, (write a "1" on the disk) pull disk one out, put disk two in, select Rescan disks, then Disk info, disk 2, S/N: 3126450789, (write a "2" on the disk).
 When a disk fails, zfs will give you the serial number of the failed disk, this just makes it easy to identify and replace the bad disk.
 name the pool whatever you like, probably something easy to type out though
 Leave 4k sector alignment unless you have a reason not to
 Encrypt if you need to, I don't
 For flash drives I would recommend turning off swap (just select 0 or nothing), I think you lose crash boot dumps though so if you need those leave it, but swap=on means more writes to your flash drive.
 Once your install is complete, for flash drives I recommend going to System > Advanced > Miscellaneous and turning RAM Disk on. If you already are using pfSense you can get a ballpark of your space needs
 for /tmp & /var with:
--- Code: --- du -hs /tmp
 du -hs /var
--- End code ---

EDIT: When performing a ZFS install to Flash media on 2.4.0 BETA, if you encounter issues booting (I have not) try adjusting the boot delay in /boot/loader.conf or /boot/loader.conf.local as follows.

--- Code:"10000"

--- End code ---

2. - A Few Zpool and ZFS Settings -

You can see your zfs settings per pool with:

--- Code: ---zfs get all yourpoolname
--- End code ---

If you install to a single disk, you can make zfs write two copies of everything to your drive. On flash this is probably a bad idea. The benefit is that if one copy of something you need gets corrupted, it's unlikely that the other will also
be corrupted so ZFS will likely recover from this corruption seamlessly.

--- Code: ---zfs set copies=2 yourpoolname
--- End code ---

You can see your zpool settings & stats with:

--- Code: ---zpool get all yourpoolname
--- End code ---

The only thing I'll mention here is setting autoreplace=on, it's saying that if the pool is degraded and you have a hot spare, resilver to the hot spare without asking you.
I do not recommend turning this on (it's off by default) unless you have set everything up (it doesn't just work on its own) and need it. But we'll talk about it later so I mention it.

--- Code: ---zpool set autoreplace=on yourpoolname
--- End code ---

ZFS can checksum your data to make sure nothing it corrupted, if it finds something corrupted AND has a redundant copy of that data, it will fix the corruption.
This is called running a scrub.

--- Code: ---zpool scrub yourpoolname
--- End code ---
Once the scrub is complete you can check the pools status and it will tell you if it repaired any errors. Scrubs DO write to your drives even if they don't repair any errors.
You can see for yourself by starting a scrub and running

--- Code: ---iostat -x -w 1
--- End code ---
You will see writes occurring intermittently throughout the scrub, and ending when the scrub is complete (if you have RAM disk enabled in pfSense and swap=off).
Because of this I only scrub monthly via cron.

3. - Partitioning, Installing Bootcode To And Adding A Hot Spare To Your zpool -

So you've installed ZFS to pfsense and want a hot spare. Since your zpool is a boot pool (ZFS on /) you need to partition and set up your hot spare accordingly.
NOTE: You can't use a hot spare that is smaller than the drives in your pool.
NOTE: Don't resilver to a hot spare unless you read the next section as well.

Take a look at your pool

--- Code: ---zpool status
--- End code ---
You'll notice that your pool is only using the second partition of each disk. We'll do the same. In order to set the sizes equal to those in the pool run:

--- Code: ---gpart show
--- End code ---
If you installed to a 2 way mirror you'll see something like this (values will be different, adjust accordingly).

--- Code: ---=>      40   8388528  da0  GPT  (4.0G)
        40      1024    1  freebsd-boot  (512K)
      1064       984       - free -  (492K)
      2048   8384512    2  freebsd-zfs  (4.0G)
   8386560      2008       - free -  (1.0M)

=>      40   8388528  da1  GPT  (4.0G)
        40      1024    1  freebsd-boot  (512K)
      1064       984       - free -  (492K)
      2048   8384512    2  freebsd-zfs  (4.0G)
   8386560      2008       - free -  (1.0M)
--- End code ---

  So to create a hot spare for this:

--- Code: ---# gpart create -s gpt da2
# gpart add -a 4k -s 512k -t freebsd-boot -l gptboot2 da2 ###This creates p1, you are using 4k alignment, size is 512k, type is freebsd-boot, label is gptboot2, you are partitioning drive da2
# gpart add -b 2048 -s 8384512 -t freebsd-zfs -l zfs2 da2 ###This creates p2, you are beginning at block 2048 and stopping at block 8384512
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2 ###This writes the bootcode to p1 of your hot spare

--- End code ---

You will have to adjust these commands to match your system. If you did it properly then all drives will appear identical in the output of gpart show.

Now simply add p2 of your hot spare to the pool:

--- Code: ---zpool add yourpoolname spare da2p2
--- End code ---

Thanks to @kpa for guiding me through this!

4. - Recovering From A Boot Drive Failure -

If a boot drive fails, your pool will show as Degraded. If your pool sustains more drive failures than your type of pool is capable of sustaining, it breaks forever and you have to reinstall.
A degraded pool will reboot just fine.
If your degraded pool is a boot pool (on pfSense with an auto install, it is) and you resilver that pool to a hot spare, it will work until you reboot. When you reboot it will hang on reboot.
To avoid hanging on reboot after resilvering your boot pool, you must remove the bad drive from the pool.

--- Code: ---zpool detach yourpoolname baddiskname
--- End code ---
This makes the hot spare a permanent part of your pool.

If you are smarter than me I'm betting you could automate this with a script, I would think something running frequently in cron along the lines of:

--- Code: ---check if pool is degraded
if no, exit
if yes, check if resilver complete
if no, exit
if yes, detach baddisk

--- End code ---

If anyone does write such a script, please share! ;)

In addition to writing a script to automate detaching the bad disk after resilver, you would need to automate resilvering. ZFS can handle that alone.

Turn autoreplace=on (covered earlier)
AND start zfsd (ZFS File Management Daemon) on boot, without zfsd running, autocomplete=on won't do anything.
Install pfSense package "Shellcmd"
Add a new command:
Command: zfsd
Type: shellcmd

Again, only do this if you figure out a script automating detaching the disk, your system will fail to boot after resilvering to a hot spare until the bad disk is detached from the pool.

To do all of this manually, when you get a degraded pool:

--- Code: ---zpool replace yourpoolname baddisk hotsparedisk

--- End code ---

 After the resilver is complete
--- Code: ---zpool detach yourpoolname baddiskname
--- End code ---

 Alternatively, you can pull the bad disk physically, add a completely new disk (or the hot spare if you remove it form the pool) partition it as above, and just the zpool replace command will heal your pool

5. - Snapshot Basics -
 Snapshots reference a moment in time of your file system that is read only, you can recover completely or partially from that moment in time.
 When a snapshot is taken it takes up no space because it is the same as your current filesystem.
 As your filesystem changes the snapshot grows proportionately.
 Snapshots probably aren't all that critical for pfSense since the config.xml restore works so well, but it is still handy.
 Since snapshots take up space over time, that means more writes. So if you want to minimize writes you can store snapshots on a seperate drive (like another cheap USB)
 If you want it go to a separate drive you can add the drive, partition it as you like and (if it is also ZFS) create a new pool for it using:
--- Code: ---zpool create yournewpoolname yournewdiskname
--- End code ---

--- Code: ---zfs snapshot -r yourpoolname@snapshotname ###Creates a recursive snapshot of your entire pool
zfs send -Rv yourpoolname@snapshotname | zfs receive -vF yournewpoolname ###Sends that snapshot to your other drive
zfs destroy -r yourpoolname@snapshotname ###Recursively destroys the entire snapshot of your pool that is stored on the boot pool
zfs list -t snapshot ###Shows you a list of all your snapshots
--- End code ---

 EDIT: I don't recommend setting a second zpool as it can cause issues with booting. If you want to send snapshots on a separate device, try a UFS filesystem on it. People smarter than myself can probably get around this, if anyone has a solution please share and I'll add it here!
 To use UFS:
After partitioning the drive follow the instructions here:

To send your snapshot to a UFS partition you can modify this for your mount point and copy and paste:

--- Code: ---zfs snapshot -r yourpoolname@`date "+%d.%b.%y.%H00"` && zfs send -Rv yourpoolname@`date "+%d.%b.%y.%H00"` | gzip > /mnt/sshot/sshot`date "+%d.%b.%y.%H00."`gz && zfs destroy -r yourpoolname@`date "+%d.%b.%y.%H00"` && zfs list -r -t snapshot -o name,creation && du -hs /mnt/sshot/sshot`date "+%d.%b.%y.%H00."`gz

--- End code ---

You can write the following to a file in pfsense (NOT in /var or /tmp if using a RAM disk, and you must modify for your mount point), make it executable (chmod +x /usr/myshells/yourfilehere) and add it to a cron job to automate your snapshots on pfSense. Just run the cron job towards the beginning of the hour as the date time group variable changes on the hour and could cause issues if the hour changes before it completes.

--- Code: ---zfs snapshot -r yourpoolname@`date "+%d.%b.%y.%H00"` && zfs send -R yourpoolname@`date "+%d.%b.%y.%H00"` | gzip > /mnt/sshot/sshot`date "+%d.%b.%y.%H00."`gz && zfs destroy -r yourpoolname@`date "+%d.%b.%y.%H00"`

--- End code ---

Once the transfer is complete, you can compare file sizes of your actual pool and the backed up snapshot if you want, it should be much less than what the zfs send verbose output estimates after compression.

--- Code: ---du -hs /mnt/yourmountdirectory/yoursnapshotbackup.gz && zfs get used yourpoolname
--- End code ---

That's it for now, I hope this is helpful! I appreciate all comments and recommendations!

You can find more discussion on this topic here:

FWIW, I've had a stable pfsense raidz2 zpool install on cheap flash drives since 20 Feb 17. On 24/7, lots of system updates to latest 2.4.0 BETA (= abnormally high number of writes). Monthly scrubs, plus the occasional scrub after power outage, no errors or issues yet. I'll update here when this finally starts throwing errors, having issues or fails as a point of reference for others.


I'm planning to by one of these:

...and have been reading in advance a lot about the installation methods in order to determine my needs. But as I have only ever played with pfsense in virtual machine I'm confused. I'm just planning ahead to see what will be the best combination of media on which to install on.

This 2.4 guide suggests using USB keys, but if I have the option to use the internal mSata SSD would that make sense to do so?

If so, and lets say I plan to use a whole bunch of packages including Squid, Suricata, etc, what would a suitable size be? My understanding with ZFS is there would still be a benefit to using ZFS when installed on a single volume... right? I'm not sure I could configure a pair of SSDs on this device. I should add that my ISP speed is low (currently 10mb) but I am over-speccing this a lot for possible much faster speeds in future, and also in case I decide to repurpose the device as something else. I understand ZFS uses more RAM, will 8G be enough?

Alternatively would it be better to use a pair of USB keys for the installation? If so what would be a suitable size? Would the SSD then be unused, or would it still be useful for non-boot functions?

Sorry for all the questions but I have to order everything in advance internationally so just want to get the hardware right first time in terms of RAM, SSD, USB. Actual installation will be later. Thanks in advance....

I don't recommend USB Flash Drives on ZFS over SSDs unless you trying to save money and don't already have an SSD. I might recommend them over an HDD because they are silent and use less power, but the advantages over SSD are only price.

Using flash drives complicates things, so if you have an SSD definitely use that, and yes there are advantages of ZFS over UFS in a single drive configuration. In fact, single drive would be the recommended configuration for almost all use cases unless you are using USB flash drives.

ZFS does use more RAM than UFS but it's not a huge amount in a firewall implementation. 8GB is way more than enough as far as ZFS is concerned.

Thanks a lot - that was exactly the information I was looking for.


[0] Message Index

[#] Next page

Go to full version