The pfSense Store

Author Topic: SSD (Solid State Drive) and pfSense (Important)  (Read 41333 times)

0 Members and 1 Guest are viewing this topic.

Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
SSD (Solid State Drive) and pfSense (Important)
« on: March 13, 2011, 03:25:03 pm »
This topic was designed to provide information but also to gather feedback from
other pfSense users operating on SSD drives.

If you were hoping to make pfSense ultra reliable by adding a SSD then this is a must read,
otherwise you could actually decrease the reliability more then your current drive.

SSDs are not simple upgrades for systems doing frequent disk writes. I wrote this
after my SSD upgrade planning so that you know what you're dealing with.

Having a pfSense firewall/router in a production environment that handles and routes data 24/7 made me rethink the hardware approach and to upgrade to prevent future problems.

The idea was to make the system run entirely solid-state with virtually no moving parts to fail (except fans and
the system I chose has redundant fans). Similar to Cisco routers and other embedded solutions, designing
an entirely solid-state system was as simple as replacing the mechanical hard drive with a solid state drive.

However, if you think it ends here there is actually much more to go and failure to do so could result
in a SSD drive life time much less than the typical mechanical drives. Without tuning, your SSD could last only
months. That's right, months.

Almost all common SSD drives today utilize NAND flash cells. NAND flash cells can only handle about 10,000
writes before they become unusable. For this reason SSD manufacturers use controllers that are able to distribute
data for even wear. Even if you really only need a few gigabytes of space make sure to upsize significantly on the
SSD so the drive controller has more memory area to spread across.

Some SSDs are better then others and use higher quality NAND flash components so it's a good idea to look around
for a good SSD and upsize considerably.

Now, no matter the SSD quality, there are several key factors on the pfSense system that you will have to address
to maximize the life of your SSD. This applies to all SSD drives of any manufacturer.

- Use higher amounts of RAM and disable the swap file (During install delete the /swap partition)
   The swap file can quickly wear out your SSD if not disabled. Remember, 10k writes per memory cell is it.

- Disable firewall rule logging
   The logging will also quickly wear out your SSD. If you require logging perhaps use a remote syslog server.

- RRD Graphing
    The RRD graphing backend runs and writes graph files to the drive (SSD) every minute. Disable RRD graphing backend.
            ** May be a good feature in pfSense to add the option to change the RRD /tmp directory from hard disk to ram disk for SSDs. Only down
            side is graphs will be lost on power down/reboot because the RAM is volatile.
 

- Upgrade to pfSense 2.0 for features such as Hard Drive S.M.A.R.T. Status
  pfSense 2.0 includes a very important tool if you're running a SSD and that is S.M.A.R.T. monitoring. With this feature you will be
  able to check the SSD's S.M.A.R.T. logs for I/O errors and drive problems. By monitoring this you can help see a failure coming.

- Disable ANY packages that could frequently write to the disk. If you are using a SSD drive for reliability it does require a sacrifice
 of many/most packages for pfSense. If you want a ultra reliable and fast pfSense system, it's crucial that you are willing to give up
 these extras.

Alternative to SSDs, you could use hardware RAID with regular hard drives if you require all of the packages, logging and extras. In my experience, you
want as little running as possible on a firewall. If you use a SSD, then you really have no choice. It's a must or you will have a failure. Time before failure
depends on the SSD quality, the SSD controller, the SSD size and frequency/size of disk writes. In some cases with continuous writes and high disk traffic
could damage the SSD in months.

This applies to all SSD drives no matter the brand, quality or features. Hybrid SSDs may work a bit differently because of the RAM buffer/cache but
you should still apply the same rules.

Keep pfSense simple and eliminate frequent drive writing and your system could last 10 years or more.

Question? Is this all absolutely necessary or is it just "optimization".
Answer: It is absolutely necessary, otherwise you're much better off using regular non-SSD drives in RAID. More then likely you chose SSDs for reliability so this is crucial. Placing SSDs in RAID is almost useless because the wear patterns would be mirrored and chances are they would both fail at the exact time assuming there are no manufacturing flaws. I simply cannot stress enough how dangerous it is to run SSDs with continuous writes. Your SSD will fail otherwise, only question is when.

Hope this helps everyone thinking about SSDs and please post all of your thoughts. I'm sure I missed other stuff.

** This does not affect NanoBSD/Embedded versions of pfSense and to resolve these issues installing the embedded version will solve all of these problems because it will run entirely from ram **

FIX: To solve the problem install the embedded version of pfSense that runs NanoBSD. This version was aimed at compact flash cards where limited writes are a factor. SSDs use the same technology so make sure to install the embedded version (NanoBSD) if you use a SSD. The embedded version loads the entire file system in to ram so that RRD, logs and any other write intentive tasks are handled in ram instead of on disk. Just prepared to lose your VGA console and you will need to use serial to configure. Default settings are 9600 baud 8-n-1. Thank you for the input/ideas everyone.

-- koukobin pointed out that there is a NanoBSD version that has VGA support. Thank you for the info!
« Last Edit: March 15, 2011, 09:51:53 am by FJSchrankJr »
-Fred

Offline ericab

  • Full Member
  • ***
  • Posts: 207
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #1 on: March 13, 2011, 04:33:18 pm »
thanks for this post.
thought id chime in myself and say, that i had gotten a kingston SSDNow 16 GB ssd, in hopes of having a fast and silent box.

all went well for about 2 and a half months until *wham*, the drive died.
got a new one from kingston since it was well within the manufacturers warranty.


Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #2 on: March 13, 2011, 04:56:30 pm »
thanks for this post.
thought id chime in myself and say, that i had gotten a kingston SSDNow 16 GB ssd, in hopes of having a fast and silent box.

all went well for about 2 and a half months until *wham*, the drive died.
got a new one from kingston since it was well within the manufacturers warranty.



Hi -- sorry this happened to you. The SSD I bought was a Kingston SSD S100 8 GB. I installed everything and got all of pfSense setup and was able to change every setting except RRD logging. You can disable most all writing except the RRD backend which writes all of those graphs every 1 minute. If we come up with a fix for that the SSD concept would be great. Only way would be to disable RRD and that's not a great option.
 

Hopefully your crash didn't take you down too long and everything turned out ok. I bailed last minute on the SSD idea until we figure it all out.

The manufacturers don't explain the dangers of the SSDs and they put the life as "1,000,000 hours" in their specs. Only problem is that must be either in off mode or read only. 10,000 writes per memory cell is not a lot at all and everyone should think twice before using the SSDs.

Now, the pfSense LiveCD loads the entire file system in to RAM which would be perfect. If they create a SSD version of pfSense that in essence is the LiveCD image but allows mounting a Read Only file system for normal operations and only mounts Read/Write for config changes then this system would be perfect for SSDs. I would rather lose my RRD data on reboots then lose the entire drive and take down all WAN traffic...

Very sorry that the system crashed, these SSDs sure are not what they seem like.

-Fred

Offline koukobin

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #3 on: March 13, 2011, 05:00:47 pm »
Do you need to do all these if you use nanobsd version?

I have been using compact flash drives two years now without a single problem.

In nanobsd version i thing even the logs are kept in RAM.

Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #4 on: March 13, 2011, 05:05:30 pm »
Do you need to do all these if you use nanobsd version?

I have been using compact flash drives two years now without a single problem.

In nanobsd version i thing even the logs are kept in RAM.
That's a very good question. I myself have never used the nanobsd version so I am not certain on that. Maybe that version does use ram entirely and only mounts the SSD/drive in read-only and read/write for config changes. If that's the case maybe I will go the nanobsd route. It's always a good idea to limit writes regardless but that sure would be better then the regular version on a SSD.
-Fred

Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #5 on: March 13, 2011, 05:08:49 pm »
Do you need to do all these if you use nanobsd version?

I have been using compact flash drives two years now without a single problem.

In nanobsd version i thing even the logs are kept in RAM.

Just checked on some things and the nanobsd version may not be a concern. It looks like you're right and the file system is loaded in to memory. Someone can hopefully confirm this for you but I think you are ok.
-Fred

Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #6 on: March 13, 2011, 05:13:23 pm »
Just confirmed and the NanoBSD version is not effected.

Here is information regarding your question from the pfSense site:

"The embedded version is specifically tailored for use with any hardware using Compact Flash rather than a hard drive. CF cards can only handle a limited number of writes, so the embedded version runs read only from CF, with read/write file systems as RAM disks. "

I will update my posting to exclude nanobsd/embedded version.

This may also be the way for others to use the SSD and pfSense without issues. I may do this myself. Thank you for posting that it sparked some ideas.
-Fred

Offline stephenw10

  • Hero Member
  • *****
  • Posts: 8094
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #7 on: March 13, 2011, 05:41:21 pm »
If you are testing this with a Nano installation you should be aware that there is currently a bug in 2.0rc1 which means that the filesystem is left RW after boot. It should be RO.
However it does not cause any undue writes as everything is set to run from RAM as you say.
If you check the filesystem don't be alarmed to find it's still RW, as I was!  ;)

Steve

Edit: Bug listed here.
« Last Edit: March 13, 2011, 05:49:26 pm by stephenw10 »

Offline FJSchrankJr

  • Full Member
  • ***
  • Posts: 127
    • View Profile
    • SMCO L.L.C. - Embedded Systems Engineering
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #8 on: March 13, 2011, 10:11:39 pm »
If you are testing this with a Nano installation you should be aware that there is currently a bug in 2.0rc1 which means that the filesystem is left RW after boot. It should be RO.
However it does not cause any undue writes as everything is set to run from RAM as you say.
If you check the filesystem don't be alarmed to find it's still RW, as I was!  ;)

Steve

Edit: Bug listed here.

Hey Steve -- thanks for the heads up! I did install pfSense back to the SSD. I went the embedded route and I took my 8GB SSD drive and loaded the 4GB nano image (didn't really need the extra space). It booted and loaded in to ram and solved my problem. Everything including RRD, /tmp and logs all run from ram now. According to stats the disk writes are virtually nothing except for config file changes. Only downside to running on embedded is you lose the VGA/Keyboard console and have to go serial to configure but not really an issue. Who knows maybe going this route will be more reliable because VGA isnt being loaded.

This looks like the perfect solution for SSDs (the same setup as CF). pfSense really did take these issues in to account when they created the embedded/CF version.

 Thanks for the help everyone.
-Fred

Offline dreamslacker

  • Hero Member
  • *****
  • Posts: 808
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #9 on: March 13, 2011, 11:23:02 pm »
Just checked on some things and the nanobsd version may not be a concern. It looks like you're right and the file system is loaded in to memory. Someone can hopefully confirm this for you but I think you are ok.

The nanobsd version can be configured to write out the logs/ RRD data at specific intervals like 30 minutes or 1 hour etc.  This allows you to retain information across reboots and yet heavily reduce the amount of I/O to the SSD/ CF.

Next on the point of the SSD, most controllers are optimized for Windows (unfortunately).  They rely on the Trim command to do garbage collection.
However, the Indilinx based drives (OCZ Vertex, Corsair X128 etc) have hardware level garbage collection which isn't as efficient but beats having none under *nix.  This would be useful to improve general performance in the long run.

The Sandforce based drives offer very very impressive write amplification and compression algorithms. This would be good for wear levelling and long term reliability.  A write amplification of 0.5X means that The equivalent wear and tear in the long run is half the data being written.  This is obtained through clever compression tricks and since the RRD/ log data is easily compressible, this is very effective as well.
SSDs like the OCZ Vertex 2(e), Gskill Phoenix Pro use the the Sandforce controllers.

I did a short write-up on the OCZ Vertex 2E drive here:
http://vr-zone.com/articles/old-dog-new-tricks-ocz-vertex-2e-reviewed/10323.html

The first section lists out the very basic pros of using the Sandforce controller and how it helps reduce the amount of write penalty in terms of performance and wear and tear as well.
Also in the testing is the Indilinx based drive which will show the difference between the effectiveness of garbage collection algorithms between the 2 controllers and the performance differences as well.

Of note is the difference between the compressible and incompressible data testing.  If you're running embedded with mostly compressible data written (logs and such), the Sandforce is likely to be a better choice.
If you run Squid and most of the content being cached are large compressed EXE files or video files, then the Indilinx is likely to be your best bet.
« Last Edit: March 13, 2011, 11:29:53 pm by dreamslacker »

Offline koukobin

  • Newbie
  • *
  • Posts: 13
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #10 on: March 14, 2011, 12:45:30 am »
If you want to have VGA and console output at the same time, you can use Hacom nanobsd Pfsense images:

http://www.hacom.net/catalog/pub/pfsense


Offline lowemissions

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #11 on: March 14, 2011, 04:15:56 pm »
SSD?
pfsense system spec:
wyse 3455xl
Ram 256 SDRAM
HD: 4 gig SSD IDE DOM
NIC: 4 Ethernet (intel) dr0-WAN, dr1-LAN, dr2-OPT1, dr3-OPT2

I installed on a 4 gig DOM SSD by Live CD for about 10 days now. I saw FJSchrankJr post a warning about SSD. What is the best solution for me? Can I load the nanobsd image by the Gui to fix my problem? or I have to reinstall from clean install.

Thank

Offline joltman

  • Jr. Member
  • **
  • Posts: 51
    • Twitter
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #12 on: March 14, 2011, 04:24:22 pm »
The nanobsd version can be configured to write out the logs/ RRD data at specific intervals like 30 minutes or 1 hour etc.  This allows you to retain information across reboots and yet heavily reduce the amount of I/O to the SSD/ CF.

Can you explain this process (or link to the page)?  I just picked up a 4GB CF card to run my pfSense 2.0 install from and it'd be great to keep the graphs.  Thanks!

Jim
pfSense 2.1
Mobo:   SUPERMICRO MBD-X7SLA-H-O
Case:    SUPERMICRO CSE-503-200B
2GB RAM
160GB Internal HDD

Offline stephenw10

  • Hero Member
  • *****
  • Posts: 8094
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #13 on: March 14, 2011, 04:29:33 pm »
What is the best solution for me? Can I load the nanobsd image by the Gui to fix my problem? or I have to reinstall from clean install.

You will want to switch to either the embedded install, sellected when you install from CD, or a NanoBSD image. You will want to do it quickly before your DOM dies!  ;)

You can probably backup your config file and restore it to the fresh install. You can't switch the install type from the GUI.

Steve

Offline lowemissions

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: SSD (Solid State Drive) and pfSense (Important)
« Reply #14 on: March 14, 2011, 04:47:01 pm »
Thank Steve