To understand why ZFS has feature flags, its important to understand a little history of ZFS.
ZFS was first conceived by Sun Microsystems, with the primary goal of addressing the storage challenges of the time. It was officially unveiled in 2004 as part of Sun’s Solaris Operating System. From its inception, ZFS was revolutionary, offering features like tremendous capacity (hence the backronym ‘Zettabyte File System’), data integrity verification, automatic repair, easy administration, and the combination of the file system and logical volume manager functions. At the time Sun had determined that ASICs and resources (RAM for RAID controllers) being used for storage were getting expensive fast, and that using the entire system’s resources (CPU, RAM, etc.) would yield major cost savings, performance gains, and allow for improved functionality for large scale RAID arrays. After all, system RAM and CPU power far outstripped the abilities of hardware RAID controllers, and at a significant cost savings.
Sun released the original ZFS pool version number (v1) in 2005. As time went on, new features were added. Each new version incremented the version number, all the way up to v28. Each new release would bring anywhere from one to a few new features. Upgrading your zpool allowed you to use the new features immediately. Sun additionally made the ZFS code open-source under the CDDL license. This allowed the rest of the world to use ZFS.
When Sun was purchased by Oracle in 2010, Oracle immediately closed-source the ZFS project. This meant that any new releases by Oracle would not be available to the general public, except by purchasing from Oracle.
The open source community was at an impasse. How do you continue to develop ZFS open-source without conflicting with Oracle’s “new” ZFS implementations?
The decision was made to change the ZFS pool version to v5000 from v28, and as part of that change to allow what was called “feature flags”. The assumption was that Oracle would never get to v5000, hence there would never be a conflict. As of the time of this writing (Jan 2024), Oracle is only up to v51.
ZFS pool version v5000 was where the open source community (now called OpenZFS) departed from the now closed-source code that Oracle uses.
Enter feature flags…
Feature flags allowed new ZFS code to utilize new features once enabled. But there was a catch. Isn’t there always a catch?
When you use a version of ZFS that uses a new feature, you must enable/activate it. Once enabled, it cannot be disabled. So if you enabled something after upgrading TrueNAS, you’d be forced to stick with a version of TrueNAS that knows how to utilize that feature. This meant whatever your current version of TrueNAS was, or newer.
Generally, when a new RELEASE version of TrueNAS comes out, it comes with some new features that can be enabled from the WebGUI or CLI. Upgrading from any TrueNAS 12.0 release to any TrueNAS 13.0 release allows you to enable a new feature- draid support. While not used for most people, you do get warnings in the WebGUI that there is an upgrade available for ZFS.
From the command line, if you run the command zpool status, you will see the following:
If you want to look at the new features that are available, but not enabled/active, you can use the command below and get a list of features. The below is an example of a system upgraded from 12.0 to 13.0 and the ZFS upgrade has not been performed yet.
zpool get all <poolname> | grep feature
tank feature@async_destroy enabled local tank feature@empty_bpobj active local tank feature@lz4_compress active local tank feature@multi_vdev_crash_dump enabled local tank feature@spacemap_histogram active local tank feature@enabled_txg active local tank feature@hole_birth active local tank feature@extensible_dataset active local tank feature@embedded_data active local tank feature@bookmarks enabled local tank feature@filesystem_limits enabled local tank feature@large_blocks enabled local tank feature@large_dnode active local tank feature@sha512 enabled local tank feature@skein enabled local tank feature@edonr enabled local tank feature@userobj_accounting active local tank feature@encryption enabled local tank feature@project_quota active local tank feature@device_removal enabled local tank feature@obsolete_counts enabled local tank feature@zpool_checkpoint enabled local tank feature@spacemap_v2 active local tank feature@allocation_classes enabled local tank feature@resilver_defer enabled local tank feature@bookmark_v2 enabled local tank feature@redaction_bookmarks enabled local tank feature@redacted_datasets enabled local tank feature@bookmark_written enabled local tank feature@log_spacemap active local tank feature@livelist enabled local tank feature@device_rebuild enabled local tank feature@zstd_compress enabled local tank feature@draid disabled local
Any that don’t say active or enabled will be enabled by upgrading from the shell or WebGUI.
So when should you upgrade?
In the example above, I mentioned the upgrade from 12.0 to 13.0. Changing from 12.0 to 13.0 is a major upgrade from a software standpoint. There is always the possibility of performance bugs or even reliability bugs. For this reason, when you first upgrade to 13.0 it is prudent to consider the value of the new features. Some may be pointless to you and some may be significant.
Generally, I recommend waiting a month or two. If you upgrade the zpool right after installing 13.0, then realize that 13.0 has some serious problem and you want to roll back to your prior boot environment you will find you cannot mount your zpool in 12.0. You are now forever locked into using a 13.0 release (or newer if one exists). For this reason you should always be cautious and exercise restraint before upgrading your zpool.
Upgrading your zpool literally takes just a couple of seconds, and can be done in production with no consequence to your production workloads.
Upgrading from the WebGUI is very straightforward, you can easily see it from the “Upgrade Pool” option as shown below.
From the shell, if you run the following command then the zpool will be upgraded and all features will be enabled:
# zpool upgrade <poolname>
That’s all there is to it.
So what’s the new feature in the upcoming TrueNAS Core 13.1 release?
zilsaxattr, head_errorlog, blake3, block_cloning and vdev_zaps_v2
If none of those just made sense to you, don’t worry about it. I’ll publish a blog post about them after TrueNAS Core 13.1 is officially released.
