All forms of data storage are subject to failure. Whether they’re clay tablets in an ancient civilisation, books in a library, or bits encoded in electronic media, sooner or later they’ll become corrupted or lost altogether. The only uncertainties are when, and what measures we need to take to postpone that event. In this context, how should we monitor the integrity of data stored in and for our Macs? Is APFS as a file system sufficient, or do we need something more?
This takes us back to a time when some inside Apple were keen to adopt the youthful ZFS. Despite enthusiasm and initial ports of the file system, the decision was made to develop Apple’s own proprietary file system APFS instead, and ever since those who have come to love ZFS have drawn attention to what they see as shortcomings, most notably in the lack of support for file data integrity checks in APFS.
APFS does have integrity checks built into the file system itself, in the form of Fletcher 64 checksums in the header of each file system object, but none that apply to file data, in the way that ZFS does, and checks those during data scrubbing. Doesn’t this make APFS prone to events such as bit rot?
Purposes
Before we go any further, it’s important to understand what ZFS and APFS were designed for.
ZFS was originally developed in 2001 by engineers at Sun Microsystems for its Solaris operating system, and throughout its life has been intended for use on server systems rather than personal computers or mobile devices. At the time of its introduction, storage systems were overwhelmingly hard disks, and it wasn’t until 2008 that the first MacBook Pro shipped with early and tiny solid-state drives, and the first iPhone with its flash storage had only shipped the previous year.
APFS was first announced just seven years ago, at WWDC in 2016, and shipped the following year in iOS 10.3 and macOS 10.13. By that time, Apple’s last dedicated server, the Xserve, had been discontinued for over five years, and Apple’s products included Macs, iPads, iPhones and the Apple Watch. While it clearly needed to support the Macs of the day and the future, APFS was intended to run on all Apple’s devices as well, and primarily to support solid-state drives rather than hard disks.
Although ZFS might have been a superior file system to APFS for advanced users of Macs, it would clearly have been a nonsense for Apple to have standardised all its products on ZFS. As it is, the presentation of APFS to device users simplifies them to give the user access to only one volume, although the underlying disk may contain other APFS volumes within the same container. This isn’t a reflection on the device user, but adapts presentation of the file system to what works best on that particular device.
Risks
My next question is whether Apple’s products need the same file integrity protection as that provided in ZFS. Although there’s a lamentable lack of information regarding the frequency of data corruption in general, and bit rot in particular, they appear to be exceptionally unusual in SSDs. Most accounts of bit rot refer to it resulting from exposure to cosmic rays; if that’s really one of its most likely causes, then unless you intend taking your Mac or device into space, I think you can safely rule out bit rot in everyday experience.
Unlike hard disks (most of which already incorporate internal systems for detecting data corruption), like most other solid-state devices, SSDs tend to work perfectly or not at all, neither of which is aided by checking data integrity at a file system level.
In case you’re not convinced, many of us look at dozens or even hundreds of JPEG images every day. As I demonstrated in experiments with deliberately corrupted images, the typical appearance of a damaged JPEG includes obvious visual artefact.
That’s the result of the corruption of just 1 byte per megabyte of image (here at JPEG 1:8 compression, baseline DCT, Huffman coding). Since switching to the exclusive use of SSDs four years ago, I’ve never encountered an image stored on an SSD with such evidence of damage.
Rationale
One important factor that’s often forgotten is why ZFS performs scrubs to check data integrity, when other file systems prefer to use a tool like fsck. This is explained in the ZFS documentation: as ZFS is intended for server file systems that run live all the time, performing periodic fscks on unmounted file systems would be severely disruptive. ZFS therefore doesn’t offer fsck, but the facility to check integrity in background scrubs of the live file system. The strategy adopted by APFS is better-suited to its uses: each time your Mac starts up, or mounts a volume, fsck performs a check on the volume. Few Macs are left running a live file system for longer than a couple of weeks, and many are started up each day.
We also forget that detecting file data corruption is only part of protecting the integrity of data: if that’s to be of any help, you then need redundancy in the file system, part of the design of a robust ZFS storage pool. As the ZFS documentation states:
“If your pool is not redundant, the risk that file system corruption can render some or all of your data inaccessible is always present.”
On Macs and devices, the remedy lies in backups, provided that they haven’t been corrupted as well.
Demand
While it might be nice to have the ability to check data integrity in APFS, all implementations come at a cost, in terms of additional storage overhead, complexity in the file system, and extra background tasks that on some Macs might take many hours or even days to complete.
It has been just over three years since I started offering my own free utilities, Dintch and its two siblings, here. Over that time, they have been downloaded less than 500 times, and are one of the least frequently downloaded of the free apps available here.
As nice as it would be for APFS to provide for optional data integrity checking at a file system level, at present there simply isn’t the demand to justify incorporating this as a new feature. As far as users are concerned, APFS doesn’t need data integrity checking, and in the absence of compelling evidence that it would serve a useful purpose, I have to agree. But I will continue to develop better tools for those who want them, and if you feel you have evidence of the benefits of their protection, please let me know.