Use Linux software RAID? Schedule periodic RAID scrubbing or lose your data

This post is bought to you by the fun of unnecessary wasted time and work rebuilding a server after a double-disk RAID array failure. RAID scrubbing is essential - and is supported by Linux's software RAID, but not used without explicit user action.

Linux's `md' software RAID isn't unique in this, but as its use is so wide spread it's worth singling out. No matter what RAID card/driver/system you use, you need to do raid scrubbing. Sometimes RAID vendors call this a "patrol read", "verify" or "consistency check", but it's all generally the same thing.

Why you should scrub your RAID arrays regularly

I had a disk fail in the backup server (some time ago now). No hassle - replace it, trigger a rebuild, and off I go. Unfortunately, during the rebuild another disk was flagged as faulty, rendering the array useless as it had a half-rebuilt spare and a second failed drive.

You'd think the chances of this were pretty low, but the trouble is that the second failed drive will have developed just a couple of defective sectors (a SMART check confirms this) that weren't detected because those sectors weren't being read. Until the drive was sequentially read during the rebuild, that is.

To reduce the chance of this, you can periodically verify your arrays and if bad sectors are discovered, attempt to force them to be remapped (by rewriting them from redundant data) or failing that fail the drive. This will also detect any areas where different disks disagree on what the correct data is, helping you to catch corruption caused by failing hardware early.

Unfortunately, Linux's software RAID doesn't do this automatically.

A simple shell script like this, dropped in /etc/cron.weekly and flagged executable, will save you a LOT of hassle:

#!/bin/bash
for f in /sys/block/md? ; do 
    echo check > $f/md/sync_action
done

Make sure to TEST YOUR EMAIL NOTIFICATION from mdadm, too. If a drive fails and you never get notified, you're very likely to lose that data the moment anything else goes wrong.

Use S.M.A.R.T too

For extra protection from failure, install smartmontools and configure smartd to run regular "long" tests of all your RAID member disks, so you're more likely to discover failing disks early.

Unfortunately, many consumer-oriented disk firmwares lie to the host and try to cover up bad sectors and read errors - probably to reduce warranty costs. Manufacturer's disk tools tend to do the same thing. Some even seem to lie during S.M.A.R.T self-testing, re-allocating sectors as they find bad ones and then claiming that everything is fine. In fact, I've actually had a consumer SATA drive that can't even read sector 0 return PASSED when queried for a SMART general health check, though at least it failed an explicitly requested self-test.

My point is that SMART testing alone isn't sufficient to ensure your disks are trustworthy, you really need to use a redundant array with some kind of parity or data duplication and do proper RAID scrubbing. And, of course, good backups.

If you use "RAID friendly" disks (usually the same physical drive with honest firmware and a much bigger price tag) you shouldn't have as many issues with SMART self-tests.