The Corruption Came Back on Different Drives — the Cause Was TRIM, Not the SSDs
In my previous post, I documented how a btrfs RAID1 pool on two healthy-looking Samsung PM9A3 NVMe drives silently corrupted a Windows VM two weeks earlier — reads returning all-zero blocks that no single bad drive could explain. That investigation ended with an honest shrug: the drives tested clean afterwards, leaving the shared M.2 riser as the strongest suspect.
Then it happened again. On different drives, on a different bus. The exact same checksum fingerprint. This time I found the root cause — and it wasn’t the SSDs.
Why this matters
If you run Unraid (or any Linux box) with a btrfs RAID1 pool on SATA SSDs and auto-TRIM enabled, this post is a heads-up: queued TRIM was silently destroying both mirrors of my pool, in the same places, simultaneously. RAID redundancy gives you zero protection against it. And SMART — which flagged a drive as failing through all of this — had nothing to do with the actual corruption.
The setup
- unRaid 7.x on an Intel i9-13900K.
- Pool
ssd: two Seagate IronWolf 110 960 GB SATA SSDs (ZA960NM10001, serialsHKR02TFK+HKR02L9P) in btrfs RAID1, mounted at/mnt/ssd, holding Docker’s data root (/var/lib/docker) and the Windows 11 VM’s virtual disk (domains/Win11Enterprise/vdisk1.img). - The VM’s vdisk is not
NOCOW, so btrfs validates every read against a checksum — the same setting that caught the previous incident.
Trivia: one of these drives (sdd) is the very drive from “I Was About to
RMA This SSD — the Fault Was a SATA Cable”
— after the cable swap it formatted clean and joined this pool.
Step 1 — a SMART “failure” that wasn’t
At 02:53 the Unraid notification bell went off:
Unraid Ssd disk SMART health [1] — Warning [UNRAID] -
raw read error rate (failing now) is 19665 (sdd = ZA960NM10001_HKR02TFK)“Failing now” is the drive declaring the attribute worse than its
threshold — the strongest wording SMART has. But the raw value is a red
herring on these Seagates: on IronWolf 110s, attribute 1’s raw field is a
vendor-encoded composite (errors in the high half, an operation counter in the
low half). 19665 = 0x00004CD1 → high half = 0 errors. The normalized
value is what tripped: 093 with a worst of 088, below the 090
threshold, so the drive self-reports FAILING_NOW / IN_THE_PAST on its
SMART return status. The twin (sdb) showed the same shape with a raw of
only 341.
Every real failure counter was zero: reallocated 0, grown bad block 0, program/erase fails 0, uncorrectable ECC 0, UDMA CRC 0, SSD life left 99 %, SMART overall PASSED, and a just-completed extended self-test was clean. This is a documented IronWolf 110 behaviour — the same “Failing” flag at 100 % lifespan hit Synology users years ago, and it’s why the community treats that model’s attribute-1 trip as firmware noise unless the real counters move.
So: false alarm on the drive, but the alarm made me go look — and that’s when I found the real damage.
Step 2 — the checksum that had no right to exist
btrfs device stats /mnt/ssd:
[/dev/sdd1].corruption_errs 27 [/dev/sdb1].corruption_errs 31All of it confined to one file: ino 261 = the Windows VM’s vdisk.
dmesg showed the same signature I’d spent a week staring at in the NVMe
incident:
btrfs: checksum verify failed on logical 1505629372416 mirror 1 wanted <expected> found 0x8941f9980x8941f998 is the CRC32C of 4 KiB of zeros — and it sat on the found
side of every failure, which is the part that matters. wanted is the checksum
stored in the csum tree (what the file is supposed to contain); found is the
checksum the kernel computed from what the disk actually handed back. The pool
was reading back empty blocks — and crucially, on both mirrors at the same
logical offsets. (On byte order: the canonical CRC32C of 4 KiB of zeros is
0x98f94189; the kernel prints the on-disk little-endian value, 0x8941f998.)
I verified it against my notes from the PM9A3 pool: identical value, identical
“deterministic zeros” behaviour, identical RAID1-defeating correlation.
That match explained half the mystery: any mechanism that zeroes out a 4 KiB page will produce 0x8941f998. But on the IronWolf pair, the drives sit on motherboard SATA ports — no M.2 riser in sight, and both had just passed fio --verify=crc32c. For both mirrors to return zeros at the exact same logical offset with zero link errors, something in the layer above the drives had to be instructing both members to erase the same blocks at once. There is exactly one mechanism in Linux that does that: discard / TRIM.
Step 3 — the root cause: queued TRIM on a drive Linux never blacklisted
Unraid’s pool config had auto-TRIM on, which mounts btrfs with
discard=async — the kernel batches frees and sends them to the drive. On
SATA SSDs that means queued TRIM (the DATA SET MANAGEMENT command
deferred as an NCQ command, SEND FPDMA QUEUED).
The IronWolf 110 is known to be unstable with queued TRIM. FreeBSD’s
kernel has a documented report — bug 264139,
“ata: NCQ_DSM_TRIM trim method for Seagate IronWolf 110 SATA SSD hangs
drives” — and the fix, merged in 2024, is blunt:
commit a6cef617660a:
“The Seagate IronWolf 110 SATA SSD drive has been reported to be unstable with
NCQ trim enabled.”
Meanwhile Linux’s own quirk list (drivers/ata/libata-core.c) — where the
kernel deliberately downgrades broken drives to non-queued or disabled TRIM —
contains Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870
and a few others, but no Seagate entry at all. My own dmesg proved the
mechanism exists and is selective:
ata6.00: Model 'Samsung SSD 840 PRO Series', rev 'DXM06B0Q', applying quirks: noncqtrim zeroaftertrim…applied to the other SSD in the box, while the two IronWolf 110s got no quirk and full queued TRIM.
Why does that corrupt a RAID1? Because btrfs RAID1 stores the same free-space map on both members — it discards the same logical extents on both drives in the same pass. A TRIM path that invalidates more (or the wrong) LBAs on this firmware zeroes the same live blocks on both mirrors at once. Your mirror becomes two copies of the same hole: the checksum fails on both, self-heal has nothing to copy from, and the data is gone.
The honest caveat: FreeBSD documents instability/hangs from the drive side, and “queued TRIM erased live data” is the mechanism I infer from that plus the fingerprint (zeros on both mirrors, SMART-clean, CRC error counter untouched). It fits every observation. The proof is the test — see the result below.
Step 4 — the fix (persistent, one variable at a time)
One change, applied twice — persistent config and live mount:
# /boot/config/pools/ssd.cfg (backup: /root/ssd.cfg.bak.20260918-0329)
diskAutotrim="off" # was "on"
# apply live without stopping the array or Docker (36 containers stayed up)
mount -o remount,nodiscard /mnt/ssd
findmnt /mnt/ssd # → rw,noatime,ssd,space_cache=v2 (no more discard=async)Then a full read-write scrub to find out what was actually damaged:
scrub started 03:29:53, finished 03:53:07 (23 min)
csum_errors: 18 corrected_errors: 16 uncorrectable_errors: 2
read_errors: 0 verify_errors: 0 super_errors: 0read/verify/super = 0 is the second half of the fingerprint: the hardware
reads fine — nothing is failing, the stored bytes just don’t match their
checksums. Of the 18 bad blocks, 16 were bad on one mirror and were healed
from the clean copy. Two uncorrectable entries were one single 4 KiB block
that was bad on both mirrors — permanent loss, ~45.2 GiB into the VM’s
vdisk (logical 1505629372416, file offset 48560156672). I zero-filled
that one block so the VM’s reads at least don’t error; the original bytes are
unrecoverable either way.
And a wrinkle worth knowing: btrfs device stats counters did not move
during the scrub even though it found 18 errors — scrub-detected errors and
read-detected errors are tracked separately. You cannot watch only device stats; you have to watch dmesg + scrub output too (my daily watchdog cron now checks and alerts on all three).
What I’d do differently
- Never enable auto-TRIM on a btrfs pool without checking the drive’s
queued-TRIM record. FreeBSD keeps a “don’t TRIM this drive” list for a
reason. Linux has the same mechanism in
libata-core.c— an afternoon of grep + admesgcheck after the first boot beats a restored-from-backup VM. - Keep TRIM on the menu, just not queued. If you want fstrim on SATA
SSDs, force non-queued TRIM (
libata.force=…:noncqtrim) and verify with a scrub after the first trim pass.discard=syncdoes not help — it still uses the queued path on drives that advertise it. - Watch all three corruption signals, not one:
btrfs device stats,dmesg | grep 'csum failed', and periodic scrubs. Each can move independently (see above). - Corroborate before joining pairs. Same-batch, same-firmware drives are a correlated-failure risk; a pair sharing one firmware bug is the correlated-failure worst case. The mirror didn’t save me either time.
- When buying enterprise SATA SSDs: the drives on Linux’s quirk list (Micron M500/M550/1100, Crucial M500/M550/MX100, Samsung 840/850/860/870) and anything with a documented NCQ-TRIM bug (IronWolf 110) are out. Samsung PM893/PM897, Micron 5400 PRO/MAX, Solidigm D3-S4610/S4620, WD Ultrastar DC SA620/SA630, and Kingston DC600M all carry PLP plus no Linux-quirk entries, and vendors that publish firmware release notes (Micron, Solidigm, WD, Kingston) can actually ship you a fix. NVMe sidesteps the SATA queued-TRIM class entirely — my PM9A3s aren’t in the NVMe quirk table either.
The result
One variable changed (auto-TRIM off), everything else identical. Immediately after:
- scrub #2 (04:10–04:39):
corrected_errors: 0— the 16 heal-blocks stayed healed, nothing new appeared during the full re-read. corruption_errsflat at sdd1=27 / sdb1=31 for hours of normal Docker + VM-less operation (counters had been climbing daily before).- No new
csum failedlines in dmesg.
The confirmation window then ran for the following weeks: Win11 VM back on (it
produces the write pattern that reproduced this), daily checks against a
baseline file, and a re-scrub on 2026-09-22. That re-scrub came back the same
way the first one did — corrected_errors: 0, no new csum failed lines, and
corruption_errs still parked at sdd1=27 / sdb1=31. Had corruption reappeared
with TRIM off, the next suspect wouldn’t have been the drives — it would have
been the RAM (no ECC on this box), and memtest the test. It stayed quiet, which
makes the IronWolf 110 queued-TRIM firmware bug the cause of both this incident
and — I’d bet — a good share of the “healthy drives corrupting my RAID” reports
the forums keep collecting.
A “failing” SMART warning, a drive with zero errors, and corruption on both mirrors of a fresh pool — every sign pointed somewhere else, and the actual culprit was a storage feature most of us enable without a second thought.
Part of my storage-failure series: the NVMe incident and the SATA cable wake-up call.
Worried your storage pool might be failing silently?
I design, audit, and troubleshoot self-hosted storage and virtualisation stacks — from Unraid and Synology deployments to btrfs/ZFS RAID pools and kernel-level corruption monitoring. If you’re hitting unexplained checksum errors, VM crashes, or SSD firmware quirks, I can isolate the root cause and put automated scrub and telemetry guardrails in place.
Reach me at [email protected] or WhatsApp +60 12-797 2969, or see what I do at hoelee.com.
Dealing with flaky containers, cloud cost bloat, or unexplained outages?
Real production reliability comes from understanding the kernel and the protocol layers, not just restarting containers. I manage 162 containers across 3 hosts running 24/7 client workloads.
