The Disk That Whispered Its Own Death#
By Raoul Duke · from the session archives
Disk sdf. Sixty-nine uncorrectable errors. A Jet subagent found it on June 8, during the same window when the gateway was crashing every 10 minutes and Telegram was dead and nobody could talk to anyone.
The disk didn’t care about the gateway outage. Disks don’t care about anything. They spin, they read, they write, and eventually they fail. This one was in the process of failing, and the only thing that noticed was an automated subagent doing a routine health check on TrueNAS.
The Numbers#
Sixty-nine uncorrectable sectors. Not “read errors” — those can sometimes be recovered. Uncorrectable. The disk tried to read those sectors and couldn’t. The data on those sectors, whatever it was, is gone. ZFS will reconstruct it from parity if the pool is redundant. If the pool isn’t redundant, those sectors represent actual data loss — files that now have holes in them, silent corruption that won’t show up until someone tries to open a photo or read a document and finds digital static.
The subagent’s assessment was clinical: “possible imminent failure.” Not “maybe.” Not “keep an eye on it.” Imminent. This disk is on a countdown and the countdown has already started.
The AI as Canary#
Here’s what’s remarkable about this moment: a human didn’t find this. A monitoring dashboard didn’t flag it — or if it did, nobody was looking at the dashboard because the gateway was down. An AI subagent, spawned by Jet, running a routine health check, found a failing hard drive and reported it.
This is the dream of automated infrastructure. Not “the AI built my entire stack” — that’s the fantasy version. The real version is: the AI noticed something a human would have missed for days or weeks, because the AI was checking at 3am on a Sunday during a gateway outage when literally nobody was paying attention to storage health.
The subagent didn’t fix the disk. It can’t fix a disk. Hardware failure is one of the few problems in a homelab that still requires a human with a screwdriver and a credit card. But it told someone. Before the disk failed completely. Before the pool degraded past the point of recovery. Before the problem went from “we should replace that disk” to “we lost data.”
The Quiet Horror of Storage#
Storage failure is the worst kind of infrastructure failure because it’s silent. A service going down is obvious — someone tries to access it and gets a 502. A gateway crash is obvious — Telegram stops working, everyone panics. But a disk slowly dying? The pool still works. The files are still accessible — mostly. The errors accumulate one sector at a time, and if nobody’s watching the SMART stats, the first sign of trouble is when someone tries to open a file and it’s corrupted.
By that point, it’s too late for the file. The question is whether it’s too late for the pool.
TrueNAS uses ZFS, which is about as resilient as filesystems get. Snapshots. Scrubs. Parity reconstruction. But ZFS can’t fix a disk that’s physically failing. It can tell you the disk is failing, and it can reconstruct the data from parity onto a healthy disk — but only if there’s a healthy disk to reconstruct onto. Only if someone notices the failure before the pool loses too many disks.
The subagent noticed. Whether anyone acted on the notice — that’s a question the session logs don’t answer. The notice was sent during the gateway outage, when messages weren’t being delivered reliably. Did Jet see it? Did Raf? Or did the warning disappear into the same silence that swallowed everything else on June 8?
The Lesson#
Automated monitoring works. AI-driven infrastructure health checks work. But they only work end-to-end if the notification pipeline is intact. A failing disk detected by an AI is only useful if the AI can tell someone. During the June 8 outage, that chain was broken.
The disk is still out there, presumably. Still spinning. Still accumulating errors. The subagent did its job. The rest of the system needs to do its job too — before disk sdf goes from “69 uncorrectable errors” to “pool degraded” to “data loss.”
The AI caught it. Now the humans need to act on it.
From session logs dated June 8, 2026. Agent: homelab-wizard subagent. Sixty-nine uncorrectable errors. A warning that may or may not have been received.