Skip to content

Add host guide for checking SSD health - #938

Open
jacobmattern wants to merge 3 commits into
mainfrom
docs/host-ssd-health
Open

Add host guide for checking SSD health#938
jacobmattern wants to merge 3 commits into
mainfrom
docs/host-ssd-health

Conversation

@jacobmattern

Copy link
Copy Markdown
Contributor

Hosts are required to run SSD storage with a dedicated Docker drive, but there was no guide for telling a worn drive from a failing one. A drive usually degrades before it dies: containers fail to start, the filesystem drops to read-only, and the machine deverifies while the drive still reports as healthy.

Adds host/ssd-health.mdx covering the NVMe health log, the SATA attribute table, self-tests, what the kernel saw, and replacing a drive inside a maintenance window. Adds the page to the Maintenance nav group and links it from the storage requirement in the verification stages.

Two things the page is deliberate about

PASSED is a low bar. It reflects a single byte on NVMe (the Critical Warning field) and one boolean on SATA, so the page treats it as a starting point rather than a clean bill of health.

On SATA, the FAIL column only means something where a threshold exists. smartctl prints --- in THRESH when it cannot read one, and an attribute with no threshold never reports NOW or Past however bad its raw value gets. On many drives that covers reallocated sectors and uncorrectable errors — the two attributes that matter most. The page reads those raw values directly and warns against scanning the FAIL column.

Drives behind a RAID controller need an explicit -d pass-through type, or smartctl reads the virtual disk and reports a healthy array over failing members. Array-level health (ZFS/mdadm/controller state) is deliberately out of scope and tracked separately.

Verification

Every command and output block was verified rather than written from memory:

  • NVMe field semantics against the NVM Express Base Specification 2.2, Figure 207
  • smartctl behaviour against the shipped man pages for both 7.2 and 7.4
  • NVMe output captured from real Ubuntu 22.04 and 24.04 environments against a physical NVMe device, so the formatting matches what hosts see
  • SATA attribute tables and device statistics captured from real WD SSDs
  • Tool availability checked against the Ubuntu Server live ISO manifests for 24.04.4 and 22.04.5, since the requirement is Server rather than Desktop

Version differences documented rather than papered over: 22.04 ships smartmontools 7.2 and 24.04 ships 7.4, NVMe self-tests need 7.4, and on 22.04 smartctl -t fails silently on NVMe — so the page routes those hosts to nvme-cli instead.

No output was invented. Where no hardware was available (SATA self-test results, SAS output, RAID pass-through), the page gives commands and description without a sample block.

@mintlify

mintlify Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
vastai-80aa3a82 🟢 Ready View Preview Sep 3, 2026, 8:39 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

Hosts are required to run SSD storage with a dedicated Docker drive, but
there was no guide for telling a worn drive from a failing one. A drive
usually degrades before it dies: containers fail to start, the filesystem
drops to read-only, and the machine deverifies while the drive still
reports as healthy.

Covers reading the NVMe health log and the SATA attribute table, running
self-tests, checking what the kernel saw, and replacing a drive inside a
maintenance window.

Two things the page is deliberate about. The overall PASSED verdict is a
low bar, reflecting one byte on NVMe and one boolean on SATA, so the page
does not stop there. And on SATA the FAIL column only means something
where a threshold exists: smartctl prints --- in THRESH when it cannot
read one, and such an attribute never reports NOW or Past however bad its
raw value gets, which covers reallocated sectors and uncorrectable errors
on many drives. The page reads those raw values directly instead.

Drives behind a RAID controller need an explicit -d pass-through type or
smartctl reads the virtual disk and reports a healthy array over failing
members. Array state itself is out of scope here.

Adds the page to the Maintenance nav group and links it from the storage
requirement in the verification stages.
The page described hosts in the third person in a few places, which reads as
talking about the reader rather than to them. The clearest case was the RAID
section opening with "Host machines often put their SSDs behind a hardware
RAID controller", telling the reader something they already know about their
own hardware.

Rewrites those to second person and reframes the RAID detection step as
confirming which controller is present rather than discovering whether one is.
@hanran-y

hanran-y commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Correctness review only. Verified hands-on with disposable Docker containers (Ubuntu 22.04 + 24.04, including a privileged container against this host's real NVMe drive for read-only checks) and cross-checked with an independent second pass against the actual smartctl(8) man page. No style/readability comments here — that's a separate pass.

Verified correct

  • Package versions: 22.04 ships smartmontools 7.2-1ubuntu0.1, 24.04 ships 7.4-2build1 (exact match to the doc's dpkg -s sample).
  • The claimed 7.2 "silent no-op" on NVMe self-test: reproduced live — smartctl -t short /dev/nvme0 on 7.2 exits 0 with the "use -a instead" message, and its -x output has zero mention of self-test.
  • Real NVMe output: -H, -x, -d test, -l selftest against actual hardware matched the doc's sample blocks field-for-field (labels, spacing, structure).
  • -a/-x equivalence on NVMe (byte-identical output, confirmed).
  • Default /etc/smartd.conf, the smartmontools.service/smartd.service alias, and ConditionVirtualization=no — all match the shipped units on both releases exactly.
  • MegaRAID (0-127), cciss (0-15), aacraid, 3ware (0-127) device-type strings/ranges and example commands — all lifted verbatim from the real man page's own examples.
  • docs.json nav placement and every internal link (/host/machine-offline, /host/cli/schedule-maint, /host/how-to-self-test, /host/verification-stages) resolve correctly.

Issues found

  1. L155-158: The RAID-detection heuristic ("if -d test says scsi on an SSD, you're looking at the array") isn't reliable — a directly-attached SAS SSD also auto-detects as generic scsi. The doc's own SAS note further down confirms SAS drives report SCSI-format output, so this would false-positive for anyone with a plain SAS SSD and no RAID controller.
  2. L168, L172: The Areca row (areca,N, range 1-24) is only the syntax for Areca SATA RAID controllers. Areca SAS controllers use areca,N/E (N: 1-128, E: 1-8, enclosure) per the man page — not mentioned.
  3. L176-181: The "use RAID-controller nodes, not logical-drive nodes" warning is a real smartmontools quote, but per the man page it's specifically about Areca/cciss/hpsa/hpahcisr. The man page's own MegaRAID and 3ware examples use /dev/sda//dev/sdb//dev/twl0 — the same style this warning seems to rule out — which could make a reader second-guess the MegaRAID/3ware rows in the table two lines above.
  4. L462-464: Step 6 (SATA/SAS) has the reader run smartctl -l selftest /dev/sda (the legacy, non-extended self-test log) but says the result appears under SMART Extended Self-test Log. Per the man page, that heading belongs to -l xselftest/-x; plain -l selftest prints the older "SMART Self-test log" section instead.
  5. L420 (minor): The -l sataphy,reset tip is mentioned inside step 5, which the doc's intro classifies as fully read-only. Resetting the PHY error counters is a (harmless) write, so that classification isn't quite accurate if a reader follows the tip.

None of these affect the mainline NVMe/SATA health-check workflow, which checked out fully accurate. They're all in the RAID pass-through / SAS edge-case material.

A directly attached SAS drive auto-detects as scsi just like a controller's
virtual disk, so that alone cannot tell an array from a drive. Areca SAS
controllers take areca,N/E rather than the SATA syntax. The rule about
addressing controller nodes rather than logical drives is specific to Areca
and HP Smart Array, not MegaRAID or 3ware.

-l selftest prints the standard self-test log, not the extended one, and
-l sataphy,reset writes to the drive rather than only reading it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants