Add host guide for checking SSD health - #938
Open
jacobmattern wants to merge 3 commits into
Open
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Hosts are required to run SSD storage with a dedicated Docker drive, but there was no guide for telling a worn drive from a failing one. A drive usually degrades before it dies: containers fail to start, the filesystem drops to read-only, and the machine deverifies while the drive still reports as healthy. Covers reading the NVMe health log and the SATA attribute table, running self-tests, checking what the kernel saw, and replacing a drive inside a maintenance window. Two things the page is deliberate about. The overall PASSED verdict is a low bar, reflecting one byte on NVMe and one boolean on SATA, so the page does not stop there. And on SATA the FAIL column only means something where a threshold exists: smartctl prints --- in THRESH when it cannot read one, and such an attribute never reports NOW or Past however bad its raw value gets, which covers reallocated sectors and uncorrectable errors on many drives. The page reads those raw values directly instead. Drives behind a RAID controller need an explicit -d pass-through type or smartctl reads the virtual disk and reports a healthy array over failing members. Array state itself is out of scope here. Adds the page to the Maintenance nav group and links it from the storage requirement in the verification stages.
jacobmattern
force-pushed
the
docs/host-ssd-health
branch
from
September 3, 2026 21:18
32a434a to
c1c19e2
Compare
The page described hosts in the third person in a few places, which reads as talking about the reader rather than to them. The clearest case was the RAID section opening with "Host machines often put their SSDs behind a hardware RAID controller", telling the reader something they already know about their own hardware. Rewrites those to second person and reframes the RAID detection step as confirming which controller is present rather than discovering whether one is.
Contributor
|
Correctness review only. Verified hands-on with disposable Docker containers (Ubuntu 22.04 + 24.04, including a privileged container against this host's real NVMe drive for read-only checks) and cross-checked with an independent second pass against the actual Verified correct
Issues found
None of these affect the mainline NVMe/SATA health-check workflow, which checked out fully accurate. They're all in the RAID pass-through / SAS edge-case material. |
A directly attached SAS drive auto-detects as scsi just like a controller's virtual disk, so that alone cannot tell an array from a drive. Areca SAS controllers take areca,N/E rather than the SATA syntax. The rule about addressing controller nodes rather than logical drives is specific to Areca and HP Smart Array, not MegaRAID or 3ware. -l selftest prints the standard self-test log, not the extended one, and -l sataphy,reset writes to the drive rather than only reading it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hosts are required to run SSD storage with a dedicated Docker drive, but there was no guide for telling a worn drive from a failing one. A drive usually degrades before it dies: containers fail to start, the filesystem drops to read-only, and the machine deverifies while the drive still reports as healthy.
Adds
host/ssd-health.mdxcovering the NVMe health log, the SATA attribute table, self-tests, what the kernel saw, and replacing a drive inside a maintenance window. Adds the page to the Maintenance nav group and links it from the storage requirement in the verification stages.Two things the page is deliberate about
PASSEDis a low bar. It reflects a single byte on NVMe (the Critical Warning field) and one boolean on SATA, so the page treats it as a starting point rather than a clean bill of health.On SATA, the
FAILcolumn only means something where a threshold exists. smartctl prints---inTHRESHwhen it cannot read one, and an attribute with no threshold never reportsNOWorPasthowever bad its raw value gets. On many drives that covers reallocated sectors and uncorrectable errors — the two attributes that matter most. The page reads those raw values directly and warns against scanning theFAILcolumn.Drives behind a RAID controller need an explicit
-dpass-through type, or smartctl reads the virtual disk and reports a healthy array over failing members. Array-level health (ZFS/mdadm/controller state) is deliberately out of scope and tracked separately.Verification
Every command and output block was verified rather than written from memory:
Version differences documented rather than papered over: 22.04 ships smartmontools 7.2 and 24.04 ships 7.4, NVMe self-tests need 7.4, and on 22.04
smartctl -tfails silently on NVMe — so the page routes those hosts tonvme-cliinstead.No output was invented. Where no hardware was available (SATA self-test results, SAS output, RAID pass-through), the page gives commands and description without a sample block.