fix(detector): fail closed on GPU health check errors

[thxCode]

Summary

On an 8x H100 node, a card whose GSP firmware stopped answering RPCs (Xid 119 timeout → Xid 154 GPU Reset Required) showed ERR!/N/A telemetry in nvidia-smi, yet gpustack-runtime detect still reported it healthy. The health check is opt-in (GPUSTACK_RUNTIME_DETECT_NO_HEALTH_CHECK defaults to true — unchanged), but even when enabled it was narrow and fail-open: the ECC query sat under a blanket contextlib.suppress, so the very errors a wedged card produces were swallowed and the card reported healthy.

The enabled health check now fails closed on every vendor carrying that pattern: a query the driver errors on reports the device unhealthy, while a query the hardware does not support keeps the previous verdict — no false positives on old drivers or ECC-less cards.

Changes

• NVIDIA — _get_memory_status catches NVMLError explicitly (NVML_ERROR_NOT_SUPPORTED tolerated, anything else unhealthy) and additionally probes NVML_FI_DEV_GET_GPU_RECOVERY_ACTION / NVML_FI_DEV_RESET_STATUS, so a GSP failure that leaves the ECC counters readable still reports unhealthy; an unreadable field falls back to the ECC verdict
• Iluvatar — extracted _get_memory_status from the two inline checks; ixmlDeviceGetHealth errors fail closed the same way
• THead — ECC counter query fails closed (HGML_ERROR_NOT_SUPPORTED tolerated)
• AMD — extracted _get_memory_status; the AMD SMI path decides per err_code (the codeless stub error falls back to ROCm SMI, as the memory read does); the ROCm SMI path tolerates RSMI_STATUS_NOT_SUPPORTED and a missing ECC symbol
• Docs — the env var docstring and the detector-alignment spec's F1 row record the coverage, and that a query against a wedged card can block until the driver's GSP RPC timeout (up to 45s)

Default behaviour (health check off) is unchanged: no extra driver calls per card per pass.

Verification

• uv run pytest: 634 passed, 20 skipped; make lint clean
• 18 new regression tests across the four fake driver bindings; the 8 fail-closed cases were verified red on the pre-fix tree
• Hardware confirmation (advisory): on a card in the Xid 154 state, GPUSTACK_RUNTIME_DETECT_NO_HEALTH_CHECK=false gpustack-runtime detect should report Status ERR

[gemini-code-assist]

Code Review

This pull request refactors and improves the device memory health check logic across multiple GPU backends (AMD, Iluvatar, NVIDIA, and THead) to implement a fail-closed policy where genuine driver query errors mark the device as unhealthy, while unsupported queries are tolerated. On NVIDIA, the check is enhanced to also probe the driver's recovery state (such as awaiting reset after a GSP failure). Comprehensive unit tests have been added to verify these behaviors under various failure scenarios. There are no review comments, so I have no feedback to provide.