fix(detector): stop AMD SMI's "N/A" sentinel from reaching numeric device fields

[thxCode]

What

gpustack-runtime detect --format json on an AMD Instinct MI300X VF emitted "power_used": "N/A", which the consumer cannot parse:

gpustack.worker.collector - ERROR - Failed to detect GPU devices:
1 validation error for GPUDeviceStatus
power_used
  Input should be a valid number, unable to parse string as a number
  [type=float_parsing, input_value='N/A', input_type=str]

pyamdsmi is not an in-house ctypes binding — it is from amdsmi import *, a bridge over AMD's official Python library. That library reports an unavailable reading in-band, rewriting the 0xFFFF / max-uint the driver answered into the string "N/A" inside dicts that otherwise hold numbers (amdsmi_interface.py:2594 for the whole power dict, :698 _validate_if_max_uint for the metrics). On a VF the host-side power telemetry is not exposed to the guest, the driver answers 0xFFFF, and amdsmi hands over "N/A". The detector defended against this convention at only 2 of 8 read sites and took every other reading at face value.

• _get_reading() — one private helper in amd.py mapping an AMD SMI reading to a value or a default, so the convention lives in one place instead of at six call sites. It is deliberately not safe_int/safe_str: those coerce, and this must be able to answer "absent" so a fallback can run.
• Usage query — a sentinel current_socket_power/average_socket_power now routes to the ROCm SMI fallback exactly the way a failed AMD SMI call already did (both mean "AMD SMI cannot tell the used power"); a sentinel average_gfx_activity/temperature_hotspot counts as absent, letting the existing "unreadable utilization is 0" guard do its job. Uses is None, so a genuine 0 W reading is not mistaken for a missing one.
• Inventory query — a sentinel power_limit falls back to rsmi_dev_power_cap_get instead of being floor-divided. This was worse than the reported symptom: "N/A" // 1000000 raises TypeError, which is not an AmdSmiException, so it escaped the handler and dropped the vendor — a host loses every AMD card. This machine's limit happened to be valid (750 W), which is the only reason it surfaced as a bad power_used instead. A sentinel driver_version / market_name is likewise no longer stored as a version or a board name.
• Hygon — a different, symmetric gap, not a sentinel one: pyrocmsmi checks every return code and raises instead of answering a sentinel, but the busy-percent, temperature and used-power reads were unwrapped where the AMD path wraps each equivalent call, so one unreadable metric raised out of detect_usage and cost the whole batch. Each is now isolated on its own. Isolation only — no sentinel checks added.

Not a regression. git log -S puts every one of these readings in f64a307 refactor: support amd detection (2025-09-23), the commit that introduced AMD support, in the form they still had. 518db70 (#17) only moved the block from detect() into detect_usage().

Other vendors are unaffected. Of the twelve bindings only three bridge a third-party library: pyamdsmi (the sentinel), pynvml and pymtml. pynvml's "N/A" occurrences all live in pynvml_utils/smi.py, the nvidia-smi table renderer; the core API this project calls signals unavailability by raising NVMLError. The other nine are in-house ctypes/CDLL bindings that return numbers and raise on failure, so they cannot produce a string sentinel.

Test plan

• uv run pytest tests/ — 613 passed, 20 skipped (was 605/20; 8 new guards). All new guards were re-run against main's source and all 7 code-path guards fail there, so none is vacuous.

  • test_amd.py — the four leak paths: both socket readings "N/A" (falls back, then reports None when ROCm SMI also cannot read it), power_limit "N/A" (no TypeError, falls back to the 700 W cap), metrics "N/A" (cores_utilization == 0, temperature is None), driver_version / market_name "N/A".
  • test_hygon.py — an unreadable power is bounded to its own card while the rest of that card's usage and the healthy card are untouched; an unreadable utilization does not carry off the temperature.
  • Both fake bindings gained one convention: a card whose fixture carries None for a reading stands for the one the binding cannot answer, and raises.

• Confirmed on the reported hardware (root@134.199.204.73, container 02132c6df882, AMD Instinct MI300X VF, driver 6.19.14.31400000). The container's amd.py was byte-identical to this branch's base, so the fixed file was dropped in and the container restarted:

  	Before	After
  detect --format json	"power_used": "N/A"	"power_used": null
  gpustack.worker.collector	1250 float_parsing failures, one per 5 s poll	none
  Exported GPU metrics	none — detection failed outright	worker_node_gpu_info, gpu_cores 304, gpu_power_limit_watts 750

  ROCm SMI cannot read the power on this VF either, which is why the field lands on null rather than a number — the intended outcome. The host was left running the patch; it does not survive recreating the container from the image.

Notes

The operator carries the same defect against the same hardware fact, in its own shape: binding/amdsmi is an in-house cgo binding, so it sees the raw integer and never a "N/A" string — quieter, not safer. Nothing crashes and 65535 is reported as a real measurement into monitoring and scheduling (PowerUsage W, CoresUtilization %, Temperature °C). That fix is out of scope here and is being made separately in gpustack.ai/gpustack, where the same block sits at pkg/devicemanager/detector/amd/device.go:315,316,323-325. Its Hygon detector already checks ret per reading and needs no change.

One asymmetry worth recording: the Python side gets an accidental layer of cover the Go side lacks — amdsmi's _validate_if_max_uint(..., isActivity=True) also rejects activity readings above 100.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gemini-code-assist]

Code Review

This pull request improves the robustness of AMD and Hygon GPU detectors by safely handling sentinel values (like "N/A") and isolating individual telemetry queries to prevent a single failure from crashing the entire detection sweep. The review feedback suggests defensive type-checking for power limit values to avoid a potential TypeError and ensuring the _get_reading helper handles non-dictionary inputs to prevent AttributeError crashes.

[gemini-code-assist]

Although the code checks if dev_power_limit is not None, it does not verify that the returned value is actually a numeric type (int or float). If dev_power_limit is an unexpected string or other non-numeric type, performing floor division (//) will raise a TypeError. Since TypeError is not suppressed by contextlib.suppress(pyamdsmi.AmdSmiException), it will propagate and crash the detector. Checking the type defensively prevents this potential crash.

Suggested change

	dev_power_limit = _get_reading(dev_power_info, "power_limit")
	if dev_power_limit is not None:
	dev_power = dev_power_limit // 1000000 # uW to W
	dev_power_limit = _get_reading(dev_power_info, "power_limit")
	if isinstance(dev_power_limit, (int, float)):
	dev_power = int(dev_power_limit) // 1000000 # uW to W

[gemini-code-assist]

The _get_reading helper function assumes that dev_info is always a dictionary and calls .get() on it. If dev_info is None or not a dictionary (which can happen if the underlying SMI call fails or returns an unexpected type), it will raise an AttributeError. Since this exception is not caught by specific handlers, it can propagate and crash the entire detector, causing the host to lose all AMD cards. Making this helper defensive against non-dictionary inputs improves robustness.

def _get_reading(dev_info: dict | None, key: str, default: Any = None) -> Any:
    """
    Read one value out of an AMD SMI answer, treating its sentinel as absent.

    AMD SMI reports an unavailable reading in-band: it rewrites the 0xFFFF (or
    max-uint) the driver answered into the string "N/A" before returning, so a
    dict otherwise holding numbers can carry a string at any key. A VF is where
    this surfaces, its host-side telemetry being invisible to the guest.

    Args:
        dev_info:
            The answer AMD SMI returned.
        key:
            The reading to take.
        default:
            What an absent or unavailable reading becomes.

    Returns:
        The reading, or the default.

    """
    if not isinstance(dev_info, dict):
        return default
    dev_reading = dev_info.get(key, "N/A")
    return default if dev_reading == "N/A" else dev_reading