Skip to content

PMM-15326: Report the executor fleet, not only the part of it that works - #1390

Open
plebioda wants to merge 2 commits into
PMM-15299-open-managerfrom
PMM-15326-fleet-states
Open

PMM-15326: Report the executor fleet, not only the part of it that works#1390
plebioda wants to merge 2 commits into
PMM-15299-open-managerfrom
PMM-15326-fleet-states

Conversation

@plebioda

@plebioda plebioda commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds GET /api/tasks/hosts/states/: one entry per host the executor backend knows about, usable or not, with reachable and driver_healthy reported separately.

GET /hosts/ answers "where can a job be placed". It gets there by filtering on three conditions at once - Status == ready, raw_exec present in Drivers, and that driver's Healthy flag - and returning a name-to-address mapping. That is the right answer for a dispatcher and all a dispatcher needs.

It is the wrong answer for anything reporting on the fleet, because the three conditions collapse into one bit and every failure lands in the same place: absence from the mapping. A machine that is missing may never have been onboarded, or be onboarded and down, or be up with a broken driver - three problems for three different people, and the caller cannot tell which, or even that the machine exists.

Nothing about /hosts/ changes and no caller is moved. get_host_states is concrete on BaseExecutor, defaulting to "everything get_hosts returns, reachable and healthy", so a backend with no notion of an unusable host says nothing (CeleryExecutor runs the work in-process); Nomad overrides it.

Three details worth a reviewer's attention:

  • The unfiltered node list is fetched without resources=True. The stub entries already carry Status and Drivers, and the detail fetch is one request per node against a Nomad that may have hundreds.
  • A missing raw_exec key reads as unhealthy, not as absent-so-fine. Nomad omits drivers it has not detected, so a never-onboarded host has no key at all, and treating that as healthy would report the emptiest case as the best one.
  • detail carries the driver's HealthDescription only when it is a problem - Nomad sets it to the literal "Healthy" on a working driver, and a field whose job is to explain failures must not be full of the word "Healthy".

The raw_exec driver name and the ready status are now named constants shared by the dispatch filter and the reporting, so the two cannot drift into disagreeing about what "healthy" means.

Worth calling out for review: this endpoint is broader than /hosts/. It returns every registered client's name, address and driver detail to any authenticated tasks user, where /hosts/ returns only the placeable ones. That is deliberate - "why can nothing run on this machine" is unanswerable otherwise - but it is a disclosure change and should be an explicit decision rather than an inferred one. Happy to admin-gate it if that is the preference.

Why this is its own PR

Wanted by OpenManager, which has to describe the hosts it cannot probe, but nothing here is OpenManager-specific: no OM code, no OM imports, and the OM app reaches this over HTTP like any other client. It is the first of four PRs replacing the single 21k-line draft #1371, which stays open until the replacements are up. The others are the core app-owned-settings extraction, the om_inventory app itself, and the side-car activation.

Base is PMM-15299-open-manager, the integration branch for the epic, created from pmm and identical to it at the time of this PR.

Two commits: the code, then the regenerated OpenAPI spec and TS client on their own so the first commit is all a reviewer has to read. Both derived files are dump_openapi.py and pnpm codegen output, not hand-written - 103 lines added to specs/tasks.json and 101 to src/generated/tasks.ts, none removed, and the other two specs and clients untouched.

Tested

Note that CI does not run here - .github/workflows/ci.yml triggers on pull_request with branches: [main], so a PR based on pmm or on this integration branch gets no test job. Everything below was run locally against this branch.

  • venv/bin/python -m pytest tests/app -q -n 8 - full suite on this branch: 9043 passed, 424 skipped
  • tests/app/tasks/execution/executors/nomad/test_models.py - get_host_states over a ready host, a down host, a host with an unhealthy driver, a host with no raw_exec key at all, and the HealthDescription passthrough
  • tests/app/tasks/execution/executors/celery/test_models.py - the base default, that a backend which does not override it reports its hosts as reachable and healthy
  • tests/app/tasks/test_routes.py - the route, its 502 mapping, and that /hosts/ is unchanged
  • tests/app/test_openapi_specs_fresh.py - the committed spec matches scripts/dump_openapi.py
  • make run-pre-commit - all 22 hooks pass, oxfmt and oxlint included
  • pnpm --filter @sep/api codegen + oxfmt --write src/generated - reproduces the committed client byte-for-byte, and leaves the other three clients untouched

Checklist

  • New/modified functions have type hints and rST docstrings
  • New tests added for new features or bug fixes
  • All tests pass locally (make test)
  • Pre-commit hooks pass (make run-pre-commit)
  • Database migrations generated if models changed (make makemigrations) - N/A, no model changes
  • User-facing changes documented (README, inline help, UI text) - N/A. The two docs that name /api/tasks/hosts/ (README.md on DEFAULT_EXECUTOR_HOST, docs/customer/nomad-driver-deployment.md on choosing a dispatch target) both describe where a job can be placed, which this does not change. The new endpoint carries its own OpenAPI description.
  • Configuration changes documented with examples - N/A, no configuration changes
  • Changelog fragment added under changelog.d/ - cannot be, and this needs a decision. scripts/changelog.py's FRAGMENT_RE only accepts SEP-<n>.<section>.md, and a filename it does not match is a hard FragmentError, not a skip - so a PMM-15326 fragment cannot exist. As it stands the whole OpenManager SEP-side feature would ship with no SEP release note. Two ways out: file a SEP-xxxx for the SEP half of this work and put the fragment there (a multi-line fragment renders one bullet per line, so one file can cover this endpoint and the app), or accept that the release note for this feature comes from PMM's side only. Tell me which and I will do it.

`GET /hosts/` answers "where can I place a job". `NomadExecutor.get_hosts` produces
it by filtering on three conditions at once - `Status == ready`, `raw_exec` present
in `Drivers`, and its `Healthy` flag - and returning a name-to-address mapping. That
is the right answer for a dispatcher and all it needs.

It is the wrong answer for anything reporting on the fleet, because the three
conditions collapse into one bit and the failures land in the same place: absence.
A machine missing from that mapping may never have been onboarded, or be onboarded
and down, or be up with a broken driver. Those need three different people to fix
them, and a caller looking at the mapping cannot tell which it is - or even that the
machine exists.

So `GET /hosts/states/` alongside it, returning one `ExecutorHostState` per host the
backend knows about, with `reachable` and `driver_healthy` reported separately and
the driver's own `HealthDescription` carried along. Nothing about `/hosts/` changes;
no caller is moved.

`get_host_states` is concrete on `BaseExecutor` rather than abstract, defaulting to
"everything `get_hosts` returns, reachable and healthy". That is true by construction
for any backend, and it means a backend with no notion of an unusable host does not
have to say so - `CeleryExecutor` runs the work in-process and has nothing to add.
Nomad overrides it. Making it abstract would have edited every implementation and
every test double to say nothing.

Three details worth keeping:

- The unfiltered node list is fetched without `resources=True`. The stub entries
  already carry `Status` and `Drivers`, and the detail fetch is one request per node
  against a Nomad that may have hundreds.
- A missing `raw_exec` key reads as unhealthy, not as absent-so-fine. Nomad omits
  drivers it has not detected, so the never-onboarded host has no key at all -
  treating that as healthy would report the emptiest case as the best one.
- `detail` carries the driver's `HealthDescription` only when it is a problem. Nomad
  sets it to the literal "Healthy" on a working driver, and a field whose job is to
  explain failures must not be full of the word "Healthy".

The driver name and the ready status are now named constants shared by the dispatch
filter and the reporting, so the two cannot drift into disagreeing about what
"healthy" means.

Wanted by OpenManager, which has to describe the hosts it cannot probe, but nothing
here is OpenManager-specific: "why can nothing run on this machine" is a question the
tasks service is the only thing able to answer, and it will outlive the app that
asked first.
…ates/

Derived, not authored: `tests/app/test_openapi_specs_fresh.py` runs
`scripts/dump_openapi.py --check` and fails the moment a route exists that the
committed spec does not carry, so the regeneration has to ride in the same change
that adds the route. Kept as its own commit so the previous one is only the code a
reviewer has to read.

Both files are what the tooling produces, not what a hand wrote:
`specs/tasks.json` from `scripts/dump_openapi.py` and `src/generated/tasks.ts` from
`pnpm --filter @sep/api codegen` followed by `oxfmt --write src/generated`, which is
`make regen-specs` minus the snapshot goldens no route here touches.

103 lines added to the spec and 101 to the client, none removed: the path, the
`ExecutorHostState` schema, and the operation type. The other two clients and the
other two specs are untouched, which is the check that this branch carries only its
own share of the contract.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant