PMM-15326: Report the executor fleet, not only the part of it that works - #1390
Open
plebioda wants to merge 2 commits into
Open
PMM-15326: Report the executor fleet, not only the part of it that works#1390plebioda wants to merge 2 commits into
plebioda wants to merge 2 commits into
Conversation
`GET /hosts/` answers "where can I place a job". `NomadExecutor.get_hosts` produces it by filtering on three conditions at once - `Status == ready`, `raw_exec` present in `Drivers`, and its `Healthy` flag - and returning a name-to-address mapping. That is the right answer for a dispatcher and all it needs. It is the wrong answer for anything reporting on the fleet, because the three conditions collapse into one bit and the failures land in the same place: absence. A machine missing from that mapping may never have been onboarded, or be onboarded and down, or be up with a broken driver. Those need three different people to fix them, and a caller looking at the mapping cannot tell which it is - or even that the machine exists. So `GET /hosts/states/` alongside it, returning one `ExecutorHostState` per host the backend knows about, with `reachable` and `driver_healthy` reported separately and the driver's own `HealthDescription` carried along. Nothing about `/hosts/` changes; no caller is moved. `get_host_states` is concrete on `BaseExecutor` rather than abstract, defaulting to "everything `get_hosts` returns, reachable and healthy". That is true by construction for any backend, and it means a backend with no notion of an unusable host does not have to say so - `CeleryExecutor` runs the work in-process and has nothing to add. Nomad overrides it. Making it abstract would have edited every implementation and every test double to say nothing. Three details worth keeping: - The unfiltered node list is fetched without `resources=True`. The stub entries already carry `Status` and `Drivers`, and the detail fetch is one request per node against a Nomad that may have hundreds. - A missing `raw_exec` key reads as unhealthy, not as absent-so-fine. Nomad omits drivers it has not detected, so the never-onboarded host has no key at all - treating that as healthy would report the emptiest case as the best one. - `detail` carries the driver's `HealthDescription` only when it is a problem. Nomad sets it to the literal "Healthy" on a working driver, and a field whose job is to explain failures must not be full of the word "Healthy". The driver name and the ready status are now named constants shared by the dispatch filter and the reporting, so the two cannot drift into disagreeing about what "healthy" means. Wanted by OpenManager, which has to describe the hosts it cannot probe, but nothing here is OpenManager-specific: "why can nothing run on this machine" is a question the tasks service is the only thing able to answer, and it will outlive the app that asked first.
…ates/ Derived, not authored: `tests/app/test_openapi_specs_fresh.py` runs `scripts/dump_openapi.py --check` and fails the moment a route exists that the committed spec does not carry, so the regeneration has to ride in the same change that adds the route. Kept as its own commit so the previous one is only the code a reviewer has to read. Both files are what the tooling produces, not what a hand wrote: `specs/tasks.json` from `scripts/dump_openapi.py` and `src/generated/tasks.ts` from `pnpm --filter @sep/api codegen` followed by `oxfmt --write src/generated`, which is `make regen-specs` minus the snapshot goldens no route here touches. 103 lines added to the spec and 101 to the client, none removed: the path, the `ExecutorHostState` schema, and the operation type. The other two clients and the other two specs are untouched, which is the check that this branch carries only its own share of the contract.
plebioda
force-pushed
the
PMM-15326-fleet-states
branch
from
August 21, 2026 13:40
d7fdc56 to
39fb30a
Compare
This was referenced Aug 21, 2026
plebioda
marked this pull request as ready for review
August 21, 2026 20:42
plebioda
requested review from
marcuscruz-percona,
nachodd,
peter-o-addo and
yyyyyyyan
as code owners
August 21, 2026 20:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
GET /api/tasks/hosts/states/: one entry per host the executor backend knows about, usable or not, withreachableanddriver_healthyreported separately.GET /hosts/answers "where can a job be placed". It gets there by filtering on three conditions at once -Status == ready,raw_execpresent inDrivers, and that driver'sHealthyflag - and returning a name-to-address mapping. That is the right answer for a dispatcher and all a dispatcher needs.It is the wrong answer for anything reporting on the fleet, because the three conditions collapse into one bit and every failure lands in the same place: absence from the mapping. A machine that is missing may never have been onboarded, or be onboarded and down, or be up with a broken driver - three problems for three different people, and the caller cannot tell which, or even that the machine exists.
Nothing about
/hosts/changes and no caller is moved.get_host_statesis concrete onBaseExecutor, defaulting to "everythingget_hostsreturns, reachable and healthy", so a backend with no notion of an unusable host says nothing (CeleryExecutorruns the work in-process); Nomad overrides it.Three details worth a reviewer's attention:
resources=True. The stub entries already carryStatusandDrivers, and the detail fetch is one request per node against a Nomad that may have hundreds.raw_execkey reads as unhealthy, not as absent-so-fine. Nomad omits drivers it has not detected, so a never-onboarded host has no key at all, and treating that as healthy would report the emptiest case as the best one.detailcarries the driver'sHealthDescriptiononly when it is a problem - Nomad sets it to the literal"Healthy"on a working driver, and a field whose job is to explain failures must not be full of the word "Healthy".The
raw_execdriver name and the ready status are now named constants shared by the dispatch filter and the reporting, so the two cannot drift into disagreeing about what "healthy" means.Worth calling out for review: this endpoint is broader than
/hosts/. It returns every registered client's name, address and driver detail to any authenticated tasks user, where/hosts/returns only the placeable ones. That is deliberate - "why can nothing run on this machine" is unanswerable otherwise - but it is a disclosure change and should be an explicit decision rather than an inferred one. Happy to admin-gate it if that is the preference.Why this is its own PR
Wanted by OpenManager, which has to describe the hosts it cannot probe, but nothing here is OpenManager-specific: no OM code, no OM imports, and the OM app reaches this over HTTP like any other client. It is the first of four PRs replacing the single 21k-line draft #1371, which stays open until the replacements are up. The others are the core app-owned-settings extraction, the
om_inventoryapp itself, and the side-car activation.Base is
PMM-15299-open-manager, the integration branch for the epic, created frompmmand identical to it at the time of this PR.Two commits: the code, then the regenerated OpenAPI spec and TS client on their own so the first commit is all a reviewer has to read. Both derived files are
dump_openapi.pyandpnpm codegenoutput, not hand-written - 103 lines added tospecs/tasks.jsonand 101 tosrc/generated/tasks.ts, none removed, and the other two specs and clients untouched.Tested
Note that CI does not run here -
.github/workflows/ci.ymltriggers onpull_requestwithbranches: [main], so a PR based onpmmor on this integration branch gets no test job. Everything below was run locally against this branch.venv/bin/python -m pytest tests/app -q -n 8- full suite on this branch: 9043 passed, 424 skippedtests/app/tasks/execution/executors/nomad/test_models.py-get_host_statesover a ready host, a down host, a host with an unhealthy driver, a host with noraw_execkey at all, and theHealthDescriptionpassthroughtests/app/tasks/execution/executors/celery/test_models.py- the base default, that a backend which does not override it reports its hosts as reachable and healthytests/app/tasks/test_routes.py- the route, its 502 mapping, and that/hosts/is unchangedtests/app/test_openapi_specs_fresh.py- the committed spec matchesscripts/dump_openapi.pymake run-pre-commit- all 22 hooks pass,oxfmtandoxlintincludedpnpm --filter @sep/api codegen+oxfmt --write src/generated- reproduces the committed client byte-for-byte, and leaves the other three clients untouchedChecklist
make test)make run-pre-commit)make makemigrations) - N/A, no model changes/api/tasks/hosts/(README.mdonDEFAULT_EXECUTOR_HOST,docs/customer/nomad-driver-deployment.mdon choosing a dispatch target) both describe where a job can be placed, which this does not change. The new endpoint carries its own OpenAPI description.changelog.d/- cannot be, and this needs a decision.scripts/changelog.py'sFRAGMENT_REonly acceptsSEP-<n>.<section>.md, and a filename it does not match is a hardFragmentError, not a skip - so aPMM-15326fragment cannot exist. As it stands the whole OpenManager SEP-side feature would ship with no SEP release note. Two ways out: file aSEP-xxxxfor the SEP half of this work and put the fragment there (a multi-line fragment renders one bullet per line, so one file can cover this endpoint and the app), or accept that the release note for this feature comes from PMM's side only. Tell me which and I will do it.