docs(runpod): v2.10.0 surface, source-of-truth posture, and golden paths as step 0 of routing - #44
Merged
Merged
Conversation
justinwlin
force-pushed
the
docs/runpodctl-v2.9.0-surface
branch
from
August 5, 2026 21:07
a7ca307 to
7cf1446
Compare
lukepiette
approved these changes
Aug 5, 2026
lukepiette
left a comment
lukepiette
left a comment
There was a problem hiding this comment.
Verified the claims against the shipped v2.9.0 release notes and ran all four hooks on the head branch (validate_marketplace, check_versions, check_runpod_branding, check_links, which also checks the new #anchor fragments): everything passes, and the factual surface matches the release, including the corrections (the pod not ready: <reason> suffix, cli_error → timeout on --wait-for-hash, stopped pods losing ssh_command, connections as []).
Two comments worth acting on, neither blocking:
1. golden-paths/15-monitor-and-debug.md is now stale on the same claim this PR corrects elsewhere. Its signals table still steers agents to raw curl for /health and /status, and the Gotchas section says "runpodctl covers create/list/delete", which undersells the CLI after v2.9.0. The worker-logs half is still true (MCP/REST-only). If leaving golden paths as point-in-time documents is deliberate, a one-line note would keep someone from re-flagging it; otherwise it's a one-line touch-up.
2. timeout retry semantics sit in tension with the file's own invariant. output-and-errors.md opens with "Branch on code, never on status or the message text", but the new timeout row needs the message to tell "job still running, poll it" from "single call timed out, retry is fine". The retry-guidance bullet does handle it, and the error-handling eval's blanket "does NOT retry timeout" assertion is defensible under that eval's no-message-parsing premise: never retrying is the fail-safe default for a code-only handler. But a strict reader could see the eval and the reference disagreeing. One sentence in the codes table acknowledging timeout as the documented exception to the no-message-parsing rule, with no-retry as the code-only default, would close the gap.
Trivial nits, take or leave: the new serverless create ... --wait example in SKILL.md landed after serverless update, breaking the create/update grouping of that block; and the new --wait decision-rules bullet is at the ceiling of rule-8 body density (fine, since the deep detail lives in command-reference.md).
justinwlin
force-pushed
the
docs/runpodctl-v2.9.0-surface
branch
from
August 19, 2026 16:58
27d055e to
9a4906d
Compare
justinwlin
changed the title
…runtimeStatus
runpodctl v2.9.0 added three serverless commands, a readiness flag on both
creates, and a new pod status field. The skill described none of it, and four
claims across two files were made false by the release.
Additions:
- `serverless run` / `status` / `health`: the payload is the handler payload
(wrapped as `{"input": ...}`, validated and size-checked locally against the
10 MiB /run limit), stdout is the job payload even for a FAILED job while
progress and errors go to stderr, the exit codes, and the one that matters most
for an agent: a `timeout` naming a `serverless status` command means the job is
still running — poll it, never re-invoke and buy a second job.
- `--wait` / `--wait-timeout` on `pod create` and `serverless create`, as a
decision rule (don't hand-roll a poll loop) plus the guard rails: refused with
`--ssh=false`, warns on CPU and on community-without-public-ip, and a wait that
gives up keeps the resource and names it in the error object's `id`.
- `runtimeStatus` / `runtimeStatusReason` / `uptimeSeconds` on `pod get`/`pod
list`, with the two edges that bite: `--status` filters `desiredStatus` only,
and `unknown` means "the lookup failed", not "the pod is down".
- Two evals: invoke via `serverless run` rather than a hand-built curl, and
create-and-wait rather than a hand-rolled readiness loop.
Corrections the release forced:
- "there's no first-class serverless worker-log command, so diagnosis relies on
/health worker counts" — twice, in the runpodctl skill and in
runpod-usage/reference/endpoint-workflows.md. Now points at `serverless health`
for the counts and keeps the true half (worker *logs* remain MCP-only).
- output-and-errors.md said a local wait loop timing out is `cli_error`;
`model add --wait-for-hash` reports `timeout` as of v2.9.0.
- output-and-errors.md pinned the not-ready ssh shape as exactly
`{"error":"pod not ready",…}`; v2.9.0 appends a reason, so an exact match broke
and a prefix match did not. Also records that a stopped pod no longer returns a
dead ssh_command and that empty `connections` is now `[]`.
- The router's invoke row justified MCP with "MCP has first-class job tools",
which no longer differentiates the lanes. Verdict unchanged; the reason is now
that MCP is typed and only MCP streams job output and worker logs.
New codes (`timeout`, `job_failed`, `wait_timeout`, `wait_interrupted`) and the
new `id` error field went into output-and-errors.md, which already owns the code
table, rather than starting a second table in command-reference.md.
Verified against the shipped v2.9.0 binary, not the PR diffs: flag surface and
defaults from live --help; every local payload validation and the 10 MiB body
check by running them; the --wait guard rails with a bogus key so nothing was
created; `serverless health` against a live endpoint; and the `timeout` code by
forcing a 1ms per-call deadline, which also established both levers (`TIMEOUT`
env with no RUNPOD_ prefix, or a top-level `timeout` in config.toml above any
[section] — a trap that silently ate the value in my first attempt).
Golden path 15 still steered agents to raw curl for /health and /status, and told them runpodctl "covers create/list/delete" — the same claim this branch corrects elsewhere, left stale in the one document whose whole subject is the observability loop. Re-ran the path against live infrastructure on the v2.9.0 binary and rewrote it from what came back. The signals table is now first-choice/fallback rather than a single source: serverless health and serverless status lead, the curl stays as the zero-dependency equivalent and the thing you hand a user to copy-paste. The worker-log half of the old claim is true and survives, stated more precisely — there is no worker-log command in runpodctl and no worker-log path on REST v1, which is a statement about the service you call, not about an endpoint's own version field (a version:1 endpoint's logs read fine through the v2 route). The live run caught a real failure rather than a clean success, which is the better teaching case and closes the one code this branch had only source- verified: a job walked IN_QUEUE to FAILED with a server-side retry, /status reported "job timed out after 1 retries", and the container log showed the worker had passed all seven fitness checks and rejected the payload. The path now says not to trust the error string alone, and records that a retry resets delayTime rather than lengthening the queue. Also corrected while re-reading the observed output: the v2 worker-list summary has no ready key (that exists only on /health), so the two views disagree by design; serverless get --include-workers returns raw worker records carrying RUNPOD_AI_API_KEY, RUNPOD_ENDPOINT_SECRET and any handler tokens in env, which is worth knowing before pasting it into a ticket. The same overreach appeared in runpod-usage/reference/gotchas.md and golden-paths/03-whisper-endpoint/README.md; both corrected, keeping the true half. output-and-errors.md now names timeout as the one documented exception to its own branch-on-code-never-the-message rule, so the reference and the error-handling eval no longer read as disagreeing — a code-only handler that never retries timeout stays correct.
19-three-region-same-file.md ends with literal </content> and </invoke> tags — authoring-tool markup that was committed as document text and has been sitting on main. It renders as stray characters and is the kind of thing a reader takes for a formatting convention rather than a mistake. Unrelated to the v2.9.0 work; separated so it can be reverted or cherry-picked on its own.
The golden paths were a paragraph buried under the development loop, described as "acceptance scenarios, not installed skills" — accurate, and precisely the framing that makes a reader skip them. They are the most expensive artifact in this repo (someone ran each one on real infrastructure and pasted back what actually happened, including the parts that went wrong) and the least advertised. Promoted to their own section, led with the instruction to look for one before improvising, and given a phrasing a user can hand straight to an agent. Added a model-delivery section pointing at the host-side HF cache. The facts were already in runpodctl/reference/model-caching.md; what was missing was anything at README level, so the default a reader carries away came from storage.md's "when in doubt, choose the network volume" — right for datasets and checkpoints, wrong for a HuggingFace model on GPU serverless. The cache is host-side, so cold starts drop to seconds and download time is not billed, and it avoids the constraint that actually hurts: a network volume is pinned to one data center, which narrows the GPU availability you can schedule against. Scoped that storage.md default rather than weakening it, since it is correct for everything except model weights, and pointed it at the model-delivery section below. Its "Getting a model to the worker" heading also said "Three ways" over a four-row table — the Model Repository row was added without updating the count — so that is corrected and given the same routing rule of thumb the runpodctl reference already carried. Both the caching guidance and its constraints (GPU-only, runpodctl >= v2.4.0, HuggingFace-hosted models) are as documented in model-caching.md and verified against the v2.9.0 binary's flag surface.
runpodctl v2.10.0 (2026-08-19) added `pod logs` and `serverless logs` with `--follow`/`--tail`/`--since`/`--source`/`--worker`, emitting json lines. That falsifies eleven assertions across five files that worker logs were reachable only through MCP or a hand-built SSE read — most of which this branch had just added while documenting v2.9.0. The worst of them was an eval that graded an agent *correct* for answering "worker logs → NOT runpodctl … the MCP lane is the only one that can read them". A false negative in an eval trains the wrong behavior in rather than merely misinforming a reader, so that is rewritten to expect `serverless logs`, with the SSE path demoted to the pre-v2.10.0 fallback. Golden path 15's section 3 gains the CLI as Route A ahead of the raw v2 REST stream (Route B) and MCP (Route C); its four-signals table, triage checklist, gotchas and folded-back facts follow. `stream-job` is now the only genuinely MCP-only capability, and the router's capability matrix says so — updated in all three SKILL.md files per AGENTS.md rule 4. Also fixes a pre-existing gotcha in 03-whisper-endpoint/variant-a-hub.md that asked for `runpodctl serverless logs <endpoint-id>` by name, which now exists. Verified against the shipped v2.10.0 binary, not the release notes.
Both changed in runpodctl #319 and both inverted what the reference said.
`--output` validation: the doc claimed the flag is matched case-sensitively and
that an unrecognized value silently falls back to JSON, so `--output=table`
*and* `--output=YAML` both return JSON. On v2.10.0 exactly the opposite holds —
`table` exits 1 with `{"code":"usage_error"}` before any API call, and `YAML`
returns real YAML because matching is now case-insensitive. Recorded with the
v2.9.0 behavior kept alongside it, since a handler has to know which binary it
is on. (Upstream's own note is worth keeping: that silent fallback is how a
table format the CLI never had ended up documented in the first place.)
`exec` exit code: it printed errors as plaintext and still exited **0**, which
the plaintext-gaps table and the "don't trust the exit code" line both relied
on. It now emits coded JSON and a non-zero exit, leaving `project` as the only
surface where the exit code cannot be trusted. The error-handling eval asserted
the old grouping, so it asserted something false.
These lines sat four rows above this branch's own edits to the same file.
The file opened by admitting `runpodctl <resource> <action> --help` is authoritative and then spent 140 lines restating it — every Hub/template/volume/ registry/info/utility flag, hand-copied and silently rotting. Nothing in this repo can keep that current, and a wrong flag list is worse than no flag list. Kept only what `--help` cannot tell you: what "ready" is defined as for `--wait`, which pod status field to trust, the `serverless run` payload/exit-code contract, `pod logs`/`serverless logs` termination semantics, the `send`/`receive` one-time-code flow, and the `--include-workers` secret leak. Everything mechanical is now a pointer at the binary. Two things this surfaces that were missing entirely: - `pod get` returns `networkVolumeId` whenever a volume is attached as of v2.10.0 (runpodctl #322). On earlier binaries both it and `networkVolume` were dropped on deserialization and read back `null`, so a pre-v2.10.0 binary cannot answer "does this pod have a volume". - `serverless update` sends zero values as of v2.10.0 (#313). Before that `--workers-min 0` was dropped by `omitempty`, which means the SKILL.md instruction to set a dev endpoint back to `--workers-min 0` was a silent no-op and the endpoint kept billing. That is a cost-safety claim with no version floor, so it gets one in both places. Also notes the two `--help` traps a reader will hit: cobra does not traverse aliases (`pod remove --help` says unknown command though `pod remove <id>` works — use `runpodctl help pod remove`), and `get pod`/`get cloud` are hidden from `--help` but still live. 165 lines, down from 222, and none of them go stale on a flag change.
…urface
Absence claims are the one kind that rots silently. The
`--help`-is-authoritative rule does not protect them — there is no flag to look
up, so nobody reverifies — and the sentence reads as authoritative right up
until the release that adds the command. It has now happened twice in the same
skill: v2.9.0 added `serverless health`, falsifying "diagnosis relies on
/health worker counts"; v2.10.0 added `serverless logs`/`pod logs`, falsifying
eleven assertions including an eval that graded the false answer as correct.
So gate them, following the precedent rule 8 already sets for runpod-migrate's
Class-3 table: find every absence claim in the skills, extract the command it
names, and fail if that command exists in a vendored surface snapshot.
- `testdata/runpodctl/command-surface.json` — 66 commands walked out of the
v2.10.0 binary, plus the deprecated paths cobra hides from `--help`.
- `hooks/gen_cli_surface.py` — regenerates it against any binary.
- `hooks/check_cli_absence_claims.py` — the gate. Blocking in validate.yml
against the snapshot; `--live` in spec-drift.yml downloads the latest release
weekly, non-blocking, which is what would have caught this PR on the day.
Deliberately narrow, because a noisy gate gets disabled: a claim about a
missing *flag* is exempt (`--help` covers those), so is a negated or
version-floored one ("no longer MCP-only", "needs >= v2.10.0"), and genuinely
true absences live in an `ALLOW` list that requires a reason. Verified both
ways — reintroducing this branch's own "runpodctl still has no serverless logs
command" fails the check; the 16 surviving absence claims pass.
New rule 9 in AGENTS.md: prefer a positive claim or a version floor over an
open-ended "cannot", and never put a false negative in an eval.
The deference posture existed only inside two lanes — runpodctl's SKILL.md
("live --help is authoritative … the binary in front of you wins") and
runpod-mcp's ("the tool list above is a map, not a contract"). The router, which
is what an agent reads first, said nothing about it, and flash and
companion-clis had no equivalent at all.
So state it once at the entrypoint, with the specific thing to ask per lane:
`--help` and a deliberate wrong invocation for runpodctl, the client's tool list
for MCP, the live openapi.json for REST v2. Plus the rule this PR exists to
learn: never assert a tool cannot do something without checking, and if a limit
does hold, name the version it holds for instead of saying "cannot".
Also fixes a pointer this branch broke: SKILL.md still sent readers to
command-reference.md for the "full flag menu" after the previous commit removed
the flag lists from it. It now describes what that file actually holds — the
part --help cannot answer.
justinwlin
force-pushed
the
docs/runpodctl-v2.9.0-surface
branch
from
August 19, 2026 18:09
9a4906d to
454cb6c
Compare
…th pass Two gaps left by the previous commit. **Rule 6.** That commit changed router behavior — a new rule about how to treat capability claims — and shipped no eval, which rule 6 binds. Adds `no-unchecked-absence-claims.eval.md`, prompting for exactly the question this PR got wrong: "can I read serverless worker logs from the CLI, or do I need MCP?" It asserts the agent answers yes and names `serverless logs`, does not answer from the pre-v2.10.0 limitation, and expresses any version caveat as a floor rather than a denial. This is the one eval whose failure mode is a model confidently reciting an older copy of these very docs, so it is worth having even though the underlying fact is now stated in four places. **flash and companion-clis.** The router's new table points at `flash --help` and each companion CLI's `--help`, but neither lane's own SKILL.md mentioned `--help` at all — zero occurrences in both — so an agent entering the lane directly never got the pointer. Both now say their tool is authoritative for the command surface, with the version check to run before concluding something is missing (`pip show runpod-flash` for flash).
The examples were structurally unreachable. The router put "How to route" at
line 80, told the agent to "read the matching skill's SKILL.md next", and then
listed the worked examples 67 lines further down — so an agent that followed the
instructions routed into a lane and handed off before ever seeing them. Three of
six lanes (flash, runpod-usage, companion-clis) never mentioned golden paths at
all, and the frontmatter description — the one part always in context — did not
either. Nothing pulled toward the examples, so they only got used when a user
asked for them by name.
Reordering rather than nagging:
- **Step 0 of "How to route"**, ahead of lane selection, with a trigger stated in
observable terms: more than one resource or lane, provisions something
billable, an ask shaped like "get X running", or worth a multi-step plan. And
an explicit skip for a single read or single CRUD call, so "list my pods" does
not pay for a file read.
- **A matching path outranks the lane table** — it was verified end to end on a
real account, so a disagreement is a bug in the table, not in the path.
- The frontmatter description now advertises the 25 examples, and the handoff
sentence sends multi-step tasks to them first.
- Every lane links back, phrased for what that lane gets wrong without them:
MCP (tool calls are easy to issue in the wrong order), flash, runpod-usage
("this skill explains; the paths demonstrate"), companion-clis (each CLI is one
step inside a larger job).
- Partial matches count. The nearest path's ordering and gotchas usually transfer
even when the model or region does not, which is most of the value.
Fixes drift the new rule forbids: **path 21 (storage tiers) had no router row**,
so routing could not reach it even though golden-paths/README.md lists it.
Rule 5 rewritten to bind all of this and merged with the old duplicate
"Golden paths" rule (there were two, both with that title); rules renumbered to
11 and the progressive-disclosure cross-reference repointed.
`consult-golden-path-first.eval.md` asserts **both** directions — a one-line stop
must not open a path, and a host-cached HF deploy must find path 20 specifically
before proposing commands. Widening the trigger to "always read an example
first" breaks that eval on purpose.
justinwlin
changed the title
…sourcing, gate holes
Self-review of the branch found four real defects, three of them mine.
**"25 verified" examples was wrong twice.** I put it in the router's frontmatter
description, having read it off the highest filename (`25-bake-vs-mount`). There
are **24** paths (there is no 24), and 23 are live-verified while path 21 is
`documented (launch only)`. Now "two dozen live-verified", which matches
golden-paths/README.md's own Status column.
**The description ran to 3 sentences**, over rule 2's 1-2. It was already 3 before
this branch and I made it longer; condensed to 2 and it now names runpod-migrate,
which it had been omitting since that lane shipped.
**#322 was stated more strongly than the evidence.** I wrote that `pod get`
returns `networkVolumeId` "with or without `--include-network-volume`" as flat
fact. That comes from the PR author's hedged aside ("I believe this is the
desired behaviour"), and only the flagged path has a test. Reworded to attribute
it and to tell the reader to pass the flag rather than rely on it. Re-checked the
neighbouring `uptimeSeconds` claim while there — that one holds, #315 fixed the
`runtime.uptimeInSeconds` mapping and shipped in v2.9.0.
**The gate had two holes.** Probing it with eleven phrasings of the same false
claim, two got through:
- `CMD_IN_TEXT` was case-sensitive, so a claim opening a sentence with a
capitalized resource — "Serverless logs are only available through MCP" —
matched an absence pattern but resolved no command name and passed.
- "you must use MCP for X" / "X requires the MCP server" asserts absence with no
absence wording in it at all, and no pattern covered that form.
Both fixed, plus command names now resolve against the whole line rather than the
matched span (the command a claim is about frequently sits outside it). Guarded
`FLAG_CLAIM` against the resulting false positive on "lacks `templateId`
support", which is a field claim about MCP.
A gate whose patterns can silently stop matching is worth little, so
`--self-test` is now part of the check and runs first in CI: 11 phrasings that
must fail, 5 legitimate ones that must not. Every MUST_CATCH entry is a phrasing
that actually evaded an earlier version of these regexes. Verified the self-test
fails when the case-sensitivity hole is reintroduced.
No runner exists in this repo and no workflow reads `evals/*.eval.md`, so "an eval asserts X" is a claim about coverage that does not exist. AGENTS.md called them "regression scenarios", which invites exactly that reading, and I compounded it in rule 5 by saying widening the routing trigger "breaks that eval on purpose" — it cannot break anything. Both spots now say plainly that evals are specifications to hand-check a model against and to re-read when behavior changes, not a gate. This matters for reading the rest of this PR: the absence-claim check is enforced in CI, the golden-path routing change is not.
Ran the change instead of asserting it. Same prompts against origin/main and this branch, agents reporting which files they opened and in what order. **The router-entry scenario shows no effect.** 4/4 agents opened golden path 20 in both arms. The router's "Want to…" table already has a near-verbatim row for that task and agents read the file end to end, so where the section sits on the page never mattered — and origin/main already said "open its golden path first". Scenario A of this eval is therefore a weak test, and I have said so in the file rather than leaving it looking like coverage. **The lane-entry scenario is where the change does the work**, and it is also the realistic one: an agent already in a lane does not re-read the router. Entering at runpod-mcp/SKILL.md with a task that has no verbatim row (3-DC HA endpoint): | | opened a path | plan source | correct architecture | | without the pointer | 0 / 2 | derived | 0 / 2 | | with it | 2 / 2 | golden-path | 2 / 2 | Both treatment runs cited the added sentence by name and found path 19, a closer match than the path 10 this eval originally predicted. The control failures are worth recording precisely, because they are not laziness — origin/main's runpod-mcp SKILL.md mentions golden paths exactly once, in passing, inside an unrelated endpointType bullet below the tool surface, so the agent correctly concluded nothing pointed it anywhere. Deprived of the example both runs then invented a limitation to justify the plan they derived: "you cannot attach one network volume to a multi-DC endpoint" and "network volumes are not a mechanism for multi-DC redundancy of weights". Path 10 live-verified the opposite on 2026-07-10 — one endpoint, --network-volume-ids vol-ro,vol-cz, both attached, workers serving from both DCs. That is the same defect class as the docs bug this PR's CI gate exists to stop, arriving from the other direction: the gate stops the repo asserting a false absence, and surfacing the examples stops a model inventing one. Adds scenario B so the eval tests the case that actually differs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
runpodctl v2.10.0 shipped
pod logsandserverless logs. This PR started life covering v2.9.0 and asserting — in eleven places, including a new eval — that worker logs were MCP-only. That half is now false, so the interesting content here is the corrections, not the additions.What v2.10.0 forced
pod logs/serverless logsexist (--follow,--tail,--since,--source,--worker; json lines, not SSE). Eleven assertions across five files said otherwise. The worst wasroute-serverless-observability.eval.md, which graded an agent correct for answering "worker logs → NOT runpodctl … the MCP lane is the only one that can read them." A false negative in an eval trains the wrong behavior in rather than merely misinforming a reader — rewritten to expectserverless logs, SSE demoted to the pre-v2.10.0 fallback. Golden path 15 §3 is restructured: CLI is Route A, raw v2 REST is B, MCP is C.stream-jobis now the only genuinely MCP-only capability.--outputinverted. The reference said matching is case-sensitive and unrecognized values silently return JSON, so--output=tableand--output=YAMLboth give JSON. On v2.10.0tableexits 1 withusage_errorbefore any API call, andYAMLreturns real YAML. Both halves backwards; both behaviors now recorded, since a handler has to know which binary it's on.execexits non-zero now and emits coded JSON, leavingprojectas the only surface where the exit code can't be trusted. The error-handling eval asserted the old grouping.--workers-min 0was a silent no-op before v2.10.0 (#313 —omitemptyate the zero). The SKILL.md instruction to reset a dev endpoint back to scale-to-zero looked like it applied while the endpoint kept billing. Cost-safety claim, now version-floored.pod getreturnsnetworkVolumeIdas of #322; earlier binaries dropped it and read backnull, so they can't answer "does this pod have a volume".Additions (the original v2.9.0 scope)
serverless run/status/health,--wait/--wait-timeouton both creates,runtimeStatus/runtimeStatusReason/uptimeSecondsonpod get/list, and thetimeout/job_failed/wait_timeout/wait_interruptedcodes plus theidfield, folded into the existing code table. Two evals (serverless-invoke-job,create-and-wait-until-usable), each asserting the negative as well as the positive.Stopping the recurrence
Absence claims are the one kind that rots silently —
--helpdeference doesn't protect them, because there's no flag to look up, so nobody reverifies. This is the second time: v2.9.0 addedserverless health(falsifying "diagnosis relies on/healthcounts"), v2.10.0 added the log commands.So it's gated, following the precedent rule 8 already sets for runpod-migrate's Class-3 table:
testdata/runpodctl/command-surface.json— 66 commands walked out of the v2.10.0 binary, plus the paths cobra hides from--help.hooks/check_cli_absence_claims.py— blocking invalidate.ymlagainst the snapshot;--liveweekly inspec-drift.yml, which is what would have caught this PR on the day.hooks/gen_cli_surface.py— regenerates the snapshot against any binary.Deliberately narrow, because a noisy gate gets disabled: missing-flag claims are exempt (
--helpcovers those), so are negated and version-floored phrasings, and true absences need anALLOWentry with a reason.Also
reference/command-reference.md222 → 165 lines. It opened by admitting--helpis authoritative, then spent 140 lines restating it — hand-copied flag lists for Hub, templates, volumes, registry, info, utilities. Kept only what--helpcan't tell you (what "ready" means for--wait, which pod status field to trust, theserverless runexit-code contract, thesend/receivecode flow, the--include-workerssecret leak); everything mechanical is a pointer at the binary.Source of truth
The lanes wrap tools on their own release trains, so the router now says so up front: take
capability and syntax from
runpodctl --help/ the client's tool list /openapi.json, not fromthese files, and never assert a tool cannot do something without checking.
command-reference.mdfollows the same rule — it holds only what
--helpcannot answer.flashandcompanion-clissaythe same about their own tools.
no-unchecked-absence-claims.eval.mdwrites the expectation down,prompting for the exact question this PR originally got wrong.
Routing: golden paths were structurally unreachable
Separate problem, same root cause — the docs described a thing the reader never got to. The
router put lane selection first, said "read the matching skill's
SKILL.mdnext", and listed theworked examples 67 lines later, so an agent following the instructions handed off to a lane before
seeing them. Three of six lanes never mentioned golden paths at all, and neither did the
frontmatter
description— the only part always in context.Now: step 0 of "How to route", ahead of lane selection, triggered on observable properties
(more than one resource or lane, provisions something billable, "get X running", or worth a
multi-step plan) with an explicit skip for single reads and single CRUD calls. A matching path
outranks the lane table, since it was verified end to end on a real account. Every lane links
back. Partial matches count.
Also fixes drift the new rule forbids: path 21 had no router row, so routing could not reach
it.
consult-golden-path-first.eval.mddocuments both directions — a one-linestop podmustnot open a path; a host-cached HF deploy must find path 20 specifically before proposing
commands.
Validated against a control, not just specified
Nothing in this repo executes
evals/*.eval.md(no runner, CI does not read them), so the routingchange was measured by hand instead: same prompts against
origin/mainand this branch, agentsreporting which files they opened and in what order.
Router entry: no effect. 4/4 agents opened golden path 20 in both arms — the "Want to…" table
already has a near-verbatim row and agents read the file end to end. Said so in the eval rather than
leaving a weak test looking like coverage.
Lane entry (the realistic case — an agent already in a lane does not re-read the router), task
with no verbatim row, 3-DC HA endpoint:
origin/mainBoth treatment runs cited the added pointer by name and found path 19 — a closer match than the
path 10 the eval predicted. Both control runs, lacking the example, invented a limitation to
justify the plan they derived: "you cannot attach one network volume to a multi-DC endpoint" and
"network volumes are not a mechanism for multi-DC redundancy". Path 10 live-verified the opposite
on 2026-07-10. That is the same defect class as the docs bug the CI gate above stops, arriving from
the other direction — the gate stops the repo asserting a false absence; surfacing the examples
stops a model inventing one.
Verification
Everything checked against the shipped v2.10.0 binary, plus v2.9.0 for the before/after claims. An independent pass re-checked every claim adversarially and corrected three of mine (see below).
pod logs/serverless logsflag surface and defaults from live--help; absent on v2.9.0.--output:table/TABLE/xml→usage_errorexit 1 on v2.10.0, JSON on v2.9.0.YAML→ real YAML on v2.10.0, JSON on v2.9.0.--helpdiffed across all 18 commands the skills exercise — byte-identical apart from the two newlogscommands. No undocumented flag drift.runpodctl <resource> <action>in the skills resolves against the real v2.10.0 command tree; all aliases used (sls,nv,tpl,reg,dc,me,pod remove) are real.--liveresolved v2.10.0 from the GitHub API.Corrections from the independent pass, all taken: the count was 11 across 5 files (not ~15); the eval was ~1/3 false (not wholly), so step 3 was rewritten rather than the file deleted; and
pod remove --helpreturnsunknown commandbecause cobra doesn't traverse aliases for--help— the right check isrunpodctl help pod remove. That trap is now documented, since the same method would have wrongly condemned the live-but-hiddenget pod/get cloud.Not verified at runtime (no cost incurred): the
job_failedcode and thetimeout-names-a-follow-up message need a real job to fail or stall; theidfield needs a real--waitto time out; theserverless logsoutput shape is taken from the binary's interface rather than a fresh live capture (the captured frames in golden path 15 are the 2026-08-06 SSE run against the identical upstream service).No version bumps — release-please owns those (AGENTS.md rule 10).