docs: formal spec for remote agents and their management - #3748
Draft
tlongwell-block wants to merge 10 commits into
Draft
docs: formal spec for remote agents and their management#3748tlongwell-block wants to merge 10 commits into
tlongwell-block wants to merge 10 commits into
Conversation
Specifies the backend-provider plugin protocol (discovery, info/deploy, payload, security obligations), the remote lifecycle model (presence-is- status, !shutdown, inactivity self-stop), and the Kubernetes binding (buzz-backend-kubernetes) in the style of git-on-object-storage.md: stated invariants (I1-I5), explicit trust boundaries, a provider conformance checklist, implementation correspondence table, and the open decisions (A-E) pending owner ruling. Design thread: buzz-remote-agents 0eba4858 (Eva/Dawn/Wren). Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
added 9 commits
July 30, 2026 10:42
…review Wren's four correctness blockers: - I4: deploy restated as a reconciliation loop with normative conflict convergence (create-conflict re-reads and returns the winner, delete-not-found is success, loop bounded by the 600s deadline) - Truncated-label selection now requires full-pubkey annotation verification before any no-op/delete/Secret-mutation/return; mismatches are never GC'd - I5 restated as bounded *idle* lifetime / inactivity reaping, conditional on a live harness event loop; not an absolute TTL - M1 restated as no persistent post-deploy session / no substrate API in the protocol, not credential absence (kubeconfig is ambient by design) Wren's three contracts: nsec->pubkey derivation as step 0 with exact annotation/label/Secret/pod-name/agent_id names; Running->no-op means zero mutation (Secret lifecycle now follows the pod generation, resolving the old 'replaced atomically on re-deploy' conflict); full pod launch ABI (entrypoint, payload->env table, comma-join agent_args honesty note, HOME/cwd, image override must carry the runtime ABI). Dawn's three findings: deletionTimestamp checked before phase (no 'Terminating' phase exists; delete-marked pods stay Running), delete returns before the name frees (poll for 404; terminal pods delete immediately), entrypoint must exec so buzz-acp is the signal-receiving PID 1, and ~37s requalified as the bounded tail after an unbounded pool shutdown. GC redefined as a preflight reconciliation pass. Conformance section now points at fake-provider + envtest/kind tests instead of mechanization. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…seconds Wren round-3 findings: - Secret split-brain race: a deterministic shared Secret name lets a losing concurrent deploy's payload win the Secret write while the winner's pod wins the name — the winning generation runs the loser's config. Fixed with per-attempt immutable uniquely-named Secrets (buzz-agent-<12hex>-<gen>); pod spec references its exact Secret, pod creation elects the winner, losing contenders delete only their own unreferenced Secret, GC reaps annotation-verified orphan Secrets. - inactivity_seconds was inert: the ABI table hardcoded BUZZ_ACP_EXIT_AFTER_INACTIVITY=7200 while the schema exposed the field. Now one knob: config field feeds the env var, 7200 is the schema default (Auto-Stop section updated to match). - 'generation' now has a concrete definition in §GC (one pod-create attempt + its uniquely-named Secret; current = referenced by the surviving pod's envFrom). Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…cs, GC age gate, hardening Fold in verified findings from Dawn, Wren, Mari, and Perci: - §Launch data: typed launch block with three-tier env precedence (overridable policy_env → user/layered env → authoritative), exact policy_env contents (team instructions, resolved session title, owner_pubkey, unconditional runtime default_env, lazy_pool=true, MCP_HOOK_SERVERS), dedup/steer deliberately unset (harness defaults), host-resolved re-derivation incl. CLAUDE_CODE_EXECUTABLE, generation token doubling as BUZZ_MANAGED_AGENT_START_NONCE - §Deploy State Machine: startup is part of create — success requires container state.running; started/never-started rows; 600s deadline is the explicit cold-start budget, expiry = startup-not-confirmed - §K8s GC: normative orphan-Secret age gate (>2× deploy deadline) closing the preflight-GC vs in-flight-attempt Secret race; ownerReferences alternative considered and rejected with rationale; log shipping named a production prerequisite with correlation contract - §Discovery: shadowed/invalid candidate diagnostics; §info: protocol_version with compatible-major rule - §Pod shape: normative hardening defaults (no SA token automount, non-root, no privilege escalation, drop-all caps, RuntimeDefault seccomp); §Image: digest-pinned default reference, :latest rejected - §Auto-Stop: reaper must not depend on pool_ready (lazy-pool deadlock), BUZZ_ACP_EXIT_AFTER_INACTIVITY joins RESERVED_ENV_KEYS; Known Defect 4 records the unimplemented reaper and the tick trap - Open Decisions F–H (mesh deployability, remote override semantics, startup budget SLO) added for ratification Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Rich guarantees are allowed; complex implementations are not. Name the rule explicitly in §Invariants: every property in the spec was chosen because its enforcing mechanism is one small, boring thing, and a conforming implementation that is not small is evidence of a spec bug. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…n-thrashing startup classification Round-6 review amendments from Dawn and Wren: - Same-clock rule (normative): the orphan-Secret age comparison must use the apiserver's clock via the HTTP Date header on the GC list call; absent/unparseable header means skip orphan GC for the pass, never a local-clock fallback. A fast desktop clock is not a race — it deterministically deletes every in-flight Secret. - Corrected ownerReferences rationale: never-started winners are repairable (kubelet retries a nil-status container regardless of restartPolicy: Never), so Secret-first wins on diagnostics, not unrepairability. ownerReferences omitted entirely in v1 under the complexity budget; constraints stated (blockOwnerDeletion: false, same-namespace, non-deploy-failing) for any future attachment. - State machine: the single fatal never-started row split into three — provably non-recoverable (evidence, not reason strings), past-bound on the apiserver clock, and self-healable-within-bound (Unschedulable during scale-from-zero, pull backoff, transient config errors) — preventing thrash and preserving the cold-start budget. - Conformance: never-started winner repairable (delete-recreate, not no-op) and fast-local-clock must-not-delete cases added; conformance item 4 reworded to the evidence-based classification. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…, preconditioned deletes, anti-livelock startup Round-7 review amendments from Wren and Dawn: - Precedence: the four record-derived behavior knobs (system_prompt, idle_timeout, max_turn_duration, parallelism) fold into launch.policy_env; provider MUST NOT remap the top-level copies — locally they sit before the user env layer (runtime.rs:716-729,763 vs :860) and are not reserved, so remapping would invert an override that works locally. - Pre-secret negotiation gate (normative + Known Defect 5): resolve provider once, invoke info, check protocol_version, then deploy on the same executable identity before the nsec crosses; UI-time probe does not satisfy the gate. provider_deploy at c1bca1b sends deploy unchecked. - Redaction follow-through on Known Defect 3: launch.env / launch.policy_env values join the literal-value scrub; env_secrets_from_request currently reads only agent.env_vars. - Controlled-view rule (normative): destructive-authorizing reads use quorum semantics (resourceVersion unset), and every classification- authorized DELETE carries UID+resourceVersion preconditions from that observation; failed precondition re-enters, never retries. Unified with the same-clock rule. - Anti-livelock: the past-bound age-delete row is removed — recoverable startup timeout is observational across calls (any finite pod-age threshold can collide with Cluster Autoscaler's new-pod-scale-up-delay and delete-recreate resets the age it keys on). 600s bounds only synchronous wait; creationTimestamp survives. Decision H narrowed to a UX-only SLO. - Conformance: classification→DELETE race, provisioning-after-600s adoption, protocol-mismatch and executable-replaced cases; item 4 rewritten. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…ytes executable identity Wren's round-8 findings on the pre-secret gate: - Missing protocol_version is an error, never a presumed 1: the one-major-cycle grace period granted the nsec to exactly the class of binary that never declared compatibility. No deployed provider population exists to grandfather; failing closed is also simpler (no migration clock). - Executable identity = staged bytes, not path+metadata: unchanged dev/inode/size/mtime misses in-place content rewrites, and a pathname can be swapped between the check and Command spawn (check-then-exec race). The gate now requires copying the resolved candidate into a desktop-owned private staging file (digest computed during copy) and running both info and deploy from that artifact, so the nsec goes to the exact bytes that answered info. Platforms where staged execution is impossible degrade explicitly to accidental-replacement detection and must surface the weaker level — never silently claim the gate. - Conformance: same-inode content rewrite and pathname-swap-after- validation cases replace the simple rename case; absent protocol_version rejection added. - Known Defect 5 and Implementation Correspondence updated to the stage-and-digest shape. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…t fingerprint Dawn's round-9 findings, with the fingerprint design converged by both reviewers: - 409 discriminator is Status.reason, never the status code: NewConflict and NewAlreadyExists both carry 409 (verified against apimachinery); reason `Conflict` -> re-read and re-classify, reason `AlreadyExists` -> the create-conflict convergence path. This does not contradict the container-waiting-reason distrust rule (API Status.reason is a machine-readable contract; kubelet waiting reasons are not). - Config-wedge escape: the never-started recoverable row splits in two (state machine is now seven rows). Fingerprint matches -> observe forever, never delete; fingerprint differs or absent -> preconditioned delete-recreate. Divergence is evidence, not a clock: age resets only on a user-initiated config change, so no autoscaler-threshold collision. - Create-intent fingerprint (annotation buzz.block.xyz/create-intent): unkeyed SHA-256 over the canonical non-secret create-intent template, computed pre-create. Scope covers only provider-controlled fields affecting scheduling/container creation; never Secret data or attempt identity — Secret values cannot cause the never-started wedge, and excluding them removes the dictionary oracle and any need for an HMAC key. Per-attempt Secret name in envFrom normalized to a placeholder; comparison is recorded annotation vs freshly-computed intent, never the admission-mutated live spec. Started pods strict no-op regardless of divergence; missing Secret stays with the absence check. - Conflict-path asymmetry: the create loser never applies the divergence row to the winner it just lost to (prevents ping-pong deletes); a subsequent deploy replaces a never-started divergent winner normally. - Conformance: third mandatory family — 409 split, identical-intent never deletes, wedge escape, started-pod no-op, admission no false divergence, structural serializer property (Secret values/generated name never change the fingerprint; fingerprinted fields do), and the conflict-path asymmetry. - Open Decision I: is divergence-triggered replacement enough, or does v1 owe an explicit "clear stuck deployment" affordance? Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Dawn's post-sign-off precision correction, seconded by Wren (non-blocking; both hold 9/9/9): "age resets only on a user-initiated config change" overclaimed. The fingerprint includes the resolved image reference, and for users who never set `image` that resolves to the digest the provider bakes at compile time — so a provider upgrade changes the computed intent with no user action. The divergence row now names both sources — user config change, and provider upgrade moving the baked default digest — and states why the second is deliberate (a fingerprint blind to the default resolution could never replace a pod wedged by a bad baked default: unpullable digest, wrong arch — exactly the population that cannot override `image`) and its bounded cost (an upgrade mid-cold-pull discards in-flight startup progress; not age-driven or periodic, so no autoscaler-livelock shape). Conformance: default-digest-change case added to the third family. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A formal specification for remote agents and their management —
docs/remote-agents.md— in the style ofdocs/git-on-object-storage.md: stated system model, named invariants, explicit trust boundaries, provider conformance checklist, and an implementation-correspondence table.Requested by Tyler in the buzz-remote-agents design thread; co-designed with Dawn and Wren (review pending).
Structure
info/deploywire contract, untrusted-output rules, the reserved-key rule, and the deploy state machine (Running → no-op).--exit-after-inactivity/BUZZ_ACP_EXIT_AFTER_INACTIVITY, default off, definition of "inactive", and why it must not share a name with the three existing timeout concepts.buzz-backend-kubernetes: kubeconfig-only auth, random-default namespace via schemadefault, the sprig image, pod shape (bare Pod,terminationGracePeriodSeconds: 60, 32-hex label / full-pubkey annotation), secrets, GC, config budget.c1bca1b56(Windows.exeid pollution; provider env inheritance vs kubeconfig exec plugins).Notes for review
Docs-only. Every code claim was verified against the tree (correspondence table maps each spec concept to its file/function). The spec deliberately documents two desktop bugs as Known Defects rather than fixing them here — fixes are follow-up PRs.