rc8: reconcile stack QA fixes

[bussyjd]

Summary

What changed:

• Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
• Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.

Why it matters:

• This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
• The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.

Risk level: high while draft

Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85

Base branch: main

Scope

• Code
• Charts / manifests
• Flows / QA scripts
• Docs / skills
• Images / dependencies
• Other: reconciliation of open WIP branches

Validation

CI checks:

Check	Status	Link
GitHub CI	pending	draft PR

Unit tests:

go test ./... -count=1
PASS on local dev machine before push

Integration tests:

bash -n flows/*.sh
PASS

git diff --check
PASS

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
flow-13-dual-stack-obol.sh	Base Sepolia Anvil fork	QA machine A	isolated rc8 worktree	PASS	receipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09	Base Sepolia Anvil fork	QA machine A	isolated rc8 worktree	PASS after verifier-rollout diagnostic	legacy paid log retained on QA machine
release-smoke.sh	mixed	QA machine A	isolated rc8 worktree	NOT GREEN YET	latest full run diagnosed flow-08 stale verifier config

Release smoke:

Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

• Live gate: Base Sepolia, rerun pending on current HEAD.
• Fork regression: Anvil fork of Base Sepolia.

RPC/provider:

• Live: configured via existing QA environment, not copied here.
• Fork: local Anvil in isolated QA worktree.

Facilitator:

• Live flow uses the deployed public facilitator.
• Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.

Contracts and tokens:

Name	Address	Version / notes
OBOL Base Sepolia	0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4	live smoke token
Fork OBOL	run-generated	ForkObolToken, EIP-712 name checked in flow-13

Wallet roles:

Role	Address	Source
Alice / seller / register	0xC0De030F6C37f490594F93fB99e2756703c4297E	canonical derived seller wallet
Bob / buyer / payer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	canonical derived buyer wallet
Facilitator / receiver	pending final live rerun	do not infer from local fork logs

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Fork OBOL	Alice seller	10000000000000000000	10001000000000000000	+1000000000000000	+1000000000000000
Fork OBOL	Bob buyer	10000000000000000000	9999000000000000000	-1000000000000000	-1000000000000000

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
flow-13 settlement transfer	0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8	Bob	Alice	1000000000000000 atomic OBOL units	PASS
legacy paid settlement transfer	0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcb	buyer	seller	1000 USDC atomic units in fork flow	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux QA machine, exact host omitted
Backend	k3d stack worktrees
Tool versions	local toolchain from stack workspace; exact versions to capture in final RC run
QA agent/model	OpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast

Images:

Component	Image	Tag / digest	Source
x402 local facilitator	ghcr.io/x402-rs/x402-facilitator	current flow tag under review	GHCR
cloudflared	cloudflare/cloudflared	chart tag plus Renovate coverage to confirm	Docker Hub

Kubernetes / stack:

Item	Value
Stack IDs	isolated per-flow QA worktrees
Namespaces	generated by each stack flow
Pod readiness	checked by flows before payment assertions
Cleanup result	legacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending

Model and routing:

Item	Value
Agent/model used	qwen36-fast
LiteLLM route	obol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint status	HTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token source	obol agent auth --runtime ... obol-agent

Artifacts and logs:

Artifact	Location / link	Notes
flow-13 receipts	retained on QA machine	includes receipt summary and settlement evidence
legacy paid log	retained on QA machine	includes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke report	retained on QA machine	latest full run is diagnostic, not a green RC receipt

Demo readiness:

Item	Status	Notes
Seller visible / registered	partial	live OBOL current-HEAD rerun pending
Buyer discovery works	PASS in flow-13	agent buy structurally confirmed by PurchaseRequest
Paid route works	PASS in flow-13 and legacy paid slice	HTTP 200 paid inference observed
Settlement visible on-chain	PASS in fork flows	live current-HEAD rerun pending

Review Notes

Known gaps:

• The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
• Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
• x402 facilitator and cloudflared image tags need one final review before ready review.

Follow-ups:

• Replace the verifier restart with the real reload fix.
• Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.

Reviewer focus:

• Is the verifier config reload diagnosis correct?
• Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
• Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.

[bussyjd]

PR reconciliation pass

Fetched current remote refs and compared the open PR heads against integration/rc8-superset.

PR	Status in #417	Notes
#412 fix/model-config-order	Included	origin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415 oisin/413-1	Included as patch-equivalent	The branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411 Hermes recreation issues	Closed; safe subset retained	The Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.

Intentional supersession:

• Fixes from problematic pr worth keeping #415 added Helm 4-sensitive helmDefaults flags (--server-side=true, --force-conflicts) in embedded/generated helmfiles. rc8: reconcile stack QA fixes #417 removes those and replaces them with the Helm 3-compatible ownership-release path plus tests that fail if those flags return.
• rc8: reconcile stack QA fixes #417 still carries the x402-verifier rollout restart as WIP diagnostic evidence only; it should be replaced before ready review.

No missing open-PR work found in this pass.

[bussyjd]

Summary

What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.

Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.

Risk level: medium

Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9

Base branch: main

Scope

• Code
• Charts / manifests
• Flows / QA scripts
• Docs / skills
• Images / dependencies
• Other:

Validation

CI checks:

Check	Status	Link
GitHub CI	Pending after ready-for-review push	PR checks

Unit tests:

bash -n flows/*.sh
PASS

git diff --check
PASS

go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:

go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.

go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
flow-01-prerequisites	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-02-stack-init-up	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-03-inference	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-04-agent	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-05-network	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-06-sell-setup	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-07-sell-verify	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-10-anvil-facilitator	Base Sepolia fork	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-08-buy	Base Sepolia fork	QA machine A	qa-fullsmoke-20260505-035957	PASS	settlement receipt archived
flow-09-lifecycle	local QA	QA machine A	qa-fullsmoke-20260505-035957	PASS	release-smoke artifacts retained on QA machine
flow-11-dual-stack	Base Sepolia live USDC	QA machine A	qa-fullsmoke-20260505-035957	PASS	registration, metadata, settlement receipts archived
flow-14-live-obol-base-sepolia	Base Sepolia live OBOL	QA machine A	qa-fullsmoke-20260505-035957	PASS	registration, metadata, settlement receipts archived
flow-13-dual-stack-obol	Base Sepolia fork OBOL	QA machine A	qa-fullsmoke-20260505-035957	PASS	mint, funding, settlement receipts archived

Release smoke:

RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network: Base Sepolia live and Base Sepolia fork regression

RPC/provider: public Base Sepolia RPC selected by flow resolver

Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression

Contracts and tokens:

Name	Address	Version / notes
OBOL token, live Base Sepolia	0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4	name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression	0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429	deployed by flow-13 on Anvil fork
USDC, Base Sepolia	canonical Base Sepolia USDC	decimals=6

Wallet roles:

Role	Address	Source
Alice / seller / register	0xC0De030F6C37f490594F93fB99e2756703c4297E	canonical funded QA seller key
Bob / buyer / payer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	deterministic derived buyer key, preseeded into remote-signer
Bob remote-signer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	verified by flows before purchase

Balances:

Token	Address	Before	After	Expected delta	Actual delta
USDC live, Alice	0xC0De030F6C37f490594F93fB99e2756703c4297E	8933000	8934000	+1000 micro-USDC	+1000 micro-USDC
USDC live, Bob signer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	4985000	4984000	-1000 micro-USDC	-1000 micro-USDC
OBOL live, Alice	0xC0De030F6C37f490594F93fB99e2756703c4297E	12000000000000000	13000000000000000	+1000000000000000 wei	+1000000000000000 wei
OBOL live, Bob signer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	999998994000000000000000	999998993000000000000000	-1000000000000000 wei	-1000000000000000 wei
OBOL fork, Alice	0xC0De030F6C37f490594F93fB99e2756703c4297E	10000000000000000000	10001000000000000000	+1000000000000000 wei	+1000000000000000 wei
OBOL fork, Bob signer	0x57b0eF875DeB5A37301F1640E469a2129Da9490E	10000000000000000000	9999000000000000000	-1000000000000000 wei	-1000000000000000 wei
USDC fork buy, buyer	flow-08 buyer	1000000000	999999000	-1000 micro-USDC	-1000 micro-USDC
USDC fork buy, seller	flow-08 seller	465417851	465418851	+1000 micro-USDC	+1000 micro-USDC

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
USDC ERC-8004 registration	0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3	Alice	ERC-8004 registry	Agent ID 5486	PASS
USDC metadata / service offer	0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919	Alice	ERC-8004 registry	metadata published	PASS
USDC settlement transfer	0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673e	Bob signer	Alice	1000 micro-USDC	PASS
Live OBOL ERC-8004 registration	0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145	Alice	ERC-8004 registry	Agent ID 5487	PASS
Live OBOL metadata / service offer	0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2a	Alice	ERC-8004 registry	metadata published	PASS
Live OBOL settlement transfer	0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6	Bob signer	Alice	1000000000000000 wei OBOL	PASS
Fork OBOL buyer funding	0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304a	fork minter	Bob signer	10 OBOL	PASS
Fork OBOL settlement transfer	0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960	Bob signer	Alice	1000000000000000 wei OBOL	PASS
Fork USDC settlement transfer	0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08a	buyer	seller	1000 micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux aarch64
Backend	k3d v5.8.3, k3s v1.31.5-k3s1
Tool versions	Docker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/model	OpenAI-compatible QA endpoint, qwen36-fast

Images:

Component	Image	Tag / digest	Source
cloudflared	cloudflare/cloudflared	2026.3.0	embedded cloudflared values
x402 facilitator	ghcr.io/x402-rs/x402-facilitator	1.4.7	flow-10 / flow-13 fork tests
x402 verifier	ghcr.io/obolnetwork/x402-verifier	latest	embedded base manifest
x402 buyer	ghcr.io/obolnetwork/x402-buyer	latest	embedded LiteLLM sidecar manifest
LiteLLM	ghcr.io/obolnetwork/litellm	sha-c16b156	embedded base manifest
OpenClaw	ghcr.io/obolnetwork/openclaw	v2026.4.21	pinned OPENCLAW_VERSION
OpenClaw chart	obol/openclaw	0.4.0	CLI-generated agent helmfile
remote-signer chart	obol/remote-signer	0.3.2	CLI-generated agent helmfile
eRPC	ghcr.io/erpc/erpc	0.0.62	embedded eRPC values

Kubernetes / stack:

Item	Value
Stack IDs	unique per flow worktree; removed by cleanup
Namespaces	x402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readiness	x402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup result	no QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining

Model and routing:

Item	Value
Agent/model used	qwen36-fast
LiteLLM route	paid/qwen36-fast through x402-buyer sidecar
Paid endpoint status	HTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token source	obol agent auth --runtime hermes obol-agent; raw bearer tokens omitted

Artifacts and logs:

Artifact	Location / link	Notes
Release report	retained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.md	sanitized summary copied above
Receipt bundles	retained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receipts	includes registration, metadata, mint/funding, settlement receipts

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery works	PASS	Hermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route works	PASS	paid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chain	PASS	exact balance deltas and settlement receipts verified

Review Notes

Known gaps:

• Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
• x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.

Follow-ups:

• Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
• Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.

Reviewer focus:

• Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
• Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
• Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

[OisinKyne]

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.

We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.