RFC: run-scoped usage, budget, and cancellation controls for AgentOptimizer (OptimizationRunContext + OptimizerCapabilities)

[caohy1988]

Summary

Optimizer-owned model calls are currently opaque to callers: AgentOptimizer.optimize(initial_agent, sampler) accepts no per-run control, exposes no usage ledger, no cancellation signal, and no capability declaration. Governance wrappers (quality flywheels, budget-enforced CI optimization, cost-attributed experimentation) cannot observe or stop the model calls the optimizer itself makes.

This RFC proposes the smallest additive seam: an optional run-scoped OptimizationRunContext on optimize(...), instrumentation of SimplePromptOptimizer and GEPARootAgentPromptOptimizer, and conservative OptimizerCapabilities defaults for every optimizer so a wrapper can fail preflight instead of discovering opacity after spend.

Target reviewed: main at 9d306f5 (2026-07-10). Downstream consumer with a pinned contract waiting on this seam: GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK#318.

Why the current surface is not enough

• SimplePromptOptimizer calls BaseLlm.generate_content_async() directly per candidate; LlmResponse.usage_metadata is available but discarded.
• GEPARootAgentPromptOptimizer / GEPARootAgentOptimizer own their reflection LLM callable and discard reflection usage; max_metric_calls bounds sampler evaluations, not reflection calls or tokens.
• RunConfig.max_llm_calls is enforced inside InvocationContext, but optimizer reflection bypasses Runner/InvocationContext; reusing it would mean manufacturing sessions and callback contexts for a non-agent workload.
• OTel spans alone are observational: they cannot reject an opaque optimizer at preflight or stop the next logical model invocation on budget exhaustion.
• Related but separate: GEPA optimizer crashes with KeyError on failed eval cases #6004 (score coverage / GEPA KeyError) and fix(eval): wire App.plugins / context-cache / resumability through adk eval #5534 (App plugin plumbing for adk eval, which leaves cli_optimize as follow-up). Neither addresses optimizer-owned call observability; this RFC does not change sampler/evaluator telemetry, which belongs to the supplied sampler/Runner.

Proposed contract (names directional)

Public shape

Existing callers pass nothing and remain observationally equivalent:

await optimizer.optimize(initial_agent, sampler)

Governed callers may supply a one-shot context:

await optimizer.optimize(
    initial_agent,
    sampler,
    run_context=run_context,
)

The context attaches to exactly one optimization. Reuse or concurrent attachment is rejected. An unsupported-optimizer preflight happens before attachment, so a typed unsupported rejection does not consume the context.

OptimizationRunContext — control and audit state

The context owns a concurrency-safe, run-local ledger of logical BaseLlm.generate_content_async() invocations. It records control-plane metadata only; no prompt or response content is retained.

Call admission is one atomic operation: check cancellation, reserve the logical-call slot, assign the sequence ID, and create the ledger entry. A preflight rejection is not a call event.

Per-call fields:

• invocation ID/sequence and extensible stage string (initial values: candidate_generation, reflection);
• requested model and returned model version;
• start/end time;
• call status: completed | provider_error | cancelled;
• sanitized provider error code / exception type when present;
• provider-reported prompt / output / reasoning / cached / tool-use / total tokens when present; and
• usage coverage: verified (provider supplied total_token_count) | partial (some counters, no authoritative total) | unreported (none). Missing values stay null — never coerced to zero.

The final immutable snapshot separately records run-level state:

• run status: completed | budget_exceeded | cancelled | failed;
• configured limits and usage-coverage summary;
• reported-token budget status: within_limit | exceeded | indeterminate;
• the invocation ID that triggered a budget stop, when applicable; and
• cancellation reason / terminal error metadata.

This separation preserves the fact that the charged call which crosses a reported-token limit completed successfully even though the run subsequently terminates for budget_exceeded.

Budgets and truthful coverage

Logical-call limits are hard. The slot is reserved atomically before the provider call begins, so exactly the configured number of optimizer-owned logical calls can start. Provider/transport retries inside one BaseLlm.generate_content_async() invocation are not claimed as separate calls.

Token limits apply to authoritative provider-reported totals. Token totals commit after a terminal response. If cumulative verified total_token_count crosses max_provider_reported_tokens, the completed call is committed first and the run terminates. This is deliberately not described as a hard bound on billing or physical tokens: if any call is partial or unreported, actual-token compliance is indeterminate even when all known totals are below the configured limit. Downstream policy may fail closed on that state.

Terminal behavior is configurable: on_budget_exceeded: "raise" | "return_partial" (default "raise"). Both modes stop scheduling immediately: a logical-call preflight rejection starts no call, while a reported-token overshoot stops after the triggering in-flight call settles.

• "raise" raises OptimizationBudgetExceeded(snapshot).
• "return_partial" returns only the best result accepted before termination. The caller-owned context snapshot remains the authoritative marker with run_status="budget_exceeded"; the existing OptimizerResult schema is unchanged.

For SimplePromptOptimizer, a partial result has overall_score=None unless a validation score had already completed; budget termination never schedules a final validation solely to populate the score. For GEPA, a partial result contains only the previously committed state/frontier, never a half-proposed candidate. An over-budget run can never carry run_status="completed".

Provider errors

With no context, existing optimizer behavior is unchanged. With a context, both raised provider exceptions and in-band LlmResponse.error_code responses commit the call as provider_error, preserve any reported usage, finalize the snapshot as failed, and terminate with a typed OptimizationProviderError(snapshot, error_code). A governed run cannot silently return success after an optimizer-owned provider error.

Cancellation and worker lifecycle

Cancellation is explicit and cooperative: thread-safe idempotent request_cancel(reason), observed before and after each optimizer-owned logical model invocation, before a GEPA sampler call is scheduled, immediately after an in-flight sampler future settles, and through a context-backed GEPA stopper.

In-flight provider/sampler operations are allowed to settle. Context-requested cancellation ends with OptimizationCancelledError(snapshot). Native task cancellation requests the same cooperative stop, shields and drains the GEPA executor future to its next boundary, then re-raises asyncio.CancelledError. A normal GEPA stopper return is post-checked so it cannot masquerade as success.

The no-orphan guarantee is conditional on the active provider/sampler operation eventually reaching a cancellation boundary; cooperative cancellation is not a time-bounded kill mechanism. A caller watchdog may request cooperative cancellation at a soft deadline and wait for a bounded grace period. If the grace period expires, the caller may abandon the wait, but then ADK cannot also guarantee that the still-blocked executor worker was drained. An absolute wall-clock bound requires provider/sampler timeouts or process isolation; native task cancellation alone cannot forcibly terminate an arbitrary blocked provider call or Python executor thread.

GEPA 0.1.x adaptation

GEPA checks stop callbacks between iterations, which is too late by itself to prevent candidate evaluation after a reflection call crosses the reported-token limit. The ADK adapter therefore uses this sequence:

1. commit the completed reflection call and set the context terminal request;
2. raise a private ADK iteration-abort sentinel from the reflection callable;
3. allow GEPA 0.1.x to convert that reflection failure into no proposal;
4. have the context-backed stopper end the run at the next loop boundary; and
5. post-check the context after gepa.optimize() returns, mapping it to raise or the previously committed partial frontier.

Optional-dependency types and the sentinel stay private to the adapter. A regression test must prove that no sampler call is scheduled after the triggering reflection invocation commits.

OptimizerCapabilities — conservative and granular by default

Non-abstract instance property on AgentOptimizer, with all support flags defaulting to false:

• accepts_run_context;
• optimizer_calls_observable;
• logical_call_limit_enforced;
• reported_token_limit_enforced;
• cooperative_cancellation; and
• sampler_usage_included (always false in this RFC — sampler/Runner accounting belongs to the supplied sampler).

Callers must omit the run_context keyword entirely when accepts_run_context is false; this preserves runtime compatibility with existing third-party optimizer implementations whose override still has the old two-argument signature.

SimplePromptOptimizer and GEPARootAgentPromptOptimizer opt into the supported capabilities. GEPARootAgentOptimizer accepts the keyword only to issue a typed unsupported error before attachment or work; its capabilities remain false until the 2.4 root/skill surface is instrumented as a fast-follow. Third-party optimizers inherit unsupported defaults untouched.

Delivery

The in-memory ledger and final immutable snapshot are the only delivery mechanisms in the first implementation. The snapshot is readable from the caller-owned context on success and failure, so a wrapper can persist the complete attempt in finally.

Explicit non-goals for ADK

Price catalogs, currency estimates, storage/BigQuery writes, promotion policy, holdout governance, incumbent registries, sampler/Runner usage accounting, and a hard wall-clock/thread-kill mechanism are downstream or separate concerns. This seam observes and stops optimizer-owned calls; nothing more.

Implementation plan — one RFC, two independently reviewable stacked PRs

1. PR 1 — contracts: OptimizationRunContext, immutable snapshots, typed terminal exceptions, granular OptimizerCapabilities, and optional keyword-only optimize(...) plumbing (new optimization/run_context.py + minimal touch to agent_optimizer.py and built-in signatures). Additive only; no optimizer enforcement behavior yet. Stack PR 2 immediately so the contract is reviewed with a concrete consumer rather than as dead API.
2. PR 2 — enforcement: instrument SimplePromptOptimizer and GEPARootAgentPromptOptimizer; implement atomic logical-call admission, normalized usage coverage, provider-error handling, the GEPA sentinel/stopper/post-check bridge, sampler-boundary cancellation checks, and executor shielding/drain; keep root/skill optimizer capabilities conservative.

The two PRs must collectively prove:

• the no-context path is observationally equivalent in call order, results, exceptions, and existing serialization;
• an old third-party optimizer override is never passed run_context after capability preflight;
• exact atomic logical-call limits, with provider retries explicitly out of scope;
• reported-token overshoot commits then terminates in both raise and return_partial modes;
• verified / partial / unreported classification, no zero coercion, and indeterminate compliance when coverage is incomplete;
• call status remains completed while the run status becomes budget_exceeded for the triggering successful call;
• partial Simple results never run post-stop validation and partial GEPA results expose only the prior committed frontier;
• raised and in-band provider errors preserve usage and cannot return governed success;
• context-requested and native cancellation cross the GEPA executor bridge, with the executor future shielded/drained once the in-flight operation reaches a boundary;
• no sampler work is scheduled after a reflection overshoot commits;
• context one-shot, pre-cancel, unsupported-preflight, reuse, concurrent-attachment, and atomic-call-admission behavior; and
• stopper/callback adaptation against both the declared minimum gepa==0.1.0 and the latest version admitted by ADK's unbounded gepa>=0.1 dependency. If that matrix is not sustainable, define and test an upper bound.

Questions for maintainers

1. Optional keyword-only context on optimize(...) (recommended for per-run state), or constructor injection for built-in optimizers?
2. OptimizerCapabilities as an instance property (recommended — support can depend on model/config), or class-level?
3. Is normalized-only usage (input/output/reasoning/cached/tool_use/total + coverage) the right experimental v1, leaving raw provider metadata out until a consumer proves need?
4. Is the caller-owned final snapshot an acceptable authoritative terminal marker for return_partial, preserving the existing OptimizerResult schema, or would maintainers prefer a context-aware outcome wrapper?

Related follow-ups, not acceptance dependencies

• GEPA optimizer crashes with KeyError on failed eval cases #6004 should independently define exact score coverage as a Sampler postcondition or replace the raw GEPA KeyError with a typed incomplete-result failure. No answer is required to accept or implement this RFC.
• fix(eval): wire App.plugins / context-cache / resumability through adk eval #5534 covers App plumbing for adk eval; cli_optimize App/plugin parity remains a separate follow-up.
• Instrumenting GEPARootAgentOptimizer is a fast-follow after the prompt-surface contract proves stable.

Definition of done

A caller can run either instrumented optimizer with an optional one-shot context; receive honest, granular capabilities for supported and unsupported optimizers; reconcile every ADK-owned logical model invocation with provider-reported usage when available; distinguish verified token-limit compliance from indeterminate coverage; terminate deterministically on logical-call or reported-token overshoot in either configured mode; stop cooperatively and drain GEPA's worker once the active provider/sampler operation reaches a boundary; and read the final immutable ledger snapshot on success or failure — with observationally equivalent behavior for existing no-context callers and no new required dependencies.

[caohy1988]

Fresh review — contract is sound after one precision pass

Verdict: no P0 architectural blocker. The proposed run-scoped seam is the right upstream boundary, all load-bearing claims check out against current ADK main, and the two-PR sequence matches ADK's focused-PR contribution rule. The fresh pass found four P1 contract gaps and six bounded P2 improvements; I applied all of them to the issue body so the thread and normative RFC now agree.

Verification performed

• The target commit 9d306f5 is still current main as of this review.
• AgentOptimizer.optimize() has no run-scoped argument today.
• SimplePromptOptimizer and both GEPA optimizers directly own reflection/candidate LLM calls and discard LlmResponse.usage_metadata.
• max_metric_calls limits GEPA sampler evaluations, not reflection calls or tokens.
• RunConfig.max_llm_calls is enforced through InvocationContext, which optimizer reflection bypasses.
• LlmResponse exposes both usage metadata and in-band error_code.
• fix(eval): wire App.plugins / context-cache / resumability through adk eval #5534 remains open and explicitly leaves cli_optimize App plumbing to a follow-up.
• GEPA optimizer crashes with KeyError on failed eval cases #6004 remains open with the missing-score GEPA KeyError framing.
• ADK declares gepa>=0.1; GEPA 0.1.0 provides stop_callbacks, but checks them at iteration boundaries and catches reflection/proposal exceptions into no proposal.

P1 findings resolved in the body

1. A token ceiling cannot be called hard when usage may be partial or unreported. The revised contract reserves hard enforcement for atomic logical-call admission. Token enforcement is now explicitly over authoritative provider-reported totals, with within_limit | exceeded | indeterminate coverage. Missing totals can never be interpreted as proof that actual usage stayed below the ceiling.

2. return_partial previously conflicted with zero-change result compatibility. Adding terminal_status to the existing Pydantic result would change normal schema/serialization, and SimplePromptOptimizer may not have a completed validation score at the stop boundary. The revised contract leaves OptimizerResult unchanged, makes the caller-owned snapshot authoritative, pins normal run status to completed, defines overall_score=None when validation never completed, and forbids post-stop validation.

3. GEPA's stopper is too late by itself. The body now pins the required sequence: commit reflection usage, set terminal state, raise a private iteration-abort sentinel, let GEPA convert the interrupted reflection to no proposal, stop at the next loop boundary, then post-check and map to raise/partial. A test must assert that no sampler call occurs after the overshoot commits.

4. The no-orphan worker guarantee was unconditional. Cooperative cancellation cannot forcibly terminate a provider call or executor thread that never returns. The revised DoD guarantees shielding/draining once the active operation reaches a boundary and explicitly names the tradeoff: after a caller's bounded grace period, abandoning the wait forfeits the drain guarantee; an absolute wall-clock bound requires provider/sampler timeouts or process isolation.

P2 improvements applied

• Split call status (completed | provider_error | cancelled) from run status (completed | budget_exceeded | cancelled | failed).
• Added atomic call-slot reservation so concurrency-safe admission cannot oversubscribe a logical-call limit.
• Made in-band LlmResponse.error_code behavior explicit: governed runs preserve usage and terminate with a typed provider error rather than silently succeeding.
• Expanded capabilities to distinguish run-context acceptance, call observability, logical-call enforcement, reported-token enforcement, cancellation, and sampler exclusion; old third-party overrides are protected by omitting the keyword entirely when capability preflight says unsupported.
• Moved GEPA optimizer crashes with KeyError on failed eval cases #6004 out of maintainer acceptance questions and into non-blocking related follow-up work.
• Expanded GEPA compatibility testing from the minimum alone to minimum plus latest admitted by the current unbounded dependency.

Non-blocking cleanup applied

• Replaced "byte-identical behavior" with observable equivalence.
• Made stage names initially defined but extensible.
• Named wall-clock/thread kill as an explicit non-goal.
• Reduced the maintainer questions to the four choices that actually affect this RFC.

Result

The RFC now makes three different claims separately and truthfully:

1. logical optimizer-call counts can be enforced exactly;
2. provider-reported token totals can be enforced reactively, with the triggering completed call committed; and
3. actual-token compliance is indeterminate whenever provider reporting is incomplete.

That distinction is the key governance property. With it, the RFC is ready for maintainer API feedback and then the stacked contracts/enforcement draft PRs. The BQAA-specific storage, pricing, promotion, and holdout responsibilities remain correctly outside ADK.

[caohy1988]

Draft implementation now up as the two-PR stack proposed above: #6358 (contracts: OptimizationRunContext, OptimizerCapabilities, terminal_status) and #6359 (enforcement in SimplePromptOptimizer + GEPARootAgentPromptOptimizer, typed rejection in GEPARootAgentOptimizer). Full tests/unittests/optimization/ suite: 53/53 passing on the stack. Both drafts — naming and API shape follow whatever direction maintainers set on the questions above.

[caohy1988]

Review of the precision pass — verified, and the draft stack is now aligned

Reviewed the fresh-review comment above against the revised issue body and the actual sources. All four P1s are correctly reflected in the body, and the load-bearing behavioral claim is empirically confirmed: GEPA's engine catches per-iteration exceptions and continues the loop (gepa/core/engine.py:588, with stop_callbacks checked at loop boundaries via _should_stop) — I reproduced this against the installed gepa matching ADK's declared floor. That claim was not academic: it exposed a real bug in the pre-revision draft #6359, where a typed budget error raised from the reflection callable was swallowed by the engine and sampler calls kept scheduling after the overshoot committed.

Both draft PRs are now updated to the revised contract:

• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 (contracts): call status split from run status (completed | budget_exceeded | cancelled | failed); token knob renamed to max_provider_reported_tokens with snapshot-level token_budget_status: within_limit | exceeded | indeterminate (any partial/unreported call ⇒ indeterminate — missing totals are never read as compliance); OptimizerCapabilities expanded to the six granular fields including accepts_run_context; terminal_status removed — OptimizerResult is unchanged and the caller-owned snapshot is authoritative; OptimizationProviderError added for governed in-band error_code termination.
• feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 (enforcement): the sentinel/stopper/post-check bridge exactly as pinned in the body — reflection commits usage, raises a private iteration-abort sentinel, GEPA converts it to no-proposal, the stopper observes any terminal run status at the next boundary, and the post-check maps the committed state to raise/return_partial (with GEPA's best-so-far candidates available in the partial path, an improvement over the pre-revision seed-only fallback). SimplePromptOptimizer terminates governed runs on in-band error_code with usage preserved, and return_partial returns overall_score=None with no post-stop validation.

The body-mandated regression test is implemented and green: no sampler call is scheduled after the overshoot commits, proven against a gepa stub with faithful engine semantics (iteration exceptions swallowed; stoppers at loop boundaries). Full tests/unittests/optimization/: 57/57 passing on the stack, including all pre-existing optimizer tests.

One scope note for maintainers reading the stack: the GEPA-compatibility test scenario in the body (minimum plus latest admitted gepa) is exercised in these drafts only against the installed release; pinning a CI matrix entry for both ends of the unbounded gepa>=0.1 range is left to the PR-review round, where the right CI mechanism is a maintainer call.

The thread, the normative body, and the draft implementation now agree. Awaiting maintainer direction on the four API questions.

[caohy1988]

Fresh cross-stack review: RFC #6357 + contracts #6358 + enforcement #6359

I re-reviewed the current issue body and both unchanged PR heads from first principles:

• Contracts PR feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 at f345eee1e2bdb171abd58ca0fde8c5a8359b0956: full contract review
• Enforcement PR feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 at c1f214ef22cf836c41da3f0c0fac42132d5730ae: full enforcement review

Fresh verdict: the RFC remains upstream-ready in direction, but the implementation stack is not yet aligned with the normative contract and should remain draft. The previous “all findings resolved / 57 of 57” conclusion overstates convergence: 57 official tests do pass, but a new RFC-derived adversarial set failed 16 of 16 cases.

What genuinely converged

The latest revisions correctly established several important foundations:

1. OptimizerResult remains unchanged and the snapshot is caller-owned.
2. Token compliance is truthful: within_limit | exceeded | indeterminate, with missing totals never treated as proof of compliance.
3. Call status and run status are distinct.
4. Capabilities are granular and conservative.
5. The GEPA budget sentinel + stop callback + post-check bridge is real and prevents continued reflection scheduling after a committed overshoot.
6. return_partial preserves GEPA's actual best-so-far candidates; Simple returns overall_score=None and skips final validation.
7. The unsupported root-agent optimizer rejects the context before work begins.

Those are meaningful improvements. The remaining gap is not the basic budget path; it is the terminal-state and cancellation contract around every other exit.

Load-bearing gaps found in the fresh pass

1. Provider failures can still return success through GEPA

The context records a provider-error call but does not make the run FAILED. GEPA catches per-iteration reflection exceptions and continues; because the stopper sees no terminal run status, both in-band and raised provider errors can be converted into eventual normal success. Raised reflection errors also lose usage-so-far, and a simultaneous token overshoot masks the provider failure.

The implementation needs atomic provider-failure commit, sanitized structured metadata, provider-error-first precedence, a GEPA sentinel conversion, and a final typed OptimizationProviderError mapping.

2. The authoritative final snapshot is frequently not final

Normal success is never marked COMPLETED. Several provider and native-cancellation paths remain None. Calls can start after a terminal token overshoot, and a late cancellation can overwrite an earlier budget terminal state. The snapshot also lacks configured limits, the triggering invocation ID, and terminal error metadata named by this RFC.

3. Advertised cancellation capability is not implemented at every pinned boundary

Simple performs baseline sampler work before checking a pre-cancelled context, does not check after candidate generation or around candidate/final scoring, and leaves native-cancelled model calls open. GEPA checks before sampler evaluation but not immediately after it. Its unconditional shield also changes legacy cancellation behavior when no context is supplied by draining the executor before propagating cancellation.

4. The governance ledger is mutable and can be corrupted across contexts

A handle from context A can be committed into context B; handle close is not atomic; attach(None) can reuse a one-shot context; budgets remain caller-mutable during a run; and the documented immutable snapshots/events are mutable Pydantic objects.

5. Contract and merge gates remain incomplete

ModelCallStage remains closed rather than extensible, negative budgets are accepted, and stale documentation still refers to the removed terminal_status. PR #6358 is not independently type-clean because it changes the abstract signature before updating built-ins. On the stacked head, Ruff, Pyink, Isort, and Mypy checks all identify new issues. The required real GEPA minimum-and-latest test matrix is absent; the critical tests use a fake GEPA loop and cover the budget path only.

Recommended precision additions to the normative body

The body does not need another redesign. I recommend adding these explicit clauses so the next implementation pass is mechanically reviewable:

Terminal transition ownership. Every optimizer that accepts a run context MUST finalize it exactly once before returning or propagating an exception: completed, budget_exceeded, cancelled, or failed. The first terminal transition wins. Any later transition attempt is a no-op or a typed misuse error, and begin_model_call() MUST reject a terminal context.

Provider-failure precedence. If a provider failure and token overshoot are observed on the same terminal response, the call usage and token-compliance state MUST both be committed, but the run terminates as failed and the caller receives OptimizationProviderError. Ledger error metadata MUST be structured and sanitized; raw prompts, responses, and arbitrary exception strings are excluded.

Native cancellation finalization. Native task cancellation MUST close or explicitly cancel any open call event, set the governed run to cancelled, preserve the final snapshot, and then re-raise asyncio.CancelledError. Cooperative cancellation checks occur both immediately before and immediately after every optimizer-owned model or sampler operation.

Absent-context parity. When run_context is None, control flow, task-cancellation propagation, executor draining behavior, results, and exceptions MUST remain observationally equivalent to the pre-seam optimizer behavior.

Ledger integrity. Budgets are copied and immutable after construction. Snapshots and events are immutable values. Call handles are bound to exactly one context, and admission/commit/close operations are atomic under that context's lock. Cross-context or concurrent duplicate commits cannot alter counters.

Definition-of-Done test additions

Before either PR is treated as aligned, the matrix should include:

• normal success produces COMPLETED for Simple and GEPA;
• pre-cancel causes zero model calls and zero sampler calls;
• cancellation during model work prevents the next sampler call;
• cancellation during sampler work prevents the next model/sampler call;
• native cancellation finalizes the event/snapshot and propagates promptly;
• no-context native cancellation preserves legacy behavior;
• in-band and raised provider errors terminate Simple and real GEPA with the typed error;
• provider error plus overshoot preserves usage/compliance while provider failure remains primary;
• no call is admitted after any terminal state;
• foreign-context handles and concurrent double-close cannot corrupt the ledger;
• budgets, snapshots, and event collections cannot be mutated;
• real GEPA minimum-supported and latest-supported versions run the budget, provider-error, and cancellation tests.

Verification record

• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 isolated optimization suite: 44 passed.
• Stacked feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 optimization suite: 57 passed.
• Fresh RFC-derived adversarial suite: 16 failed, 0 passed.
• Ruff: unused import.
• Pyink: 3 files need formatting.
• Isort: 2 files need sorting.
• Mypy comparison: feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 adds 3 errors over the reviewed main surface; feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 adds 15 more over feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358.

This does not invalidate the producer/consumer ownership boundary or the two-tier compatibility model. Those remain strong. It means the RFC should stay the normative target, while the draft PRs need another lifecycle-and-negative-path pass before the downstream flywheel or design doc can claim the upstream governance seam exists.

[caohy1988]

Cross-stack update following the adversarial review round: both drafts are rewritten to close every P1 — the contracts PR (#6358, head bd5b94d) now implements the full first-terminal-wins state machine, provider-failure precedence, and a frozen/ownership-validated ledger, with the review's five failing adversarial probes now first-class passing tests; the enforcement PR (#6359, head 0542043) implements every pinned cancellation boundary, the provider-error sentinel bridge (including the seed-evaluation escape path that runs outside GEPA's swallowing loop), governed-only shielding for absent-context parity, and exit-path finalization — verified by a Definition-of-Done test matrix including real-GEPA integration tests. Full optimization suite: 81/81. The recommended normative clauses (terminal transition ownership, provider-failure precedence, native cancellation finalization, absent-context parity, ledger integrity) are all now true in code; the min-plus-latest gepa CI matrix remains the one open gate, which needs maintainer CI configuration. Both PRs stay draft pending the API-shape questions.

[caohy1988]

Fresh cross-stack review after the adversarial rewrite

Reviewed the RFC and both unchanged new heads from first principles, including the complete diffs, current main, official tests, strict type/lint gates, real GEPA at both currently admitted versions, and a new set of contract-derived adversarial probes.

• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 contracts at bd5b94d: complete contract review
• feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 enforcement at 0542043: complete enforcement review

Fresh verdict: the rewrite resolves most of the previous round, but the “all normative clauses are now true in code” disposition is still premature. Keep both PRs draft. The GEPA budget/provider bridge is now strong; the remaining P1s are concentrated at boundary normalization, concurrent terminal propagation, absent-context equivalence, setup failure, and native-cancellation accounting.

What is verified resolved

1. Context attachment is one-shot even for owner=None.
2. Foreign handles cannot mutate another context; handle close is lock-atomic.
3. Budgets, snapshots, events, and event collections are immutable; negative limits are rejected.
4. Stages are extensible strings.
5. New call admission rejects a terminal context; a late observed cancellation does not overwrite an earlier terminal.
6. Provider failure is primary over simultaneous token overshoot while usage and compliance evidence survive.
7. GEPA's private sentinel + stopper + post-check bridge prevents sampler scheduling after a reflection overshoot.
8. Raised and in-band provider failures cannot become success through GEPA's per-iteration exception swallowing.
9. The seed-evaluation sentinel escape is mapped to typed outcomes.
10. Post-sampler cancellation, governed native GEPA cancellation/drain, and no-context native cancellation propagation work in independent probes.
11. Partial Simple and GEPA results preserve the intended score/frontier semantics.
12. Normal Simple and GEPA runs finalize COMPLETED.
13. Current code passes the complete optimization suite against both gepa==0.1.0 and the current latest gepa==0.1.1: 81/81 on each. A real-GEPA provider-failure probe also passes on both.
14. Both heads merge cleanly with current main; intervening main changes do not touch the optimization surface.

Remaining P1s

1. No-context GEPA behavior changed

#6359 inspects LlmResponse.error_code and returns early inside _generate() even when run_context=None. The old path continued consuming the stream. A reproduced two-response stream returns the later "legacy final" on the legacy logic but an empty prompt on this head.

This is a direct failure of the RFC's absent-context observational-equivalence requirement. The error-code branch must be governed-only.

2. Provider/control metadata is not normalized at the governance boundary

The contract claims sanitized identifiers but accepts and persists arbitrary values. A raised provider exception with numeric error_code=429 causes snapshot validation to throw Pydantic ValidationError, replacing the promised OptimizationProviderError and making later snapshots unreadable. Long/multiline strings are retained unchanged.

Usage conversion is also exception-unsafe: internal state is mutated before conversion completes, so invalid metadata can leave an event half-closed with no terminal state.

Normalization, bounding, and usage extraction must happen before atomic commit.

3. Terminal propagation has two concurrency gaps

• finalize_success() can mark a pending cancellation COMPLETED, producing cancel_requested=True with completed status.
• When two calls were already admitted and the first completion wins the budget terminal, the second completion commits and returns normally rather than propagating the existing terminal. Its caller may schedule downstream work after termination.

First-terminal-wins therefore protects the status field but does not yet protect all callers from continuing work. Success finalization must atomically observe pending cancellation/open calls; every already-in-flight completion must receive the first terminal before returning.

4. GEPA setup failures leave a consumed context in progress

#6359 attaches the context before importing GEPA, constructing the adapter/model, or reading sampler IDs, while its failure finalization begins only around the executor await. A reproduced sampler setup failure propagates normally but leaves run_status=None on a context that can no longer be reused.

Every failure after attachment must finalize, or fallible setup must occur before attachment.

5. Simple native cancellation drops usage already observed

If a streamed response reports usage and native cancellation arrives before the next response, Simple closes the call via finalize_cancelled() without committing last_usage. A reproduced seven-token call ends as cancelled with total_tokens=None, usage_coverage=unreported, and completed_calls=0.

Cancellation needs a call-level terminal commit carrying usage/model metadata before run finalization.

Smaller contract and delivery findings

• Every FAILED terminal is re-emitted as OptimizationProviderError, including adapter/sampler failures finalized as non-provider crashes.
• Cancelled events are terminal but do not increment completed_calls; counter semantics need to be made consistent or renamed.
• start_time / end_time are time.monotonic() process-relative values. Rename them accordingly or add UTC timestamps for the persisted audit surface.
• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 is not type/lint clean despite the disposition: Ruff finds one unused import and strict Mypy reports five new errors. Pyink and Isort pass. feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359's own Ruff/format/import-order checks pass but inherit the contract errors.
• PR bodies remain stale at 16/57 tests and do not describe the final heads.
• The repository lacks first-class GEPA tests for governed native cancellation, no-context cancellation parity, post-sampler cancellation, and real-GEPA provider errors. Independent probes pass those paths, but they are load-bearing and should live in the suite.
• Concurrent call-slot admission and concurrent attachment are lock-safe by inspection but are still missing the RFC-requested concurrent regression tests.

Recommended RFC precision additions

The ownership boundary remains correct. Add these clauses so the next pass is mechanically reviewable:

Boundary normalization. OptimizationRunContext is the enforcement boundary for ledger data. Error code/type and usage metadata are normalized and validated before internal state is mutated. Invalid metadata cannot make snapshot construction fail or leave a partially committed event. Error identifiers are bounded, structured, and exclude raw provider/request text.

Atomic success finalization. finalize_success() succeeds only when no model call remains open and no cancellation request is pending. A pending cancellation observed at finalization terminates as cancelled rather than producing completed plus cancel_requested=true.

Concurrent settlement. Calls admitted before another call wins the terminal race may settle and commit their evidence, but each such caller receives the already-committed first terminal outcome before returning. No post-model sampler work may be scheduled by a losing in-flight call.

Cancellation accounting. Native cancellation preserves any usage/model metadata already reported, closes the active call explicitly as cancelled, then finalizes and re-raises. Terminal-call counter semantics include cancelled calls or are named/documented to exclude them.

Post-attachment finalization. Every exception after successful context attachment produces a terminal snapshot, including dependency, adapter, sampler-discovery, and other setup failures.

Absent-context stream parity. Provider-error interpretation introduced by this seam is conditional on a supplied context. Without a context, stream consumption and result construction remain equivalent to the pre-seam implementation as well as preserving exception and cancellation behavior.

Definition-of-Done additions

• no-context multi-response stream containing an error_code produces the legacy result;
• numeric, oversized, multiline, and malformed error identifiers produce a stable sanitized typed failure;
• malformed/non-finite usage cannot partially mutate the ledger;
• pending cancellation racing success cannot become COMPLETED;
• two already-admitted calls both observe the first terminal after settlement;
• native cancellation after a partial usage response preserves that usage;
• failure at every fallible GEPA setup step leaves a terminal snapshot;
• non-provider failures are not reclassified as provider failures;
• cancelled-event counter semantics are pinned;
• real GEPA provider, budget, and cancellation tests run under both minimum and latest admitted versions in CI.

Verification record

• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 isolated optimization suite: 60 passed.
• feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 full optimization suite with GEPA 0.1.1: 81 passed.
• feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 full optimization suite with GEPA 0.1.0: 81 passed.
• New failure probes: 11 reproduced failures.
• New positive GEPA boundary/provider probes: 4 passed.
• Ruff: feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 fails one unused import; feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 passes.
• Pyink / Isort / diff whitespace: pass.
• Mypy: five new feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 errors, inherited by the stack.
• Current GitHub checks are lightweight header, CLA, and workflow scanning only; substantive unit/type matrices are not running on these fork drafts.

GEPA matrix disposition

This is narrower than the previous open-gate wording: compatibility itself is currently verified locally at both existing endpoints, 0.1.0 and 0.1.1. The remaining gap is durable CI automation for both endpoints, or an upper bound if that matrix becomes unsustainable. That remains a legitimate maintainer configuration decision, not a demonstrated code incompatibility.

The RFC does not need another architectural rewrite. One focused contract pass and one enforcement pass should close these gaps. Until then, downstream #318/#357 should continue treating the upstream governance floor as not yet released and the drafts as implementation evidence, not a completed seam.

[caohy1988]

Fresh cross-stack review — RFC #6357 + PRs #6358/#6359

I performed a new review against the unchanged current heads:

• Contracts PR feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358: bd5b94d
• Enforcement PR feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359: 0542043
• Current ADK main reviewed: 57e1ba6

Converged verdict: the architecture remains sound, but the implementation is not yet an authoritative governance boundary. Keep both PRs draft. The earlier disposition comment describes real improvements, especially the GEPA sentinel/stopper/post-check bridge, but it predates a later adversarial round. Because neither head has changed since that round, those findings remain open.

What is verified working

• The original GEPA swallowed-exception defect is addressed: a governed reflection failure is converted to a private sentinel, the stopper observes the terminal state at a loop boundary, and the post-check maps it back to a typed result.
• Provider failures preserve usage on the exercised GEPA paths.
• Budget overshoot stops subsequent sampler scheduling in the mandated regression scenario.
• Governed and ungoverned native-cancellation executor behavior passed independent tests.
• Official suites pass: feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 has 60 passing tests; the feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 stack has 81 passing tests on both GEPA 0.1.0 and 0.1.1.
• Both PRs merge cleanly against current main and pass git diff --check.

Remaining implementation blockers

1. Atomic ledger integrity (feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358): invalid usage or error metadata can fail after the record is marked closed, producing half-committed snapshots. Numeric provider codes can corrupt snapshot validation; raw error metadata is not sanitized at the context boundary.
2. Incomplete terminal concurrency (feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358): pending cancellation can be overwritten by finalize_success(), and an already-inflight call settling after another call terminates the run can return normally instead of observing the existing terminal.
3. Failure-type collapse (feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358): every generic FAILED terminal is reconstructed as OptimizationProviderError, including sampler/adapter failures.
4. No-context regression (feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359): the GEPA stream now returns early on an error-bearing chunk even without a context, changing legacy output and contradicting the compatibility claim.
5. Nonterminal attached lifecycle (feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359): failures between attach() and the executor try/except—including sampler-ID setup—leave run_status=None.
6. Cancellation evidence/finality (feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359): Simple native cancellation loses already-reported usage, and cancellation requested during final validation can end as COMPLETED.

Independent adversarial suite result: 11 failed / 4 passed. These are deterministic contract probes, not timing flakes.

Merge-quality gates

• Mypy comparison on the affected optimization surface:
  • current main: 24 existing errors;
  • feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358: 32 total (8 new);
  • feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 stack: 50 total (26 new).
• feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 adds one new Ruff error; feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 adds none relative to main.
• Current GitHub checks show CLA/header/change-detection only; the substantive unit, lint, type, and GEPA matrix are not visible merge gates.
• PR descriptions are stale: feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 reports 57 tests instead of 60; feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 reports 57 instead of 81.

Recommended closure sequence

1. Make PR feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358’s ledger validate-before-commit and make every terminal observable by every caller, including already-inflight settlements.
2. Give non-provider failures a truthful generic type and pin cancelled/closed call-counter semantics.
3. Restore byte-for-byte no-context stream behavior in feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359.
4. Extend feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359’s terminalization boundary from successful attach() through the complete optimizer lifecycle.
5. Preserve accumulated usage on native cancellation and add the missing post-validation cancellation check.
6. Promote the adversarial probes to repository tests, clear new Ruff/Mypy deltas, update both PR bodies, and add the GEPA minimum-plus-latest CI matrix.

The RFC’s producer/consumer boundary is still the right design: ADK should expose truthful model-call governance evidence, while BQAA owns sampler/evaluation accounting, pricing, promotion policy, and persistence. The implementation needs the fixes above so downstream policy can safely treat the caller-owned snapshot as authoritative rather than merely best-effort.

[caohy1988]

Round-2 closure: both heads rewritten to close every finding from the second adversarial round — contracts (#6358, 9ed15ac): atomic validate-before-commit settlement, complete terminal observation (including the in-flight-settlement-after-terminal gap and defensive finalize_success), truthful failure typing (OptimizationFailedError vs provider-reserved OptimizationProviderError), settled-call counter semantics, wall-clock event timestamps; enforcement (#6359, b9293de): governed-only in-band error handling restoring byte-level no-context stream parity, full attached-lifecycle terminalization from attach() onward, cancellation with usage evidence preserved, post-validation cancellation boundary, dual-layer provider-metadata normalization. Every reproduction from the round-2 reviews is now a first-class passing repository test. Suites: 70/70 isolated contracts, 96/96 stacked. Remaining open gates: the repo-config mypy comparison (not reproducible with default flags locally — will chase under the reviewer's invocation if deltas persist) and the min-plus-latest gepa CI matrix (maintainer-side). Both PRs remain draft pending the API-shape questions.

[caohy1988]

Round-3 cross-stack review — RFC #6357 + PRs #6358/#6359

Reviewed current heads from scratch:

• contracts feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358: 9ed15ac;
• enforcement feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359: b9293de;
• current ADK main: 57e1ba6.

Verdict: Round 2 correctly closes its entire finding list, but the implementation is not yet an authoritative governance boundary. Keep both PRs draft. The remaining architecture is sound; the fresh blockers are lifecycle ordering and repository delivery gates, not a need to rewrite the RFC.

Verified Round-2 closure

• Atomic validate-before-commit usage/error settlement works.
• NaN/inf/negative counters cannot half-close a record.
• Numeric provider error codes are normalized.
• Concurrent in-flight settlement receives the first terminal.
• Generic and provider failures retain truthful types.
• Settled-call counting and wall-clock timestamps work.
• No-context GEPA stream behavior is restored.
• Failures during attached GEPA setup finalize the context.
• Native Simple cancellation preserves observed usage.
• Cancellation during final validation cannot become COMPLETED.
• Suites pass: 70/70 isolated contracts; 96/96 stacked on GEPA 0.1.0 and 0.1.1.

Fresh P1 blockers

1. Success finalization does not enforce ledger completeness. finalize_success() accepts open calls and silently returns for an existing raise-mode terminal.
2. GEPA has a post-admission exception gap. A failure while constructing/scheduling a reflection request is converted by GEPA to no proposal; the run later returns COMPLETED with an open call (started_calls=1, completed_calls=0).
3. Pre-cancel is observed after fallible discovery. Both optimizers touch sampler discovery first; if discovery fails, FAILED wins over the already-pending cancellation.
4. Provider metadata normalization is incomplete. An invalid returned model version can be committed successfully and make every subsequent snapshot fail validation.

Delivery blockers

• Repository-config Mypy comparison is reproducible: main 47 errors, feat(optimization): run-scoped usage, budget, and cancellation contracts (RFC #6357, PR 1/2) #6358 52 (+5), stacked feat(optimization): enforce run-context usage, budgets, and cancellation in the prompt optimizers (RFC #6357, PR 2/2) #6359 72 (+25).
• The new public run_context.py module violates ADK's private-by-default module rule and is not exported through optimization.__init__ / __all__.
• Fractional usage such as 1.9 is truncated to 1 and marked verified.
• RFC-requested concurrent attachment/call-admission tests remain absent.
• The durable GEPA minimum-plus-latest matrix remains open, and real-GEPA provider/native-cancel coverage is not yet in the repository suite.

Suggested implementation order

1. Contracts: make finalize_success() return only for existing COMPLETED, re-raise every other terminal, and fail closed on open records.
2. Contracts: normalize every event field before mutation, including returned model metadata; reject fractional token counters.
3. Enforcement: observe cancellation immediately after attachment, before all sampler/model/dependency setup.
4. Enforcement: make the complete post-admission reflection interval exception-safe. Local setup/scheduling failures need generic call settlement plus OptimizationFailedError, then the private GEPA sentinel.
5. GEPA: correct the swallow-location documentation and explicitly pin raise_on_exception=True for governed calls.
6. Repository integration: rename the implementation module private, export the experimental API deliberately, clear all normalized Mypy deltas, and add the missing concurrency and real-GEPA matrix tests.

RFC precision text worth adding

Success completeness. finalize_success() may commit completed only when every admitted model call has settled, no cancellation is pending, and no earlier terminal exists. An existing non-completed terminal is re-raised, never silently acknowledged.

Post-admission exception completeness. From successful call admission until terminal call settlement, every exception path must settle the call and commit a truthful run terminal. Optimizer/request/scheduling failures are generic optimization failures, not provider failures. A governed optimizer can never return success with an open call.

Pre-cancel ordering. After successful attachment, pending cancellation is observed before dependency initialization, model construction, sampler discovery, sampler evaluation, or model-call admission. A competing setup failure cannot replace an already-pending cancellation.

Event normalization. Every field copied into the durable event, including requested/returned model metadata, is normalized and validated before internal state is mutated. Snapshot construction cannot fail because of previously committed provider metadata.

The ownership boundary remains right: ADK owns optimizer-call governance evidence; BQAA owns sampler accounting, prices, persistence, and promotion policy. These focused fixes make the snapshot safe enough for that downstream contract.