V2-04: close remaining Memory Governor production-activation blockers

[MikeRoss27]

Summary

• Fix WriteCoordinator::build panicking when a shared MemoryDomain refuses a second engine's bootstrap memtable reservation — now a typed CoordinatorError::AdmissionRefused, never a panic.
• Add the ADR-072 §B memtable-rotation-specific RSS/stress oracle (40 real generations, pinned readers, flush, release, teardown) that GOV-RSS01-03 left open.
• Close the documented ADR-072 rotation-charge shutdown race: owner_commands now holds the coordinator alive across execute() + charge attach instead of re-upgrading a Weak afterward, which could silently lose a rotated generation's charge.
• Wire and prove EmergencyHeadroom operationally (GOV-U32): a real store rotates/flushes repeatedly under a domain deliberately undersized for two generations at once — genuine backpressure at the hard cap, bounded by a timeout so a self-deadlock fails the test instead of hanging.
• Expose memory governor activation as genuinely public API: basemyai::storage::{open_with_key_governed, MemoryGovernorDomain, MemoryGovernorConfig} — a free function (not an inherent method, since NativeMemoryStore's method surface is frozen pending V2-03 per xtask's PRODUCT_API_OWNERS guard) returning a basemyai-owned wrapper (never naming a basemyai_engine::* type, per ADR-068 §10). Proven end to end by an out-of-crate integration test using only public API.

Includes the pre-existing P1 governor gate-closure round (already reviewed/merged into dev locally) that this branch was built on top of.

Note on merge strategy

This branch's history contains merge commits from the earlier P1 round (fc46d15, 9f10e23, d1f7c79, cf2e5fc, a5245ce) — the repo's branch ruleset on dev forbids merge commits landing on dev directly, so this PR should be merged via squash or rebase, not "create a merge commit".

Test plan

• cargo fmt --check
• cargo xtask check (includes the ADR-068 v2-layering gate)
• cargo xtask test
• cargo test -p basemyai-engine --features test-util --lib (618/618)
• cargo test -p basemyai --features test-util --lib (158/158)
• cargo xtask test-crash-consistency (13/13)

🤖 Generated with Claude Code

[prelint]

Memory governor first appears in the public API before its flush-path enforcement is complete

Product decisions in this change

1. A caller can open a store under a shared `MemoryGovernorDomain` through a new public free function, with a 13-field config struct that includes a headroom field derivable only from a workspace-private function.

The public entry point is real and needed: external callers gain a governed activation path today. Three concerns reduce confidence.

First, the 13-field MemoryGovernorConfig exposes DRR scheduler internals verbatim. A caller who only wants to cap total memory to N bytes must still set reserved_foreground_slots, reserved_maintenance_slots, foreground_weight, and max_admission_waiters. These belong to the engine's scheduling model, not the user's task.

Second, emergency_flush_headroom_bytes is a public field whose doc comment says to derive it using estimate_emergency_flush_headroom_bytes. That function lives in basemyai_engine, which is workspace-private. A user of basemyai cannot call it. The integration test sets the field to zero. A real caller following the only available public example gets no flush headroom, and the field has no practical effect today because the flush path does not yet consume governed reservations.

Third, the API is self-described as likely throwaway: the doc comment says adding a method is the right final design. Publishing a known-transitional entry point creates migration cost for any external adopter.

Option	What it gives users	What it costs	Effort to change later
Current: 13-field struct, free function	Full DRR control; works today	Users must understand DRR; emergency_flush_headroom_bytes is un-derivable from basemyai	High: every field is breaking API surface
Builder with only budget_bytes required	Works without DRR knowledge	Less per-class tuning	Low: add fields later non-breakingly
Wait for V2-03 and ship as a method	One stable API surface	Governor activation deferred	None: designed correctly once

2. A typed storage error, not a panic, is returned when a shared `MemoryGovernorDomain` cannot host a second store's bootstrap reservation.

Before this fix, sharing a domain across two stores could panic the second store's constructor with no recovery path. A typed error lets the caller handle the refusal and either retry with a larger domain or surface the resource constraint to the operator. This is especially important because the domain is explicitly designed to be shared between multiple stores.

3. The production activation gate stays explicitly closed even though a public governed API ships: write admission is governed, but the flush working set is not.

Keeping the gate closed is the right call. The flush path does not consume governed reservations, so emergency_flush_headroom_bytes acts as a static budget reduction rather than a real per-flush enforcement mechanism. The concern is user expectation: a caller who invokes open_with_key_governed reasonably believes they have opened a fully governed store. The public doc comment does not explain that flush-path memory is ungoverned. A caller who sets a tight budget expecting full enforcement may find their process RSS exceeds the limit during heavy writes that trigger flushes, with no error or backpressure from the governor.

4. `EmergencyHeadroom` is statically carved out at domain construction, reducing the visible budget, rather than dynamically reserved by each flush.

Static reservation is the correct interim design: it prevents the flush from being starved at the hard cap by shrinking the budget the rest of the system can claim. The concern is that the formula to size this field (estimate_emergency_flush_headroom_bytes) is not accessible from basemyai, and setting it to zero — as the integration test does — means no headroom protection and allows the flush to allocate beyond the domain limit without the governor detecting it. The derivation is thorough and the grounding test is strong, but the gap between formula derivation and actual enforcement is not communicated to the caller.

5. The per-write admission permit releases on every coordinator outcome, because it covers only transient WAL and intent bytes, never memtable residency.

This is a correctness fix with no behavior change: the code already released the permit on every outcome, and the prior RETAIN entries in the ADR table were documentation residue from a rejected design option. Memtable residency is accounted for separately by the rotation mechanism's upfront generation reservation. The correction makes the documentation match the code.

6. `CB-T4` is removed from the pre-activation requirements for the §B rotation mechanism, requalified as a future phase 2 design question about MG-17 wiring.

reserve_blocking — the admission path §B uses for rotation — cannot structurally return AdmissionWouldBlock. Its contract is to block until capacity is available or the domain closes. Testing that maybe_flush tolerates AdmissionWouldBlock would test a code path that §B can never reach. The requalification correctly identifies that the underlying concern belongs to a separate, unscoped chantier with its own design decision. Removing a structurally inapplicable requirement from the gate list is correct.

7. ADR acceptance is split into two distinct levels: architecture accepted (the invariants are correct) versus implementation production-ready (the gate tests pass).

The original single-status wording made it impossible for a reader of ADR-071 to determine whether the governor was safe to activate in production. The split removes that ambiguity and provides a precise, actionable answer: build on this architecture, but do not activate it in production yet. This has direct product impact: teams using the ADR as a readiness signal now have a clear answer rather than having to read footnotes.

8. The governed activation entry point is a free function rather than a method on `NativeMemoryStore`, to avoid touching the method surface frozen pending V2-03.

The reasoning is technically correct: adding a method to a type whose construction protocol V2-03 will reshape creates API surface the team expects to supersede. The concern is the user experience. A caller sees NativeMemoryStore::open_with_key and NativeMemoryStore::open_encrypted as methods, but open_with_key_governed as a free function. That asymmetry is confusing without reading the doc comment. If this function ships in a versioned release, any downstream adopter must migrate when the method API lands. Publishing a known-transitional API is a cost that compounds with adoption volume.

Option	What it gives users	What it costs	Effort to change later
Free function (current)	Working governed activation today	Inconsistent with method-based API	Medium: add method later, potentially deprecate function
Method on frozen surface	Consistent API	Would require V2-03 first or a gate exemption	Not viable now
Wait for V2-03	Clean API from day one	Governor activation deferred to V2-03 milestone	None: designed correctly once

9. `MemoryGovernorDomain` wraps the engine governor as a `basemyai`-owned type, ensuring no `basemyai_engine::*` type appears in the public API.

This directly enforces ADR-068 §10 and the layering invariant from ADR V2-00. The wrapper is lightweight, the field conversion from MemoryGovernorConfig to the engine's GovernorConfig is mechanical, and the boundary prevents callers from depending on engine-internal types. If basemyai_engine is restructured, the public MemoryGovernorDomain shape is not forced to change.

10. An in-crate test is labeled `GOV-U32` while the gate documentation in the same PR states that the gate condition this label names remains open.

The test proves a real and valuable property: writes complete without deadlock under a domain tight enough to force genuine backpressure at every rotation, and data is preserved across rotations. That is a necessary condition for production activation. It is not the condition the ADR defines as GOV-U32. The ADR gate requires the flush path itself to consume a governed reservation — so the governor enforces the budget during the flush working set, not only during write admission. The current test proves forward progress through the rotation mechanism's reserve_blocking, which is a different mechanism. Naming this test GOV-U32 creates a collision: a reader who sees the PR description ("wired and proved") and the gate documentation ("remains open") in the same diff cannot determine which claim is authoritative. The right decision is to give this test a name that describes what it actually proves — for example, write_loop_makes_forward_progress_at_the_governed_rotation_hard_cap — and reserve GOV-U32 for the gate-closing test once phase 2 flush wiring exists.

Open questions

• Is basemyai_engine ever intended to be a public, versioned crate? If yes, estimate_emergency_flush_headroom_bytes should be re-exported through basemyai so callers can populate emergency_flush_headroom_bytes correctly. If no, the field should either be removed from the public config or replaced with a type that enforces a correctly derived value.

• What is the intended user experience when a governed store's process RSS exceeds the domain budget during a flush? Does the product intend to warn callers explicitly that flush-path memory is ungoverned today, or is that considered an internal implementation detail?

• Does any planned external consumer of open_with_key_governed know this API is expected to change after V2-03? If the function ships in a published crate version, semver forces either a major version bump or a deprecation path when the method API lands.

• What is the intended migration path for MemoryGovernorConfig's 13-field struct? If V2-03 reshapes the coordinator construction protocol, does the struct's shape also change, and is any field today considered stable API surface?

• Is the GOV-U32 label on the in-crate test in native_store/mod.rs an intentional redefinition of that gate condition, or a naming error that should be corrected before any gate list adopts it?

Recommendation

Ship with changes
The core engineering work — the typed error, the test suite, the ADR corrections, the MemoryGovernorDomain wrapper — is sound and should ship. Two changes are needed before the public surfaces land: the in-crate test should be renamed from GOV-U32 to a name that describes what it actually proves, so the team does not conflate a necessary condition with the sufficient gate-closing condition; and the open_with_key_governed doc comment should explicitly warn that flush-path memory is ungoverned until phase 2 wiring is complete, so callers who activate governance today hold accurate expectations about what is and is not enforced.

[For0day]

Good code