Repository navigation
docs(drafts): design committed session history and execution kernel - #2179
Conversation
|
I see where you're coming from and I like the direction as far as I can tell. The initial implementation did have some limitations and we are definitely starting to see the wrinkles. My main ask is that these constructs stay behind pluggable storage drivers, with a portable history format so we can move sessions between deployment types without changing their semantics. In regards to these being breaking changes, I'm fine with that. We're in alpha stage, so if there's a time to break things, it's now. I did take a look at what other harnesses do. OpenCode and OpenHands provide useful implementation references. In OpenCode, durable events and registered projections are written in the same database transaction, then readers are notified after commit. Readers catch up from persisted sequence numbers. A notification tells you to check storage. OpenHands V1 separates streaming progress from persisted, sequenced events. Its reconnect path subscribes, buffers live events, replays through a captured boundary, and deduplicates the buffer by sequence. I mentioned ActiveGraph the other day and it seems to me like it's a good reference for commit-before-projection and provenance, but as you identified earlier, it's not a great fit for all of our needs. Its event log is authoritative and ordinary reconstruction does not rerun behaviors. Both are worth preserving here. Its operator documentation explicitly excludes a distributed runtime, though, and expected-head checks are not the same contract as checking the head, ownership epoch, and unexpired claim atomically. If we were to take some learnings from this, I'd borrow its causal references and inspection ideas without bringing in its graph ontology, which is consistent with the draft's stated scope. Run IDs, call IDs, parent/child relationships, and fork positions should give us useful answers to “where did this come from?” without another domain model for beliefs and reactive edges. |
|
Before we commit to anything, I'll do a survey of prior art for sure. If we go this direction, there will be some big refactors and it'll be painful. I do plan to keep pluggable implementations and, I think, this points the way for a more comprehensive idea of an extension to the main engine/loop in a way we don't have now. Just like CRDs can extend k8s, extensions here should be able to write events and manage long running tasks, etc. I see some parallels there. The first test would be to move subagents out of the core and make the recursive usage of the engine go through an extension. That feels like a really big test of the extension points. I'm not happy with the doc as it is so will keep working on it and will probably write an implementation plan too that breaks down the modules and interfaces and starts to map out how we get there. |
|
@JAORMX I did a prior-art pass and added a companion doc to this PR. There are useful ideas to draw from the other harnesses, but nothing that fundamentally changes the direction. I didn’t find a complete equivalent to the graceful server-side handoff during upgrade that we’re aiming toward. OpenHands is the closest operational reference, but the implementations surveyed don’t establish the whole coordinated drain, durable park, and fenced takeover flow. That remains a later phase of our proposal, rather than part of the first cut. There are useful precedents for canonical event histories and for extension-owned persisted records, though those capabilities differ across systems. In particular, storing custom events doesn’t necessarily mean their continuation state can be reconstructed or safely transferred to another server. |
Capture the session-history design independently of the earlier implementation work, with a focused Modelith companion and shared-client steer semantics. Co-authored-by: Mecatl <noreply@mecatl.dev>
Use sentence-per-line formatting, split dense paragraphs, and highlight domain concepts without changing the design semantics. Co-authored-by: Mecatl <noreply@mecatl.dev>
Define run lifecycle, vocabulary, derived views, and component responsibilities against current code. Preserve the discovery-only prior-art shortlist while deeper research waits on search credentials. Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
Capture short-PR staging, cancellation and memory-only result settlement, bounded optional request receipts, and single-provider session catalogs. Split the catalog and command protocol model from session history, document commit costs, and extend the prior-art comparisons. Co-authored-by: Mecatl <noreply@mecatl.dev>
Define lifecycle phases, commit gates, recovery evidence, and cancellation transitions. Add durable-execution research and cache-stable reload requirements while preserving the narrow in-harness design scope. Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
Add a short overview, shared vocabulary, and six independently reviewable topics. Retain detailed contracts as lookup references and preserve open design decisions. Co-authored-by: Mecatl <noreply@mecatl.dev>
Explain safe pause points and same-run continuation for compatible-replica maintenance. Distinguish automatic client reconnection from transport migration, preserve crash and first-cutover limits, and add the later handoff acceptance scenario. Co-authored-by: Mecatl <noreply@mecatl.dev>
Clarify execution, replay, and command terminology across the session-history design and regenerated domain model. Record the naming convention in the shared vocabulary and a narrowly scoped project rule. Co-authored-by: Mecatl <noreply@mecatl.dev>
Explain execution checks, tool-specific adapters, output disclosure, and subsequent model input in plain language before the detailed authorization rules. Co-authored-by: Mecatl <noreply@mecatl.dev>
Explain actors, problems, and examples before detailed constraints. Preserve existing requirements, qualifications, links, diagrams, and open decisions in the review packets. Co-authored-by: Mecatl <noreply@mecatl.dev>
5eae6b4 to
ce95d1f
Compare
Add implementation considerations for stable snapshots, explicit cloning and borrowing contracts, and candidate enforceable representation boundaries without changing the core design. Co-authored-by: Mecatl <noreply@mecatl.dev>
Capture extraction considerations from identity work while preserving execution authority and the existing committed-history design. Treat storage observations as diagnostic input rather than a canonical decomposition. Co-authored-by: Mecatl <noreply@mecatl.dev>
Describe delegation and learning as composed capabilities using different integration seams, with optional agent tools and client commands. Preserve the narrower durable-record protocol and existing delivery scope. Co-authored-by: Mecatl <noreply@mecatl.dev>
Capture source-based capability extraction candidates, kernel-retained invariants, and staged preparation before storage integration. Preserve existing cutover scope and separate Go, stored-data, and wire compatibility assessments. Co-authored-by: Mecatl <noreply@mecatl.dev>
Record coupled loop/coordinator/session vocabulary, stable invocation ownership, admission and cancellation semantics, and serialized history acceptance. Define the isolated first kernel harness while preserving concurrent read-only tools and later runtime integration. Co-authored-by: Mecatl <noreply@mecatl.dev>
First committed-history kernel sliceSlug: session-history-kernel Planning proposal only; no implementation or cutover is authorized by this document. The plan is shared as a conversation comment on design PR #2179 at Joe's request, not checked into the repository. That PR remains documentation-only. Implementation starts only on a separate request. Joe confirmed the package map, first-slice scope and session-lifetime ownership on 2026-10-29. Independent critique and targeted correction recheck completed with no remaining identified blockers; integrated-baseline prerequisites still apply. Goal and decisionsBuild the actual new kernel behind a small composition API. Demonstrate a complete model → tools → model cycle in an isolated, growing harness; connect a real LLM immediately after the first scripted cycle works, before completing the broader fault matrix. The early phases are an additive, isolated experiment alongside the current runtime. Production entry points keep using the existing engine. Shared changes should be small and justified: build/architecture/API checks and genuinely useful primitives, not feature extraction or runtime activation. We can change direction or discard the experiment without requiring project-wide adoption. Production cutover is a later explicit decision; permission for disruptive changes is not an instruction to disrupt the existing project during the experiment. Joe's architectural priority is tight, focused packages with compiler-enforced representation boundaries. Do not rely on file conventions inside a large package. Use a new minimal session aggregate, not an embedded/wrapped legacy Feature continuity is also not a constraint. Joe explicitly accepts temporarily breaking or removing major functionality, including learning, with only a coarse direction to rebuild it as an extension. No complete replacement design, bridge implementation, dual-runtime feature parity, or restoration date is required before making a clean boundary. Do not bend the kernel to keep every legacy capability working. This is permission to simplify where the slice needs it, not a task to proactively dismantle unrelated features. Identify deliberately unavailable functionality plainly; missing functionality must not masquerade as successful execution. Safety, truthful outcomes, accepted-fact preservation and the selected slice's contracts remain requirements. Settled scope:
Excluded: production storage or fencing, server/client wiring or protocol changes, runtime-selection flags, legacy migration, full capability extraction, extension loader, delegation/learning/compaction implementation, production authorization/hooks/disclosure workflows, remote jobs, and same-run crash resume. The fixture has only explicit harmless tools and fixed inputs; omitted production controls are not implemented or qualified. Memory acceptance proves protocol behavior, not physical durability. Grounding and baseline
Implementation prerequisite: choose a main-reachable integrated baseline after the identity stack, recheck these seams, and capture the implementation branch's actual start commit. If APIs materially differ, update this plan rather than silently substituting a different architecture. No pins to invented versions or copied identity-branch files. Package map and ownershipAll new paths below are proposed. The kernel remains in the existing engine Go module. Call authoritative committed facts records, not events, to distinguish them from the existing activity events being replaced.
Import direction (additional direct inward imports are allowed only where used):
Drivers remain existing externally composed model/tool implementations. The first slice installs no universal driver framework, process registry, or private capability bundle. Tool execution receives detached action data plus separately live context/environment capabilities. If a current tool contract cannot express an observation, kernel-owned metadata supplements it without changing every existing tool signature. Use a narrow session-owned preparation/application contract: validate against accepted state, establish commitment, then reflect accepted facts. Do not mutate published accepted state speculatively, clone the entire legacy aggregate per append, or introduce generic transactions/rollback. Reuse/extract an appropriate canonical prefix validator where that is clean. If doing so would force legacy aggregate coupling or an awkward public API, copy the small algorithm into the new owner and add shared behavioral/parity cases for canonical occurrence/replay invariants. Preserve one canonical identity meaning, not necessarily one implementation of every helper. Classify source API changes for any shared seam separately; do not add compatibility wrappers merely to avoid a justified break. Mechanical boundaries and coexistence guidance
Material contractsCommitted records and accepted state. Start with the records required for run/input acceptance, effective model input, completed model response, effective tool intent, canonical tool result, cancellation/terminal settlement. Version the record vocabulary; no production serialization format is promised. Model request identity/turn anchor is scoped by existing session incarnation and run identity plus an explicit turn index; it is not provider identity or commit position. Transition IDs identify proposed facts; positions are assigned by acceptance. Reconstruction checks the whole ordered prefix and never invokes a provider, tool, hook, or adapter to recreate missing facts. Unknown/malformed records or broken pairing fail closed. Capture the complete effective logical model input used by this slice before each provider call: system layers, actual projected messages and private replay, tool definitions, model selector and the continuation basis. Identity transformations have explicit no-rewrite evidence; omitted hooks are not fabricated transformation success. The submitted provider request must be detached from the committed input. Preserve argument byte representation, bindings, phase and opaque replay; do not round-trip raw arguments through map normalization or assume JSON serialization preserves bytes. A live smoke that loses required provider replay is a failed milestone. Storage seam. Append carries history incarnation, expected accepted head, stable batch/transition identities and detached records. Acceptance returns authoritative assigned positions/head; rejection definitively means this attempt did not establish those facts; unresolved means acceptance is not known. Identity is scoped to the history incarnation and bound to the exact ordered batch contents and original expected-head basis. Repeating the same accepted batch returns its original acceptance even after the head advances: duplicate recognition precedes ordinary expected-head conflict handling. Reusing a batch or transition identity with different contents fails explicitly, never attaches to the old acceptance. Resolve/lookup addresses that exact batch identity and expected-head basis, returning accepted, definitively not accepted, or still unresolved. A missing lookup result alone is not proof of rejection while an append may still complete. Do not automatically resubmit under a new identity. Read returns an owned contiguous accepted prefix through a selected head for reconstruction. Backend durability/claim/fencing implementations remain deferred, not falsely represented by the volatile adapter. Coordination. At most one unresolved batch per history. Verify a candidate against accepted state; append without holding a control lock; reflect accepted facts before the next append. Bounded ready facts may group only when no gate separates them. Already-admitted work can return while a write waits. Cancellation closes admission promptly in the same local ordering domain as final admission; it does not wait for append I/O or reserve another append slot. Late intent acceptance cannot undo earlier cancellation. A stalled history does not stall another history. Execution and outcomes. Commit effective input before inference; completed model response before tool preparation; effective intent before final tool admission; canonical permitted results before next dependent model request. Canonical invocation IDs are minted once at accepted model-response conversion, not taken from provider tokens. Invalid/failed/cancelled model streams do not admit tentative tool calls. No implicit model/tool retry after ambiguous effects. For each invocation, separate closed kernel observations (admitted, interruption requested, returned) from produced content. No arbitrary error strings become safe durable metadata; a context error proves neither nonexecution nor cessation. Handed-off input and returned output have independently owned nested storage. The fixture accepts only its explicitly supported processing path; unsupported authorization/disclosure continuations fail closed instead of being invoked as ordinary tools. Cancellation before admission prevents invocation even if intent later resolves accepted. After admission, signal interruption and process returned evidence normally, including successful completion by a non-interruptible tool. Use an injected settlement deadline. A bounded local stop must distinguish pending commitment/unresolved external activity from a committed terminal outcome; do not publish fictitious closure. After the settlement boundary, fence late advancement by invocation/run identity. No successor run starts until required predecessor pairing/settlement is accepted. Reconstructing an active prefix establishes interrupted/unknown work, never automatic replay of effects; any recovery close-out is a new accepted fact, not invented historical data. Session lifetime and obligations. The kernel owns the session lifetime, which is longer than any run. Its per-history handle retains coordination and outstanding obligations after run termination or a bounded wait returns. The coordinator owns append resolution and session-level settlement; the loop reports invocation returns without requiring a live model loop. A run ending does not dispose of its tool/environment resources or erase pending work. Collect and settle obligations when evidence arrives, or explicitly mark them abandoned under their original identities with safe uncertainty evidence. Abandonment closes eligibility for stale run advancement; it is not proof that a tool stopped, a cleanup action succeeded, resources were released, or an append was rejected. Where abandonment is a canonical decision it must be accepted through the same commit path; unavailable storage leaves that decision pending, not confirmed. Late authoritative append acceptance is always reconciled/reflected even after run termination or an abandonment decision, without reopening execution. Late tool output cannot replace an already accepted canonical close-out. The handle keeps live invocation references and a bounded collection path until they return, or reports still-outstanding activity if they cannot settle; no generic supervisor or driver lifecycle framework is required in this slice. Caller-supplied store/provider/environment resources must remain valid while obligations can still use them. Bounded shutdown stops admission and reports unsettled/abandoned obligations separately from actual quiescence. Safe resource teardown requires actual quiescence or substantiated release evidence; logical abandonment alone does not establish it. The command's volatile store makes process exit an explicitly non-durable boundary, not proof of settled external effects. Read batching follows actual Dependency-ordered implementation tasksThese are small reviewable outcomes, not permission to implement or a requirement for one PR per package. Agree coherent PR boundaries when implementation begins; never merge nonbuilding consumers.
Observable acceptance and test schedulesUse channel/barrier handshakes and controlled clocks, not sleep-based timing claims. Exercise the real facade, coordinator and aggregate; storage doubles replace dependencies only.
Compatibility and verificationAPI compatibility: the expected change is additive public engine Go packages, but additive-only is not a design constraint. Prefer a justified clean break over compatibility machinery. Regenerate API snapshots and classify additions/breaks under the current pre-v1 engine policy; identify and update affected callers together. Existing production behavior need not be redesigned for this isolated slice. If a retained provider/tool contract obstructs a necessary boundary, bring the specific change and consumer impact back for agreement rather than silently weakening the boundary or introducing an adapter framework. The new record protocol is not a legacy snapshot/event migration and promises no interchangeability. No remote API behavior, protobuf, feature identifier or harness major changes are expected in this slice; assess separately and obtain specific approval before a discovered wire break. Later production cutover remains a separate incompatible slice. Focused iterations: Integrated-change checks owned by the implementing agent: run Use tech-writer when authoring the implementation's READMEs and run Review completed implementation with panel-review against the recorded implementation start commit, not moving main. Incremental fixes use the last reviewed candidate; rebase requires separate authorization. This plan is shared on #2179 for design discussion, not as implementation in that PR. If an implementation PR is later requested, preserve its latest full publishable plan in one identifiable conversation comment with privacy/identity/readback checks. Keep comment URLs/IDs in ignored publication metadata rather than in the published plan body. Humans alone merge. No automatic commits, push, release, or live monitoring. Open items and risks
|
Overview
Design committed session history and a focused execution kernel for server-owned sessions that clients can leave, reload, and share without losing the facts needed to understand or safely continue the work.
Today, continuation snapshots, optional activity history, and live streams have different guarantees. The proposal makes private committed records authoritative, with continuation state, retained history, and authorized client views derived from accepted facts. Required commits gate execution so recovery can distinguish established outcomes from work whose effects are unknown without automatically repeating it.
This is a documentation-only discussion draft. It contains no kernel implementation, production activation, or approval to cut over. The first-slice implementation plan is shared in a PR comment, rather than checked in as a permanent backlog.
Review guide
For the proposed kernel boundaries and near-term work, start with the execution-kernel note and the first-slice plan. The tracked ownership note covers stable snapshots and detached data; the components note distinguishes capability extensions from durable-record handlers and discusses learning and compaction boundaries. Concrete proposed Go paths and the task breakdown live in the plan comment.
For the wider session design, read the overview, then choose a review packet. There is no required six-packet reading sequence.
The reading map links the shared vocabulary, architecture and contract references, YAML-backed domain models, and optional prior-art research. Packet boundaries are review topics, not code packages or implementation phases. This continues the design independently of the earlier implementation attempt in #2078.
Kernel and ownership direction
The kernel is coupled execution, coordination, and session-state machinery. It is not a separate service, OS sandbox, or custom-coordinator plugin framework. The proposed package boundaries give each responsibility a narrow contract and private representation:
session.Sessionaggregate.Use records for authoritative committed facts, distinct from the existing activity events. Check a transition against accepted state, establish acceptance, then reflect it in memory. Initially allow one unresolved append batch per history. Storage waits must not block prompt cancellation or prevent already-admitted work from returning. Concurrent read-only tools retain independent progress: a ready result can commit while a slow sibling is still executing, subject to all prerequisite gates and mutation barriers.
Detached invocation data and independently owned output prevent later mutation from changing the accepted action or result. Execution metadata stays separate from response content. Cancellation before final admission prevents invocation; cancellation afterward requests interruption and proves neither cessation nor absence of effects. Run termination does not end the kernel's responsibility to collect, settle, or explicitly abandon outstanding obligations, and abandonment does not prove resource release or append rejection.
Trusted capability drivers consume specific extension points. Delegation can own durable records and operation state; learning can consume eligible completion/evidence and memory/skill interfaces without becoming a second history authority. Broad private capability bundles and a universal extension interface are not the goal.
First executable slice
Start with an isolated, growing harness around the real new kernel. The early phases are an additive side experiment alongside the existing runtime, with no major production changes required. Production entry points keep using the current engine; the experiment can change direction without requiring project-wide adoption.
The plan proposes
engine/kernelwith public history/storage contracts, focused internal loop/coordinator/session packages, andcmd/mecakernelfor executable composition. README and AGENTS guidance will distinguish this work from the current runtime. Tight package boundaries and mechanical dependency checks are intentional; internal file conventions alone are insufficient.The planned sequence is:
Implementation uses an integrated main baseline after the canonical identity stack beginning with #2404. Reuse canonical identities and replay semantics, but copy small structures or algorithms where reuse would force a bad boundary. The experiment does not require production storage, server/client integration, full authorization/hooks/disclosure machinery, or capability extraction. In-memory acceptance tests prove protocol behavior, not physical durability or distributed fencing.
This is an early project. A later clean integration may deliberately break APIs or temporarily remove major functionality, including learning, without first implementing a replacement. That flexibility is not an instruction to disrupt the current runtime during the experiment or to compromise safety and truthful outcomes.
Broader design and later integration
The wider proposal covers shared API controls and retry receipts, catalogs and lifecycle authority, complete logical model-input capture and private provider replay, retained history through compaction, authorized reload/feed continuity, forks, deletion, and backend-enforced ownership fencing. Potentially sensitive unreleased results remain memory-only; loss is explicit and does not justify rerunning a tool. Existing guardrail decision semantics remain unchanged.
Production integration and cutover are separate decisions. Any runtime-selection mechanism belongs there, not in the initial harness. The proposed first cutover includes required commit gates, trustworthy reload, shared controls, registered in-process reconstruction for
Subagent,Parallel, andTeam, and explicit interrupted-run recovery. Legacy bytes remain untouched, with no migration, fallback, or mixed-version rollout.Compatible-replica handoff that preserves the same active run and delegation tree is a later phase. Autonomous continuation, cross-run background work, shared catalogs, and search also remain later capabilities. Whether predecessor-result closure and successor admission must share one atomic commit remains open; planning readiness is scoped to the slice being built.
Compatibility and validation
This PR changes design documentation and its terminology rule, not shipped Go, wire, or storage behavior. The plan's proposed APIs and formats require their own compatibility assessment during implementation; justified breaking changes are acceptable but must identify affected callers and upgrade implications.
The first-slice plan received independent design critique and a targeted correction recheck with no remaining identified blockers. That is planning evidence, not runtime qualification. Earlier work recorded documentation and model lint/render checks; those checks were not rerun for this title/description update. No production test-suite, race-suite, live-provider, durability, or fencing pass is claimed by this PR description.