docs(stovepipe): propose request event history

doc/rfc/index.md

@@ -29,6 +29,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting
 - [Build stage](stovepipe/steps/build.md) - Trigger-only stage and Stovepipe's URI-based BuildRunner contract
 - [Buildsignal stage](stovepipe/steps/buildsignal.md) - Build polling, terminal status persistence, and the handoff to record
 - [Record stage](stovepipe/steps/record.md) - Immutable validation facts keyed by `(queue, uri, project)`, monotonic last-green bookmark advancement and ref promotion, and the deferred hook-event and analyze handoffs
+- [Request Event History](stovepipe/request-event-history.md) - Append-only request lifecycle model, durable source context, idempotent storage, and reliable write and repair paths
 
 ## Runway
 

doc/rfc/stovepipe/request-event-history.md

@@ -0,0 +1,290 @@
+# Stovepipe Request Event History
+
+## Summary
+
+Stovepipe retains an append-only history for each validation request. It records every durable `Request.State` transition plus three asynchronous milestones needed to explain those transitions and the public verdict:
+
+- `build_triggered`;
+- `build_finished`;
+- `validation_fact_recorded`.
+
+The model deliberately follows SubmitQueue's distinction between statuses describing where a request is and events describing important activity that does not move it. It remains a bounded request-lifecycle history rather than a generic event bus or an audit of every correlated operation.
+
+`Request`, `Build`, and `ValidationFact` remain the operational sources of truth. History is a required audit/read model derived from successful durable operations; it does not replace their optimistic-locking state machines.
+
+The separate [Stovepipe Request History API](request-history-api.md) defines public lookup, projection, materialization, ordering, pagination, and retention semantics.
+
+Related documents are [SubmitQueue Gateway Request History APIs](../submitqueue/history-api.md), [Stovepipe Workflow](workflow.md), [Process](steps/process.md), [Build](steps/build.md), [Buildsignal](steps/buildsignal.md), and [Record](steps/record.md).
+
+## Problem and Scope
+
+Stovepipe currently retains only the latest mutable state of a validation request and build. A `ValidationFact` retains the immutable verdict for a commit, but none of these records explains how a request reached that verdict.
+
+State changes alone are insufficient. A request remains processing while a build is triggered and finishes, and its terminal state becomes durable before record creates the fact that establishes green or broken. These milestones must remain visible without inventing request states for work occurring underneath the request's position.
+
+An occurrence belongs in request history only when it is either:
+
+1. a durable `Request.State` transition; or
+2. an asynchronous durable milestone required to explain the next request transition or public verdict.
+
+The following remain with their owning entities, metrics, or structured logs:
+
+- Queue latest-request and last-green bookmarks;
+- source-control promotion;
+- project analysis and project facts;
+- hooks and downstream notifications;
+- build-slot claims, waits, and releases;
+- queue handoffs, delivery attempts, holds, nacks, and visibility timeouts;
+- transient dependency errors and DLQ mechanics.
+
+## Goals
+
+1. Retain every request-state transition with its successful request version.
+2. Retain the build and fact milestones needed to explain request progress and verdict creation.
+3. Make writes idempotent under redelivery, restart, and uncertain failure.
+4. Preserve enough source context to repair history after a source write succeeds.
+5. Keep storage queue-scoped and compatible with key/value, SQL, document, remote, and in-memory backends.
+6. Preserve persist-before-publish and controller-owned version arithmetic.
+
+## Non-goals
+
+1. Replacing the Request, Build, or ValidationFact stores with event-sourced projections.
+2. Recording every durable operation correlated with a request.
+3. Reconstructing transport behavior or unchanged build polls.
+4. Providing global ordering or cross-request search.
+5. Using history as the source of durable greenness; only `ValidationFact` establishes green or broken.
+6. Publishing internal history entries directly as cross-domain hook events.
+
+## Existing SubmitQueue Pattern
+
+SubmitQueue retains `RequestLog` separately from its mutable orchestrator `Request`. Each entry is either a customer-facing status or an event that happened while the request remained at a status. Status entries may carry the orchestrator request version for reconciliation; events carry no request version and never compete to become current status.
+
+Its status vocabulary covers accepted, validating, batched, speculating, landing, landed, error, and cancellation. Its deliberately small event vocabulary covers building, built, waiting, and invalidated rather than every queue hop, retry, storage update, or side effect.
+
+Producers use stable occurrence values in message IDs to deduplicate repeated publication of one logical occurrence. The gateway appends a flat `RequestLog` with type, status or event, timestamp, request version, error, and metadata. Consumer retries may still retain duplicate log rows because the stored record itself has no idempotency key.
+
+Stovepipe adopts the status/event split, bounded vocabulary, queue scoping, and per-occurrence identity. It strengthens retained idempotency with a stable entry ID so retries converge on one stored occurrence.
+
+Stovepipe writes history directly because every stage belongs to the same service and resolves the same queue-scoped storage aggregate. SubmitQueue's cross-service log topic is unnecessary unless history ownership later crosses a service boundary.
+
+## History Entity
+
+The retained unit is `entity.RequestHistoryEntry`:
+
+```go
+type RequestHistoryEntry struct {
+    // ID is the stable identity of one logical occurrence within the request. It is opaque, stable across redelivery, and never derived from time or randomness.
+    ID string
+    // Queue is the logical queue containing the request and scopes RequestID.
+    Queue string
+    // RequestID identifies the request whose history contains this entry.
+    RequestID string
+    // TimestampMs is the durable occurrence time in Unix milliseconds.
+    TimestampMs int64
+    // State is the durable request state recorded by a state entry. It is unset on an event entry.
+    State RequestState
+    // Event identifies the occurrence recorded by an event entry. It is unset on a state entry.
+    Event RequestEvent
+    // RequestVersion is the successful request version recorded by a state entry. It is zero on an event entry.
+    RequestVersion int32
+    // SupersededByRequestID identifies the newer request that caused a superseded outcome.
+    SupersededByRequestID string
+    // BuildID identifies the build associated with the occurrence, or is empty when no build applies.
+    BuildID string
+    // OutcomeReason is the domain reason for a terminal request state and never names a transport mechanism.
+    OutcomeReason RequestOutcomeReason
+    // FactDegree is the recorded health degree on validation_fact_recorded. The event kind distinguishes degree zero from absence.
+    FactDegree float64
+}
+```
+
+Enums are strings with unknown sentinels, and the entity has no storage or transport dependency. Constructors enforce exactly one of state and event and validate the required context for that kind. The populated field identifies the entry kind without a separate discriminator.
+
+The sparse explicit-column shape follows SubmitQueue's `RequestLog`, except Stovepipe omits SubmitQueue's redundant `Type` column. It is easier to inspect, validate, and project than an opaque payload while the vocabulary remains bounded. New fields belong only to an approved request state or explanatory event.
+
+Immutable Request context such as URI, build strategy, and base URI remains on `Request` and is returned once by the read API rather than copied into individual history entries. Build status and version remain on `Build`; the triggered and finished event kinds plus the terminal Request state describe the lifecycle without duplicating Build snapshots. Diagnostic error codes remain in structured logs until a concrete public vocabulary is required.
+
+## Vocabularies
+
+### State entries
+
+| State | Required retained context |
+|---|---|
+| `accepted` | Request version 1 |
+| `processing` | Request version |
+| `superseded` | Superseding request ID, outcome reason, and request version |
+| `succeeded` | Outcome reason, winning build ID, and request version |
+| `failed` | Outcome reason, optional build ID, and request version |
+| `cancelled` | Outcome reason, build ID, and request version |
+
+### Event entries
+
+| Event | Meaning | Required retained context |
+|---|---|---|
+| `build_triggered` | A runner accepted a build and its Build row became durable. | Build ID and creation time |
+| `build_finished` | The Build first reached a write-once terminal status. | Build ID and status-change time |
+| `validation_fact_recorded` | The immutable whole-repository fact became durable. | Degree and fact creation time |
+
+Build running and unchanged polls are not retained. Trigger and terminal result explain the request outcome without turning polling into an unbounded history. Project facts remain outside the initial vocabulary.
+
+### Evolution
+
+State and event strings never change meaning or get reused. A new event must satisfy the scope rule and update required-field validation, storage mapping, public projection, and compatibility tests.
+
+Schema evolution is additive: nullable or defaulted columns and tolerant readers land before writers populate a new field or kind. Incompatible reinterpretation requires a new field or event rather than changing stored meaning.
+
+### Outcome reasons
+
+Terminal entries retain domain reasons rather than transport mechanisms. Initial reasons include:
+
+- `build_succeeded`;
+- `build_failed`;
+- `build_cancelled`;
+- `processing_failed`;
+- `build_polling_exhausted`;
+- `validation_timeout`;
+- `superseded_by_newer_head`.
+
+## Stable IDs and Idempotency
+
+`stovepipe/core/requesthistory.Recorder` constructs opaque IDs from durable identities:
+
+| Entry | Stable identity inputs |
+|---|---|
+| Request state transition | Request ID and request version |
+| Build triggered | Request ID, event kind, and build ID |
+| Build finished | Request ID, event kind, and build ID |
+| Validation fact recorded | Request ID, event kind, and whole-repository fact identity |
+
+The recorder calls `RequestHistoryStore.Create`. If the ID already exists, it loads the stored entry and compares every semantic field. Identical content is idempotent success; conflicting content is an internal consistency error, and the stored entry is never overwritten.
+
+## Storage Contract
+
+The queue-scoped store is append-only and key/value-shaped:
+
+```go
+type RequestHistoryCursor struct {
+    // TimestampMs is the timestamp of the last entry returned by the previous page.
+    TimestampMs int64
+    // EntryID is the ID of the last entry returned by the previous page.
+    EntryID string
+}
+
+type RequestHistoryQuery struct {
+    // Cursor is the exclusive continuation boundary when HasCursor is true.
+    Cursor RequestHistoryCursor
+    // HasCursor distinguishes the first page from a cursor whose values are zero.
+    HasCursor bool
+    // Limit is the positive maximum number of entries returned.
+    Limit int
+}
+
+type RequestHistoryStore interface {
+    Create(ctx context.Context, entry entity.RequestHistoryEntry) error
+    Get(ctx context.Context, requestID, entryID string) (entity.RequestHistoryEntry, error)
+    List(ctx context.Context, requestID string, query RequestHistoryQuery) ([]entity.RequestHistoryEntry, error)
+}
+```
+
+`Create` returns `ErrAlreadyExists` for an existing ID. `List` returns at most `query.Limit` entries ordered by `(timestamp_ms ASC, entry_id ASC)`. When `HasCursor` is true, only entries strictly after `(Cursor.TimestampMs, Cursor.EntryID)` are eligible. An empty result means that no entries remain at that boundary; the controller loads `Request` separately to distinguish an unknown request from an existing request with no retained entries.
+
+The controller owns the opaque, versioned public page token and translates it to this typed cursor. The storage contract never parses a wire token. This follows SubmitQueue's queue-list split between controller-owned token encoding and a backend-neutral keyset query.
+
+The contract exposes no unbounded list, update, delete, query-by-state, query-by-event, query-by-build, cross-request search, or cross-queue enumeration. A key/value backend can represent the request as a partition and entries as ordered immutable child records, and can satisfy the cursor with a bounded range scan within that partition.
+
+The initial MySQL table uses:
+
+```sql
+PRIMARY KEY (queue, request_id, entry_id)
+KEY idx_request_history_order (queue, request_id, timestamp_ms, entry_id)
+```
+
+The ordering index covers the request partition, exclusive keyset cursor, stable order, and limit in one bounded scan rather than introducing query-by-attribute capability.
+
+## Write and Repair Protocol
+
+History durability is part of completing a pipeline transition. The source write succeeds first, the required history entry is retained second, and a dependent handoff is published only after history creation or identical-existing reconciliation succeeds.
+
+For a Request transition, the controller:
+
+1. builds an immutable updated copy with transition context and `StateChangedAtMs`;
+2. computes `newVersion = oldVersion + 1`;
+3. calls `RequestStore.Update(updated, oldVersion, newVersion)`;
+4. assigns the in-memory version only after the store succeeds;
+5. asks the recorder to create history from durable Request data;
+6. publishes the downstream handoff.
+
+Request creation, Build changes, and fact creation use the same source-write, history-write, dependent-publish ordering. A history outage can leave a source update visible, but it cannot allow dependent processing to move past an unrecorded transition.
+
+### Controller ownership
+
+| Owner | Source operation and retained history | Retry repair |
+|---|---|---|
+| Ingest | Create accepted Request, then retain accepted. | An existing Request ensures accepted before process publication. |
+| Process | CAS to superseded or processing, then retain that state. | An existing state is reconstructed from Request context before ack or build publication. |
+| Build | Create Build after runner acceptance, then retain `build_triggered`. | An identical existing Build ensures the event before buildsignal publication. |
+| Buildsignal | Persist terminal Build and retain `build_finished`; CAS the Request outcome and retain its terminal state. | Existing terminal Build and Request outcome each ensure their own entry before record publication. |
+| Record | Create or verify the whole-repository fact, then retain `validation_fact_recorded`. | An identical fact owned by the Request ensures the event before bookmark or promotion work. |
+| Reconciler | CAS an unrecoverable non-terminal Request to failed, then retain failed. | An existing terminal Request is repaired from its persisted outcome without relabeling it. |
+
+Build running and unchanged polls create no entry. A failed runner trigger that creates no Build creates no event. Cancelled and superseded requests create no validation fact.
+
+No controller infers an outcome reason from `RequestStateFailed` alone, invents a broken fact, or records DLQ as the domain reason.
+
+### Repair trigger requirement
+
+Every writer retains a retry trigger until history succeeds. A queue stage retries its delivery, or its DLQ reconciler repairs history and republishes the original handoff. A history or publish failure after a source transition never forces a different outcome.
+
+Stages without a suitable DLQ reconciler add one or use a subscription policy that cannot discard the only repair trigger. Ingest relies on caller retry of the same `(queue, URI)` to complete an accepted Request whose history or process publication failed.
+
+This is rollout work, not deferred cleanup: mandatory history without a durable repair trigger could turn a history-store outage into silently stranded pipeline work.
+
+## Rollout and Retention
+
+Rollout therefore:
+
+1. deploys source timestamp and provenance fields, history storage, recorder, and readers;
+2. enables writers and verifies every repair path stage by stage;
+3. enables the public API after every writer and repair path is active.
+
+The service does not fabricate prior state transitions or emit a synthetic snapshot. Requests accepted before history writers are active are not exposed through the history API and are not backfilled.
+
+Initial history retention matches Request and request-URI mapping retention. The API RFC owns the resulting selector guarantees.
+
+## Failure Semantics
+
+- History-store unavailability is retryable and prevents dependent publication.
+- Invalid entry construction is non-retryable.
+- `ErrAlreadyExists` followed by identical content is success.
+- `ErrAlreadyExists` followed by conflicting content is a non-overwriting consistency error and operator alert.
+- A history failure cannot roll back its source write; retry or reconciliation must converge the missing entry.
+
+Identifiers, outcome reasons, and reasonable per-request build counts have explicit limits. Raw dependency responses, credentials, tokens, stack traces, and unrestricted error messages are never retained.
+
+## Verification and Observability
+
+Contract tests cover required-field validation, stable IDs, idempotent create/reload, conflict detection, queue binding, deterministic ordering, first and subsequent page boundaries, equal-timestamp cursor behavior, limits, and empty final pages.
+
+Writer tests cover source success followed by history failure, redelivery with history absent or present, CAS loss, conflicting terminal writers, downstream publish failure, stable timestamps, and controller-owned version arithmetic. End-to-end tests cover successful, failed, cancelled, superseded, and fail-closed paths plus idempotent redelivery.
+
+Tests reconstruct the latest Request state from state entries by request version and compare it with `RequestStore.Get`. They separately verify that only a durable fact produces green or broken.
+
+The recorder reports create, identical-existing, conflict, validation failure, and storage failure counters tagged only by bounded state or event. IDs, URIs, and errors remain structured log fields rather than metric tags. Alerts cover sustained repair gaps and content conflicts.
+
+## Alternatives Considered
+
+### Record request state changes only
+
+Rejected because build activity occurs while the request remains processing, and terminal Request outcome precedes the fact establishing green or broken.
+
+### Record every correlated durable operation
+
+Rejected because bookmarks, promotion, project facts, hooks, and operational bookkeeping have different owners and audiences. Correlation alone does not make an operation part of request lifecycle history.
+
+### Use a generic versioned envelope
+
+Rejected for six request states and three explanatory events. Explicit fields are simpler to inspect, constrain, test, and expose. A future migration should be justified by actual vocabulary growth.
+
+### Copy SubmitQueue's log topic
+
+Rejected while one Stovepipe service owns every writer and the queue-scoped storage aggregate. A topic becomes useful if ownership later crosses a service boundary.