Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/rfc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting
- [Buildsignal stage](stovepipe/steps/buildsignal.md) - Build polling, terminal status persistence, and the handoff to record
- [Record stage](stovepipe/steps/record.md) - Immutable validation facts keyed by `(queue, uri, project)`, monotonic last-green bookmark advancement and ref promotion, and the deferred hook-event and analyze handoffs
- [Request Event History](stovepipe/request-event-history.md) - Append-only request lifecycle model, durable source context, idempotent storage, and reliable write and repair paths
- [Request History API](stovepipe/request-history-api.md) - Queue-scoped request-ID and URI lookup, public projection, materialization decision, ordering, and completeness

## Runway

Expand Down
184 changes: 184 additions & 0 deletions doc/rfc/stovepipe/request-history-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Stovepipe Request History API

## Summary

Stovepipe exposes retained request history through queue-scoped point lookups by request ID and exact URI. Both selectors return the same ordered, cursor-paginated history.

The API reads the append-only model defined by [Stovepipe Request Event History](request-event-history.md) directly. It does not replay history into current state or introduce a second persisted history projection. Current commit status remains a separate read concern derived from operational entities rather than request history.

## API Coverage

The API supports the same selectors as [SubmitQueue Gateway Request History APIs](../submitqueue/history-api.md):

1. request ID, for callers retaining the receipt returned by `Ingest`;
2. exact URI, for callers starting from a commit identity.

Both methods require a queue because Stovepipe storage and authorization are queue-scoped. A selector belonging to another queue is not found rather than resolved across shards.

SubmitQueue's URI method returns several histories because the same change may be submitted repeatedly. Stovepipe ingest permanently deduplicates `(queue, URI)` to one request, so its URI method returns exactly one history. Relaxing that invariant later would require a new plural method rather than changing this method's cardinality in place.

## Representative Contract

The final protobuf receives a separate compatibility review before implementation. Its representative shape is:

```proto
message GetRequestHistoryByRequestIDRequest {
string queue = 1;
string request_id = 2;
int32 page_size = 3;
string page_token = 4;
}

message GetRequestHistoryByURIRequest {
string queue = 1;
string uri = 2;
int32 page_size = 3;
string page_token = 4;
}

message ValidationResult {
double degree = 1;
}

message HistoryEntry {
string id = 1;
int64 timestamp_ms = 2;
oneof occurrence {
string request_state = 3;
string event = 4;
}
string superseded_by_request_id = 5;
string build_id = 6;
string outcome_reason = 7;
ValidationResult result = 8;
}

message RequestHistory {
string request_id = 1;
string uri = 2;
string build_strategy = 3;
string base_uri = 4;
repeated HistoryEntry entries = 5;
}

message GetRequestHistoryByRequestIDResponse {
RequestHistory history = 1;
string next_page_token = 2;
}

message GetRequestHistoryByURIResponse {
RequestHistory history = 1;
string next_page_token = 2;
}

service Stovepipe {
rpc GetRequestHistoryByRequestID(GetRequestHistoryByRequestIDRequest) returns (GetRequestHistoryByRequestIDResponse) {}
rpc GetRequestHistoryByURI(GetRequestHistoryByURIRequest) returns (GetRequestHistoryByURIResponse) {}
}
```

Entry IDs and page tokens are opaque. Clients may compare entry IDs and pass page tokens back to the method that issued them but never parse either format. Request-state and event values are strings so clients can tolerate additive vocabulary changes. Request version remains persisted for internal transition ordering but is not part of the public entry, matching SubmitQueue's API boundary.

## Selection Flow

Request-ID lookup validates the queue, ID, page size, and optional token; loads the queue's `Request` to validate the selector and supply immutable wrapper context; and lists one bounded page of its `RequestHistoryStore` entries.

URI lookup resolves the existing `RequestURIStore` primary key and delegates to the same paginated request-ID path. It does not scan history by an entry's `URI` field and requires no new storage index. A missing mapping is not found; a mapping whose Request is missing is an internal consistency error.

The request-URI mapping must be repaired and retained with its Request and history. Otherwise URI lookup could lose coverage while request-ID lookup still succeeds. The loaded Request supplies URI, build strategy, and base URI once on the `RequestHistory` wrapper; these immutable values are not duplicated on every entry.

## Public Projection

Each stored `RequestHistoryEntry` in the selected page maps to exactly one public `HistoryEntry`. The API does not deduplicate, coalesce, or infer missing occurrences.

State entries set `request_state` and preserve each durable transition. The public vocabulary is:

| Public request state | Meaning |
|---|---|
| `accepted` | The request was durably admitted. |
| `processing` | The request was admitted to validation with a selected strategy. |
| `superseded` | A newer request replaced this request before validation completed. |
| `succeeded` | The request's validation work completed successfully. |
| `failed` | The request's validation work failed or could not continue. |
| `cancelled` | The request's validation work was cancelled. |

These strings intentionally match the current domain states one-to-one, but they are a stable public history vocabulary: an internal refactor cannot rename or reinterpret an existing wire value. In particular, `succeeded` and `failed` remain distinct rather than collapsing into a derived snapshot phase such as `finalizing`.

Build events set `event` and `build_id`. `validation_fact_recorded` supplies `ValidationResult`; result presence distinguishes a recorded degree of zero from no verdict. The `occurrence` oneof makes state and event mutually exclusive without a redundant type field. A terminal Request state entry never substitutes for the fact event.

Raw dependency errors, credentials, stack traces, and unrestricted metadata are not exposed. `outcome_reason` uses a bounded public vocabulary.

## Materialization

SubmitQueue's `PersistLog` operation performs two jobs after receiving a log message:

1. append the `RequestLog` row that is itself returned as history;
2. consider status rows for a gateway-owned `RequestSummary`, resolve out-of-order candidates using request version, terminal precedence, and timestamp, then update URI and queue-list projections.

Event rows remain in SubmitQueue history but never participate in current-status materialization. The materializer exists because the gateway owns public reads but cannot read the orchestrator's mutable Request store.

Stovepipe has no equivalent ownership gap. The same service owns the queue-scoped `Request`, `ValidationFact`, request-URI mapping, and history stores. Operational reads use their owning entities, while request history reads retained entries directly.

Stovepipe therefore does not add `RequestSummary`, replay history to determine current state, or materialize another history table. The controller performs only an in-memory wire projection from stored entries to protobuf messages. This avoids a second winner-selection algorithm competing with Request CAS state.

## Ordering and Consistency

Entries are returned across pages by `(timestamp_ms ASC, entry_id ASC)`. Timestamps are display order, not conflict resolution. Persisted Request versions provide internal causal ordering for state transitions, while Build identity and write-once terminal status ensure one triggered and one finished occurrence per build.

History may briefly lag a source entity between the source write and history creation. The pipeline blocks its dependent handoff during that window, and retry or reconciliation repairs the missing entry. The read API never fabricates an occurrence from current state.

Request-ID and URI lookup return the same stored entries, ordering, and pagination semantics. A later repair may insert an older occurrence into its correct chronological position.

## Pagination and Retention

The public shape follows SubmitQueue's queue `List` convention: an empty token selects the first page, zero page size selects the server default, and the response returns an opaque `next_page_token` that is empty on the last page. The initial default is 50 entries and the maximum is 200. The controller requests one more entry than the effective page size, returns only the requested page, and issues a token only when the extra entry proves that another page exists.

Pagination uses the immutable keyset `(timestamp_ms ASC, entry_id ASC)`. The versioned token represents the last returned ordering tuple and is bound to the original selector kind and value, queue, and resolved request ID. The binding does not prescribe whether those values are embedded or fingerprinted in the token. Reusing a token with another method, queue, request ID, or URI is invalid. Page size is not bound into the token, so a caller may change it between pages within the server maximum. The controller decodes the public token into the storage contract's typed exclusive cursor; storage implementations never parse wire tokens.

Tokens are traversal cursors rather than snapshot handles. A new occurrence ordered after the cursor can appear on a later page. A repair that inserts an older occurrence at or before an already-consumed cursor may be observed only by starting a fresh traversal. This is the paginated form of the API's existing eventual-consistency guarantee: a token prevents duplicates from stable retained entries but does not freeze history while pipeline writers and repair are active.

History, `Request`, and request-URI mapping retention must support the same advertised lookup period. The API does not promise a lifetime longer than every record required by its selector. The initial rollout exposes only requests accepted after all history writers and repair paths are active; older requests are outside the lookup period and are not backfilled.

## Errors and Authorization

- Empty queue or selector is invalid.
- Negative page sizes, page sizes above the server maximum, malformed tokens, unsupported token versions, and tokens reused with another query are invalid.
- An unknown request ID or URI, including one scoped to the wrong queue, is not found.
- A URI mapping whose Request is missing and a request within the advertised history lookup period whose required history is missing are internal consistency errors.
- Retryable storage failures are unavailable; context cancellation and deadline errors retain their canonical codes.

Authorization follows the same queue policy as other Stovepipe reads. Possession of a request ID alone does not bypass queue authorization.

## Testing

Contract and controller tests cover:

- identical request-ID and URI responses;
- first, middle, and final pages through both selectors;
- default, changed, maximum, and invalid page sizes;
- malformed, unsupported-version, and cross-query page tokens;
- equal-timestamp keyset boundaries and no duplicates across stable pages;
- deterministic equal-timestamp ordering;
- state, build-event, and fact-event public mapping;
- queue isolation and cross-queue not found;
- unknown selectors versus dangling mappings;
- a repaired older occurrence appearing in chronological position;
- unknown future request-state and event strings remaining readable.

## Alternatives Considered

### Materialize current status from history

Rejected because `Request` and `ValidationFact` already own current operational state and verdicts. Replaying history would add another reconciliation path without enabling either selector.

### Query history directly by URI

Rejected because it would add query-by-attribute capability to `RequestHistoryStore`. The existing exact request-URI mapping resolves the primary request key for any backend.

### Return several histories by URI

Rejected while ingest enforces one request per `(queue, URI)`. A repeated field would imply cardinality the domain does not permit and make a future deduplication change silently alter existing responses.

### Return every entry in one response

Rejected even though SubmitQueue's request-history methods currently do this. Stovepipe permits several builds for one request and anticipates project-scoped builds, so a bounded event vocabulary does not bound entry cardinality. Establishing page semantics now avoids later changing an existing return-all method into a partial response that older clients would silently misread as complete.
Loading