Introduce EntityEventHistory and EntityEventStorage#1649
Introduce EntityEventHistory and EntityEventStorage#1649alexander-yevsyukov wants to merge 21 commits into
EntityEventHistory and EntityEventStorage#1649Conversation
Open Phase D of the de-event-sourcing plan with the journal cleanup: the journal types move to the entity level — events are emitted by entities, not by aggregates alone. - Define the entity-level journal protos in a new `event_history.proto`: `EntityEventHistory`, `EntityEventRecord`, `EntityEventRecordId` — no snapshot arm anywhere. - Add Kotlin `EntityEventStorage`, `EntityEventRecordColumn`, and `EntityEventRecords` under `io.spine.server.entity.storage`; add `StorageFactory.createEntityEventStorage()`. - Re-point the `AggregateStorage` journal to the new storage and re-type the class to `AbstractStorage<I, EntityEventHistory>`. `read(id, batchSize)` becomes a single journal-tail read matching its documented "maximum number of the events" contract, and `UncommittedHistory.get()` returns a single `EntityEventHistory`. - Remove `ReadOperation` and `HistoryBackwardOperation` (both package-private); the removal also retires the latent same-version pagination hazard of the old batch cursor. - Deprecate the legacy pair — `AggregateEventStorage`, `AggregateEventRecordColumn`, `createAggregateEventStorage()`, and the snapshot-index `truncateOlderThan` — keeping the truncation wired to the legacy journal records. Document the retention status of `AggregateHistory` and `AggregateEventRecord(Id)` in the proto. - Migrate the test fixtures; add `EntityEventStorageSpec` and `EntityEventRecordsSpec`; state the new-record-kind data caveat in the migration guide (§9). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The change is breaking for storage vendors (`AggregateStorage` type parameter and test-fixture APIs), so the increment rounds up to the next multiple of ten. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6a39a41f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## de-event-sourcing-PR-B3 #1649 +/- ##
===========================================================
+ Coverage 87.89% 87.92% +0.02%
===========================================================
Files 1071 1063 -8
Lines 23246 22922 -324
Branches 1079 1065 -14
===========================================================
- Hits 20433 20155 -278
+ Misses 2424 2383 -41
+ Partials 389 384 -5 🚀 New features to boost your workflow:
|
Add `option deprecated = true` to `Snapshot`, `AggregateEventRecordId`, `AggregateEventRecord`, and `AggregateHistory`, following the direction set for `InboxLabel.IMPORT_EVENT`. The messages stay in place for wire compatibility; the generated Java classes now carry `@Deprecated`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR advances the de-event-sourcing plan (Phase D) by moving the journal types from aggregate-specific (AggregateHistory/AggregateEventRecord) to entity-level (EntityEventHistory/EntityEventRecord) and migrating AggregateStorage to use the new entity journal storage while keeping legacy journal support for deprecated snapshot-index truncation.
Changes:
- Introduce
EntityEventHistory/EntityEventRecord(Id)protos and a newEntityEventStorage(+ columns & record factory) over the standardRecordStorageSPI. - Migrate
AggregateStorageto read/write entity-level journal records and simplifyread(id, batchSize)into a single tail read (with legacy truncation explicitly operating on legacy records only). - Update/deprecate legacy journal APIs and add/migrate tests and migration docs to reflect the new record kind and the “pre-upgrade journal invisibility” caveat.
Reviewed changes
Copilot reviewed 31 out of 33 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| version.gradle.kts | Bump published snapshot version to .430. |
| docs/dependencies/pom.xml | Align docs dependency BOM version with .430. |
| docs/dependencies/dependencies.md | Regenerate dependency report for .430. |
| server/src/main/proto/spine/server/entity/event_history.proto | Add new entity-level event history + journal record protos. |
| server/src/main/proto/spine/server/delivery/inbox.proto | Mark IMPORT_EVENT as deprecated for wire-compat only. |
| server/src/main/proto/spine/server/aggregate/aggregate.proto | Mark legacy snapshot/journal protos deprecated and update retention docs. |
| server/src/main/kotlin/io/spine/server/entity/storage/EntityEventStorage.kt | New storage for entity event journal (tail reads, maintenance ops). |
| server/src/main/kotlin/io/spine/server/entity/storage/EntityEventRecords.kt | New factory for building EntityEventRecords from events. |
| server/src/main/kotlin/io/spine/server/entity/storage/EntityEventRecordColumn.kt | Column definitions for querying/sorting entity journal records. |
| server/src/main/java/io/spine/server/storage/StorageFactory.java | Add createEntityEventStorage; deprecate aggregate-level factory method. |
| server/src/main/java/io/spine/server/aggregate/AggregateStorage.java | Re-type to EntityEventHistory; route journal I/O via entity journal; keep legacy truncation storage. |
| server/src/main/java/io/spine/server/aggregate/UncommittedHistory.java | Return a single EntityEventHistory instead of segmented aggregate histories. |
| server/src/main/java/io/spine/server/aggregate/TruncateOperation.java | Retarget truncation to legacy journal and centralize newest-first query sort helper. |
| server/src/main/java/io/spine/server/aggregate/AggregateRecords.java | Remove event/snapshot journal record factories; leave state record factory. |
| server/src/main/java/io/spine/server/aggregate/AggregateEventStorage.java | Deprecate and clarify as legacy-only journal storage. |
| server/src/main/java/io/spine/server/aggregate/AggregateEventRecordColumn.java | Deprecate legacy journal columns and point to entity journal columns. |
| server/src/testFixtures/java/io/spine/server/aggregate/given/StorageRecords.java | Migrate fixture records to EntityEventRecord. |
| server/src/testFixtures/java/io/spine/server/aggregate/AggregateStorageTest.java | Update fixture tests to new history/record types and semantics. |
| server/src/testFixtures/java/io/spine/server/aggregate/AggregateHistoryTruncationTest.java | Explicitly simulate pre-cutover legacy journals via legacyJournal(). |
| server/src/test/kotlin/io/spine/server/entity/storage/EntityEventStorageSpec.kt | New Kotlin tests for read windowing and delete/deleteAll. |
| server/src/test/kotlin/io/spine/server/entity/storage/EntityEventRecordsSpec.kt | New Kotlin tests for record creation and input validation. |
| server/src/testFixtures/java/io/spine/server/aggregate/given/ReadOperationTestEnv.java | Remove legacy test env tied to snapshot-era read operation. |
| server/src/test/java/io/spine/server/aggregate/TestAggregateStorage.java | Remove unused storage test double tied to removed APIs. |
| server/src/test/java/io/spine/server/aggregate/ReadOperationTest.java | Remove tests for deleted ReadOperation. |
| server/src/main/java/io/spine/server/aggregate/ReadOperation.java | Remove snapshot-era paginated read operation (superseded by tail read). |
| server/src/main/java/io/spine/server/aggregate/HistoryBackwardOperation.java | Remove legacy history read helper (superseded by EntityEventStorage). |
| docs/migration/aggregates-without-event-sourcing.md | Add §9 describing entity-level journal migration + new-record-kind data caveat. |
| .agents/tasks/introduce-event-history-type.md | Task status/implementation notes update (review-exempt). |
| .agents/tasks/de-event-sourcing-plan.md | Plan progress note for Phase D (review-exempt). |
| .agents/tasks/archive/transaction-to-kotlin.md | Archived task material (review-exempt). |
| .agents/tasks/archive/java-to-kotlin-transactional-entity.md | Archived task material (review-exempt). |
| .agents/memory/MEMORY.md | Memory index update (review-exempt). |
| .agents/memory/kotlin-storage-spi-interop-traps.md | New repo memory note (review-exempt). |
Reverse the retain-and-deprecate decision for the storage-level legacy journal API (product owner, during the PR review). Org-wide usage research found no production references: the only hits were the two vendor truncation tests, which subclass the test fixture removed here and die on the vendors' scheduled core bump. - Remove `AggregateEventStorage`, `AggregateEventRecordColumn`, `TruncateOperation`, `StorageFactory.createAggregateEventStorage()`, and the snapshot-index `AggregateStorage.truncateOlderThan` / `doTruncate` overloads together with the `legacyJournal()` accessor. `AggregateStorage` no longer materializes a legacy journal store for every aggregate type. - Remove `AggregateHistoryTruncationTest`, its in-memory subclass, and the fibonacci fixtures, which existed only for the truncation tests. - Keep the legacy proto messages (marked `deprecated`) so the journal records persisted by the earlier versions stay parseable; the runtime never reads them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the removed snapshot-index truncation with the trimming designed for the current journal (plan item D3), so that production systems can purge the history as it grows: - `EntityEventStorage.truncate(keepMostRecent)` keeps the given number of the most recent records for each entity, in the `historyBackward` order. - `EntityEventStorage.truncate(keepMostRecent, olderThan)` deletes only the records older than the given time and never cuts into the per-entity recent window, so a date-based purge cannot hollow out the `IdempotencyGuard` window. - `AggregateStorage.truncate(...)` delegates to the journal storage, taking the place of the removed `truncateOlderThan`. Cover the operation with the `EntityEventStorageSpec` cases and a vendor-facing `TruncateJournal` group in `AggregateStorageTest`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without the guard, the in-memory storage treats a non-positive query limit as "no limit", so `historyBackward(id, 0)` would read the whole journal of the entity instead of failing fast. Addresses the Codex and Copilot review comments on PR #1649. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dedicated identifier type was a leftover of the legacy design, where snapshot records had no event behind them and needed synthetic string identifiers. An `EntityEventRecord` always wraps exactly one `Event`, and an emitted event has exactly one producer, so the `core.EventId` of the stored event is a proper primary key — matching how `DefaultEventStore` (`MessageStorage<EventId, Event>`) keys its records. The blank-identifier guard in `EntityEventRecords.create` now checks `EventId.value` directly, which makes it actually enforceable; the corresponding spec case is back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also: * Remove outdated `transient` modifier for `Aggregate.recentHistoryLoader`; annotate as `@Nullable`.
…de-event-sourcing-phase-D
Move `AggregateRecords.newStateRecord(aggregate)` into the `Aggregate` class, placing it beside its inverse, `restore(EntityRecord)`. With the journal record factories gone earlier, this was the only member left in `AggregateRecords`, so the utility class is deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docs used the same-looking "D<n>" labels for two different things: the ADR's decision points (D1-D10) and the items of the delivery plan's Phase D. The plan and task files now spell the latter as "Phase D, item N", keeping the bare labels for the ADR decisions only. Also bring the phase-opener coordination note up to date: the trimming shipped in PR #1649 and the legacy truncation is removed, not "wired" anywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ent)` `AggregateStorage` knew how the journal builds its records. Now the journal storage accepts an `Event` and transforms it into the record itself: the record is keyed by the event identifier, and the entity is determined by the producer ID of the event context — an emitted event knows its producer. The `EntityEventRecords` factory dissolves into the private `EntityEventStorage.toRecord()`. Removing the enrichments before storing rebuilds the event, which also validates it, so an incomplete instance is rejected with a `ValidationException`; the explicit precondition checks became unreachable and are dropped. The tests writing events now emit them on behalf of the entity they later read the history by, using producer-bound `TestEventFactory` instances. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`Aggregate.historyBackward(depth)` branched between the loader callback installed by the repository and the in-memory recent history. The duality now lives where the data lives: `RecentHistory.read(depth)` serves the reads from the installed `RecentHistoryLoader` — the durable journal of the entity — or, when no loader is installed (an instance created outside a repository), from the in-memory copy. The loader interface moves to the `entity` package, and `TransactionalEntity.setRecentHistoryLoader()` becomes the wiring point, so extending `ProcessManager` with the event journal functionality later only requires wiring its repository side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the in-memory copy of the recent events. Entities use their recent history while handling signals and are not kept in memory afterwards; even when an instance is cached, caching the events belongs to the storage side, not to the entity. The durable journal, accessed via the installed `RecentHistoryLoader`, is now the only source of the recent history; an entity created outside a repository has no journal, so its reads return no events. Accordingly, `TransactionalEntity.appendToRecentHistory()` and `clearRecentHistory()` are removed along with the deque-backed API of `RecentHistory` (`iterator()`, `stream()`, `isEmpty()`), and `Aggregate.commitEvents()` only clears the uncommitted events. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The `created` column and the related docs conflated the record write time with the event's own time. The value is the timestamp of the event context, so all ordering and truncation docs now describe it as the time the event was created/emitted (the column name stays `created`). Also, per the reviewers: - Correct the `write(Event)` KDoc: `clearEnrichments()` clears the event's own enrichments and the first-level origin's, not "all". - Explain why `EntityEventStorage` is final (vendors customize via the `RecordStorage` delegate) and document the unsupported-identifier failure of `historyBackward`. - Note that `writeEventRecord` is off the runtime write path and stores a pre-built record as-is. - Refresh the snapshot-era wording in `IdempotencyGuard` private docs. - Align the retention comments of the legacy protos; fix Javadoc links targeting a private method; polish test naming and line lengths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opens Phase D of the de-event-sourcing plan with the journal cleanup
(task spec):
the journal types move to the entity level — events are emitted by entities, not by
aggregates alone. The initial rollout covers aggregates;
ProcessManagerjournaling is aplanned follow-up, so nothing in the new types is aggregate-specific.
What changed
spine/server/entity/event_history.proto:EntityEventHistoryandEntityEventRecord. Events only; no snapshot arm anywhere. The records need nodedicated identifier type: they are keyed by the
core.EventIdof the stored event,matching how
DefaultEventStorekeys its records.EntityEventStorage(+EntityEventRecordColumn) inio.spine.server.entity.storage, over the standardRecordStorageSPI.StorageFactory.createEntityEventStorage()added. The storage encapsulates its recordformat:
write(Event)transforms the event into the journal record itself, keying itby the event identifier and journaling it under the event's producer, so
AggregateStorageknows nothing about record building.AggregateStoragemigrated — re-typed toAbstractStorage<I, EntityEventHistory>;the journal writes/reads (
writeEvent,historyBackward,readHistoryBackward) gothrough the new storage.
read(id, batchSize)is now a single journal-tail read(most recent
batchSizeevents, emission order) — matching its long-documented"maximum number of the events" contract, and retiring the latent pagination hazard of
the old
version < lastVersionbatch cursor (post-cutover, all events of one commandshare a version, so a batch boundary inside such a group silently dropped its
remainder).
UncommittedHistory.get()returns a singleEntityEventHistory.the task file's original retain-and-deprecate plan):
AggregateEventStorage,AggregateEventRecordColumn,StorageFactory.createAggregateEventStorage(),TruncateOperation, and the snapshot-indexAggregateStorage.truncateOlderThan/doTruncateoverloads, plus their test fixtures (AggregateHistoryTruncationTestandthe fibonacci aggregate that existed only for them). Grounds: org-wide usage research
found zero production references (the only hits are the jdbc/gcloud truncation tests,
which subclass the deleted fixture and are removed on their scheduled core bump); the
trimming was partial by design (never past the newest snapshot); and keeping the class
made every
AggregateStorageeagerly materialize an empty legacy journal store peraggregate type. Also removed (package-private):
ReadOperation,HistoryBackwardOperation,AggregateStorage.writeSnapshot, theAggregateRecords.newEventRecord(...)factories, and the deadTestAggregateStoragetest double.
Snapshot,AggregateHistory,AggregateEventRecord(Id)) with retention-status commentsand
option deprecated = true(ebc25b56c1), following the direction set by815db27f30forInboxLabel.IMPORT_EVENT. Pre-cutover journal rows stay on disk asinert, parseable data; the runtime never touches them.
the removed snapshot-index truncation, so production systems can bound journal growth:
EntityEventStorage.truncate(keepMostRecent)keeps the N most recent records perentity;
truncate(keepMostRecent, olderThan)deletes only records older than thegiven time and never cuts into the per-entity recent window — a date purge cannot
hollow out the
IdempotencyGuardwindow. DelegatingAggregateStorage.truncate(...)maintenance methods sit where the old truncation lived; the migration guide (§7) shows
the recipe and the
historyDepthcaution.EntityEventStorageSpec(journal reads, event-to-recordtransformation, truncation); a vendor-facing
TruncateJournalgroup added toAggregateStorageTest;AggregateStorageTestandStorageRecordsmigrated to thenew record kind, with test events emitted on behalf of the entity they are read
back by (the journal keys records by the event producer).
RecentHistory.read(depth)always serves from the installed
RecentHistoryLoader(the durable journal); thein-memory copy of recent events is removed together with
TransactionalEntity.appendToRecentHistory()/clearRecentHistory()— event cachingbelongs to the storage side, not to the entity. An entity created outside a repository
has no journal, so its reads return no events. The loader interface lives in
io.spine.server.entitywithTransactionalEntity.setRecentHistoryLoader(...)as thewiring point, so the future
ProcessManagerjournaling only wires its repository side.EntityEventRecordis a new recordkind, so pre-upgrade journal rows are invisible to the new reads; the
IdempotencyGuardwindow andhistoryBackward(depth)start empty after the upgrade.Breaking / compatibility notes
AggregateStoragetype parameter,writeAll/writeEventRecord/historyBackwardsignatures, and the publishedtest-fixture APIs changed, and the legacy journal classes are gone — hence the version
rounds up to
2.0.0-SNAPSHOT.430. Vendors recompile against the sameRecordStorage-level SPI (no rewrite) and delete their two truncation-testsubclasses (
JdbcAggregateStorageTruncationTest,DsAggregateStorageTruncationTest).815db27f30adds[deprecated = true]toInboxLabel.IMPORT_EVENT;662519036barchives two completed task files.Left out on purpose: the
EventImportedandAggregateHistoryCorruptedsystemevents are also no longer produced, but they still have live (un-deprecated) emitter and
observer code —
EntityLifecycle.onEventImported/onCorruptedStateandTraceEventObserver.on(EventImported). Marking those protosdeprecatedbelongs togetherwith deprecating that code, which is the import/diagnostics cleanup rather than the
journal one — can follow in a separate small PR if desired.
Verification
./gradlew clean build dokkaGenerate— green (all module test suites + Dokka).spine-code-review,kotlin-engineer,review-docs— all findings applied(incl. the
.430version bump,@JvmOverloadsonhistoryBackward, theTruncateOperation.newestFirstrename of the mislabeled "chronological" helper, anddelete/deleteAllspec coverage)..430(Phase C list), per the task's verification section.
🤖 Generated with Claude Code