Skip to content

Introduce EntityEventHistory and EntityEventStorage#1649

Open
alexander-yevsyukov wants to merge 21 commits into
de-event-sourcing-PR-B3from
de-event-sourcing-phase-D
Open

Introduce EntityEventHistory and EntityEventStorage#1649
alexander-yevsyukov wants to merge 21 commits into
de-event-sourcing-PR-B3from
de-event-sourcing-phase-D

Conversation

@alexander-yevsyukov

@alexander-yevsyukov alexander-yevsyukov commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Opens Phase D of the de-event-sourcing plan with the journal cleanup
(task spec):
the journal types move to the entity level — events are emitted by entities, not by
aggregates alone. The initial rollout covers aggregates; ProcessManager journaling is a
planned follow-up, so nothing in the new types is aggregate-specific.

Stacked on #1648 (base = de-event-sourcing-PR-B3). Only the Phase D commits show in
this diff; the PR retargets to master automatically once #1648 merges.

What changed

  • New protosspine/server/entity/event_history.proto: EntityEventHistory and
    EntityEventRecord. Events only; no snapshot arm anywhere. The records need no
    dedicated identifier type: they are keyed by the core.EventId of the stored event,
    matching how DefaultEventStore keys its records.
  • New Kotlin storageEntityEventStorage (+ EntityEventRecordColumn) in
    io.spine.server.entity.storage, over the standard RecordStorage SPI.
    StorageFactory.createEntityEventStorage() added. The storage encapsulates its record
    format: write(Event) transforms the event into the journal record itself, keying it
    by the event identifier and journaling it under the event's producer, so
    AggregateStorage knows nothing about record building.
  • AggregateStorage migrated — re-typed to AbstractStorage<I, EntityEventHistory>;
    the journal writes/reads (writeEvent, historyBackward, readHistoryBackward) go
    through the new storage. read(id, batchSize) is now a single journal-tail read
    (most recent batchSize events, emission order) — matching its long-documented
    "maximum number of the events" contract, and retiring the latent pagination hazard of
    the old version < lastVersion batch cursor (post-cutover, all events of one command
    share a version, so a batch boundary inside such a group silently dropped its
    remainder). UncommittedHistory.get() returns a single EntityEventHistory.
  • Removed — the whole legacy journal machinery, per the review decision (reversing
    the task file's original retain-and-deprecate plan): AggregateEventStorage,
    AggregateEventRecordColumn, StorageFactory.createAggregateEventStorage(),
    TruncateOperation, and the snapshot-index AggregateStorage.truncateOlderThan /
    doTruncate overloads, plus their test fixtures (AggregateHistoryTruncationTest and
    the fibonacci aggregate that existed only for them). Grounds: org-wide usage research
    found zero production references (the only hits are the jdbc/gcloud truncation tests,
    which subclass the deleted fixture and are removed on their scheduled core bump); the
    trimming was partial by design (never past the newest snapshot); and keeping the class
    made every AggregateStorage eagerly materialize an empty legacy journal store per
    aggregate type. Also removed (package-private): ReadOperation,
    HistoryBackwardOperation, AggregateStorage.writeSnapshot, the
    AggregateRecords.newEventRecord(...) factories, and the dead TestAggregateStorage
    test double.
  • Retained for wire parseability — the legacy proto messages (Snapshot,
    AggregateHistory, AggregateEventRecord(Id)) with retention-status comments
    and option deprecated = true (ebc25b56c1), following the direction set by
    815db27f30 for InboxLabel.IMPORT_EVENT. Pre-cutover journal rows stay on disk as
    inert, parseable data; the runtime never touches them.
  • Count/date-based journal trimming (Phase D, item 3 — pulled in) — the replacement for
    the removed snapshot-index truncation, so production systems can bound journal growth:
    EntityEventStorage.truncate(keepMostRecent) keeps the N most recent records per
    entity
    ; truncate(keepMostRecent, olderThan) deletes only records older than the
    given time and never cuts into the per-entity recent window — a date purge cannot
    hollow out the IdempotencyGuard window. Delegating AggregateStorage.truncate(...)
    maintenance methods sit where the old truncation lived; the migration guide (§7) shows
    the recipe and the historyDepth caution.
  • Tests — new Kotlin EntityEventStorageSpec (journal reads, event-to-record
    transformation, truncation); a vendor-facing TruncateJournal group added to
    AggregateStorageTest; AggregateStorageTest and StorageRecords migrated to the
    new record kind, with test events emitted on behalf of the entity they are read
    back by (the journal keys records by the event producer).
  • Recent-history reads are journal-only and entity-levelRecentHistory.read(depth)
    always serves from the installed RecentHistoryLoader (the durable journal); the
    in-memory copy of recent events is removed together with
    TransactionalEntity.appendToRecentHistory()/clearRecentHistory() — event caching
    belongs to the storage side, not to the entity. An entity created outside a repository
    has no journal, so its reads return no events. The loader interface lives in
    io.spine.server.entity with TransactionalEntity.setRecentHistoryLoader(...) as the
    wiring point, so the future ProcessManager journaling only wires its repository side.
  • Migration guide — new §9 with the data caveat: EntityEventRecord is a new record
    kind, so pre-upgrade journal rows are invisible to the new reads; the
    IdempotencyGuard window and historyBackward(depth) start empty after the upgrade.

Breaking / compatibility notes

  • Breaking for storage vendors: the AggregateStorage type parameter,
    writeAll/writeEventRecord/historyBackward signatures, and the published
    test-fixture APIs changed, and the legacy journal classes are gone — hence the version
    rounds up to 2.0.0-SNAPSHOT.430. Vendors recompile against the same
    RecordStorage-level SPI (no rewrite) and delete their two truncation-test
    subclasses (JdbcAggregateStorageTruncationTest, DsAggregateStorageTruncationTest).
  • Wire compatibility preserved: no proto fields or messages removed.
  • Rider commits from the parallel session: 815db27f30 adds [deprecated = true] to
    InboxLabel.IMPORT_EVENT; 662519036b archives two completed task files.

Left out on purpose: the EventImported and AggregateHistoryCorrupted system
events are also no longer produced, but they still have live (un-deprecated) emitter and
observer code — EntityLifecycle.onEventImported / onCorruptedState and
TraceEventObserver.on(EventImported). Marking those protos deprecated belongs together
with deprecating that code, which is the import/diagnostics cleanup rather than the
journal one — can follow in a separate small PR if desired.

Verification

  • ./gradlew clean build dokkaGenerate — green (all module test suites + Dokka).
  • Reviewers: spine-code-review, kotlin-engineer, review-docs — all findings applied
    (incl. the .430 version bump, @JvmOverloads on historyBackward, the
    TruncateOperation.newestFirst rename of the mislabeled "chronological" helper, and
    delete/deleteAll spec coverage).
  • Follow-up after merge/publish: smoke-build the storage vendors against .430
    (Phase C list), per the task's verification section.

🤖 Generated with Claude Code

alexander-yevsyukov and others added 5 commits July 9, 2026 20:02
Open Phase D of the de-event-sourcing plan with the journal cleanup:
the journal types move to the entity level — events are emitted by
entities, not by aggregates alone.

- Define the entity-level journal protos in a new `event_history.proto`:
  `EntityEventHistory`, `EntityEventRecord`, `EntityEventRecordId` —
  no snapshot arm anywhere.
- Add Kotlin `EntityEventStorage`, `EntityEventRecordColumn`, and
  `EntityEventRecords` under `io.spine.server.entity.storage`; add
  `StorageFactory.createEntityEventStorage()`.
- Re-point the `AggregateStorage` journal to the new storage and re-type
  the class to `AbstractStorage<I, EntityEventHistory>`.
  `read(id, batchSize)` becomes a single journal-tail read matching its
  documented "maximum number of the events" contract, and
  `UncommittedHistory.get()` returns a single `EntityEventHistory`.
- Remove `ReadOperation` and `HistoryBackwardOperation` (both
  package-private); the removal also retires the latent same-version
  pagination hazard of the old batch cursor.
- Deprecate the legacy pair — `AggregateEventStorage`,
  `AggregateEventRecordColumn`, `createAggregateEventStorage()`, and the
  snapshot-index `truncateOlderThan` — keeping the truncation wired to
  the legacy journal records. Document the retention status of
  `AggregateHistory` and `AggregateEventRecord(Id)` in the proto.
- Migrate the test fixtures; add `EntityEventStorageSpec` and
  `EntityEventRecordsSpec`; state the new-record-kind data caveat in the
  migration guide (§9).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The change is breaking for storage vendors (`AggregateStorage` type
parameter and test-fixture APIs), so the increment rounds up to the
next multiple of ten.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6a39a41f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.75776% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.92%. Comparing base (1ae3b57) to head (e06410f).

Additional details and impacted files
@@                     Coverage Diff                     @@
##           de-event-sourcing-PR-B3    #1649      +/-   ##
===========================================================
+ Coverage                    87.89%   87.92%   +0.02%     
===========================================================
  Files                         1071     1063       -8     
  Lines                        23246    22922     -324     
  Branches                      1079     1065      -14     
===========================================================
- Hits                         20433    20155     -278     
+ Misses                        2424     2383      -41     
+ Partials                       389      384       -5     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add `option deprecated = true` to `Snapshot`, `AggregateEventRecordId`,
`AggregateEventRecord`, and `AggregateHistory`, following the direction
set for `InboxLabel.IMPORT_EVENT`. The messages stay in place for wire
compatibility; the generated Java classes now carry `@Deprecated`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR advances the de-event-sourcing plan (Phase D) by moving the journal types from aggregate-specific (AggregateHistory/AggregateEventRecord) to entity-level (EntityEventHistory/EntityEventRecord) and migrating AggregateStorage to use the new entity journal storage while keeping legacy journal support for deprecated snapshot-index truncation.

Changes:

  • Introduce EntityEventHistory/EntityEventRecord(Id) protos and a new EntityEventStorage (+ columns & record factory) over the standard RecordStorage SPI.
  • Migrate AggregateStorage to read/write entity-level journal records and simplify read(id, batchSize) into a single tail read (with legacy truncation explicitly operating on legacy records only).
  • Update/deprecate legacy journal APIs and add/migrate tests and migration docs to reflect the new record kind and the “pre-upgrade journal invisibility” caveat.

Reviewed changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
version.gradle.kts Bump published snapshot version to .430.
docs/dependencies/pom.xml Align docs dependency BOM version with .430.
docs/dependencies/dependencies.md Regenerate dependency report for .430.
server/src/main/proto/spine/server/entity/event_history.proto Add new entity-level event history + journal record protos.
server/src/main/proto/spine/server/delivery/inbox.proto Mark IMPORT_EVENT as deprecated for wire-compat only.
server/src/main/proto/spine/server/aggregate/aggregate.proto Mark legacy snapshot/journal protos deprecated and update retention docs.
server/src/main/kotlin/io/spine/server/entity/storage/EntityEventStorage.kt New storage for entity event journal (tail reads, maintenance ops).
server/src/main/kotlin/io/spine/server/entity/storage/EntityEventRecords.kt New factory for building EntityEventRecords from events.
server/src/main/kotlin/io/spine/server/entity/storage/EntityEventRecordColumn.kt Column definitions for querying/sorting entity journal records.
server/src/main/java/io/spine/server/storage/StorageFactory.java Add createEntityEventStorage; deprecate aggregate-level factory method.
server/src/main/java/io/spine/server/aggregate/AggregateStorage.java Re-type to EntityEventHistory; route journal I/O via entity journal; keep legacy truncation storage.
server/src/main/java/io/spine/server/aggregate/UncommittedHistory.java Return a single EntityEventHistory instead of segmented aggregate histories.
server/src/main/java/io/spine/server/aggregate/TruncateOperation.java Retarget truncation to legacy journal and centralize newest-first query sort helper.
server/src/main/java/io/spine/server/aggregate/AggregateRecords.java Remove event/snapshot journal record factories; leave state record factory.
server/src/main/java/io/spine/server/aggregate/AggregateEventStorage.java Deprecate and clarify as legacy-only journal storage.
server/src/main/java/io/spine/server/aggregate/AggregateEventRecordColumn.java Deprecate legacy journal columns and point to entity journal columns.
server/src/testFixtures/java/io/spine/server/aggregate/given/StorageRecords.java Migrate fixture records to EntityEventRecord.
server/src/testFixtures/java/io/spine/server/aggregate/AggregateStorageTest.java Update fixture tests to new history/record types and semantics.
server/src/testFixtures/java/io/spine/server/aggregate/AggregateHistoryTruncationTest.java Explicitly simulate pre-cutover legacy journals via legacyJournal().
server/src/test/kotlin/io/spine/server/entity/storage/EntityEventStorageSpec.kt New Kotlin tests for read windowing and delete/deleteAll.
server/src/test/kotlin/io/spine/server/entity/storage/EntityEventRecordsSpec.kt New Kotlin tests for record creation and input validation.
server/src/testFixtures/java/io/spine/server/aggregate/given/ReadOperationTestEnv.java Remove legacy test env tied to snapshot-era read operation.
server/src/test/java/io/spine/server/aggregate/TestAggregateStorage.java Remove unused storage test double tied to removed APIs.
server/src/test/java/io/spine/server/aggregate/ReadOperationTest.java Remove tests for deleted ReadOperation.
server/src/main/java/io/spine/server/aggregate/ReadOperation.java Remove snapshot-era paginated read operation (superseded by tail read).
server/src/main/java/io/spine/server/aggregate/HistoryBackwardOperation.java Remove legacy history read helper (superseded by EntityEventStorage).
docs/migration/aggregates-without-event-sourcing.md Add §9 describing entity-level journal migration + new-record-kind data caveat.
.agents/tasks/introduce-event-history-type.md Task status/implementation notes update (review-exempt).
.agents/tasks/de-event-sourcing-plan.md Plan progress note for Phase D (review-exempt).
.agents/tasks/archive/transaction-to-kotlin.md Archived task material (review-exempt).
.agents/tasks/archive/java-to-kotlin-transactional-entity.md Archived task material (review-exempt).
.agents/memory/MEMORY.md Memory index update (review-exempt).
.agents/memory/kotlin-storage-spi-interop-traps.md New repo memory note (review-exempt).

Comment thread server/src/main/java/io/spine/server/aggregate/TruncateOperation.java Outdated
alexander-yevsyukov and others added 14 commits July 9, 2026 22:07
Reverse the retain-and-deprecate decision for the storage-level legacy
journal API (product owner, during the PR review). Org-wide usage
research found no production references: the only hits were the two
vendor truncation tests, which subclass the test fixture removed here
and die on the vendors' scheduled core bump.

- Remove `AggregateEventStorage`, `AggregateEventRecordColumn`,
  `TruncateOperation`, `StorageFactory.createAggregateEventStorage()`,
  and the snapshot-index `AggregateStorage.truncateOlderThan` /
  `doTruncate` overloads together with the `legacyJournal()` accessor.
  `AggregateStorage` no longer materializes a legacy journal store for
  every aggregate type.
- Remove `AggregateHistoryTruncationTest`, its in-memory subclass, and
  the fibonacci fixtures, which existed only for the truncation tests.
- Keep the legacy proto messages (marked `deprecated`) so the journal
  records persisted by the earlier versions stay parseable; the runtime
  never reads them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the removed snapshot-index truncation with the trimming designed
for the current journal (plan item D3), so that production systems can
purge the history as it grows:

- `EntityEventStorage.truncate(keepMostRecent)` keeps the given number
  of the most recent records for each entity, in the `historyBackward`
  order.
- `EntityEventStorage.truncate(keepMostRecent, olderThan)` deletes only
  the records older than the given time and never cuts into the
  per-entity recent window, so a date-based purge cannot hollow out the
  `IdempotencyGuard` window.
- `AggregateStorage.truncate(...)` delegates to the journal storage,
  taking the place of the removed `truncateOlderThan`.

Cover the operation with the `EntityEventStorageSpec` cases and a
vendor-facing `TruncateJournal` group in `AggregateStorageTest`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without the guard, the in-memory storage treats a non-positive query
limit as "no limit", so `historyBackward(id, 0)` would read the whole
journal of the entity instead of failing fast.

Addresses the Codex and Copilot review comments on PR #1649.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dedicated identifier type was a leftover of the legacy design, where
snapshot records had no event behind them and needed synthetic string
identifiers. An `EntityEventRecord` always wraps exactly one `Event`,
and an emitted event has exactly one producer, so the `core.EventId` of
the stored event is a proper primary key — matching how
`DefaultEventStore` (`MessageStorage<EventId, Event>`) keys its records.

The blank-identifier guard in `EntityEventRecords.create` now checks
`EventId.value` directly, which makes it actually enforceable; the
corresponding spec case is back.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Also:
 * Remove outdated `transient` modifier for `Aggregate.recentHistoryLoader`; annotate as `@Nullable`.
Move `AggregateRecords.newStateRecord(aggregate)` into the `Aggregate`
class, placing it beside its inverse, `restore(EntityRecord)`. With the
journal record factories gone earlier, this was the only member left in
`AggregateRecords`, so the utility class is deleted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docs used the same-looking "D<n>" labels for two different things:
the ADR's decision points (D1-D10) and the items of the delivery plan's
Phase D. The plan and task files now spell the latter as "Phase D,
item N", keeping the bare labels for the ADR decisions only. Also bring
the phase-opener coordination note up to date: the trimming shipped in
PR #1649 and the legacy truncation is removed, not "wired" anywhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ent)`

`AggregateStorage` knew how the journal builds its records. Now the
journal storage accepts an `Event` and transforms it into the record
itself: the record is keyed by the event identifier, and the entity is
determined by the producer ID of the event context — an emitted event
knows its producer. The `EntityEventRecords` factory dissolves into the
private `EntityEventStorage.toRecord()`.

Removing the enrichments before storing rebuilds the event, which also
validates it, so an incomplete instance is rejected with
a `ValidationException`; the explicit precondition checks became
unreachable and are dropped.

The tests writing events now emit them on behalf of the entity they
later read the history by, using producer-bound `TestEventFactory`
instances.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`Aggregate.historyBackward(depth)` branched between the loader callback
installed by the repository and the in-memory recent history. The
duality now lives where the data lives: `RecentHistory.read(depth)`
serves the reads from the installed `RecentHistoryLoader` — the durable
journal of the entity — or, when no loader is installed (an instance
created outside a repository), from the in-memory copy.

The loader interface moves to the `entity` package, and
`TransactionalEntity.setRecentHistoryLoader()` becomes the wiring
point, so extending `ProcessManager` with the event journal
functionality later only requires wiring its repository side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the in-memory copy of the recent events. Entities use their
recent history while handling signals and are not kept in memory
afterwards; even when an instance is cached, caching the events belongs
to the storage side, not to the entity. The durable journal, accessed
via the installed `RecentHistoryLoader`, is now the only source of the
recent history; an entity created outside a repository has no journal,
so its reads return no events.

Accordingly, `TransactionalEntity.appendToRecentHistory()` and
`clearRecentHistory()` are removed along with the deque-backed API of
`RecentHistory` (`iterator()`, `stream()`, `isEmpty()`), and
`Aggregate.commitEvents()` only clears the uncommitted events.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 45 out of 47 changed files in this pull request and generated 3 comments.

The `created` column and the related docs conflated the record write
time with the event's own time. The value is the timestamp of the event
context, so all ordering and truncation docs now describe it as the
time the event was created/emitted (the column name stays `created`).

Also, per the reviewers:
- Correct the `write(Event)` KDoc: `clearEnrichments()` clears the
  event's own enrichments and the first-level origin's, not "all".
- Explain why `EntityEventStorage` is final (vendors customize via the
  `RecordStorage` delegate) and document the unsupported-identifier
  failure of `historyBackward`.
- Note that `writeEventRecord` is off the runtime write path and stores
  a pre-built record as-is.
- Refresh the snapshot-era wording in `IdempotencyGuard` private docs.
- Align the retention comments of the legacy protos; fix Javadoc links
  targeting a private method; polish test naming and line lengths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexander-yevsyukov alexander-yevsyukov moved this from 🏗 In progress to In Review in v2.0 Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

2 participants