Skip to content

[Migration Engine Part 2] Add a migration checkpoint table to SQL Server and PostgreSQL - #5897

Merged
warwickschroeder merged 6 commits into
warwick/migration-engine-1from
warwick/migration-engine-2
Oct 9, 2026
Merged

warwickschroeder merged 6 commits into
warwick/migration-engine-1from
warwick/migration-engine-2

Conversation

@warwickschroeder

@warwickschroeder warwickschroeder commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #5894 (warwick/migration-engine-1). Review against that branch, not master.

What this adds

Somewhere for the migration engine to save its progress on SQL Server and PostgreSQL, so a stopped copy can resume rather than restart, plus fixes to how the RavenDB source report starts. Nothing writes checkpoints yet: the table is created empty and only the tests fill it.

  • A checkpoint table on both EF Core backends, created by one new migration each, run by --setup.
  • A checkpoint saves in the same transaction as the rows it describes.
  • A save from an out-of-date copy is refused rather than silently overwriting a newer one.
  • The embedded RavenDB source waits for its own server instead of racing it, and cannot build indexes on the customer's database.
  • --migration-source-report prints its error and exits 1 instead of crashing.
  • Docs gain the report's new failure and what to do about it.

Tests

  • ServiceControl.Persistence.Tests/EFCore: the table's columns and timestamps, the store's reads and saves, and the transaction and stale-save cases, on SQL Server and PostgreSQL.

@warwickschroeder warwickschroeder self-assigned this Sep 16, 2026
@warwickschroeder
warwickschroeder added this pull request to stack #5898 September 16, 2026 04:25
@warwickschroeder
warwickschroeder force-pushed the warwick/migration-engine-2 branch from bd65b15 to 4456c32 Compare September 16, 2026 13:20
Comment thread src/ServiceControl.Persistence.EFCore/EfCoreExtensions.cs
Comment thread docs/migration/ravendb-to-sql-migration-overview.md Outdated
- Introduced a new table `MigrationCheckpoints` to track migration states and progress.
- Implemented `MigrationCheckpointEntity` to represent the checkpoint data structure.
- Added `EFMigrationCheckpointStore` for managing checkpoint data with methods for reading and upserting checkpoints.
- Created `MigrationCheckpointConfiguration` for configuring the entity in the DbContext.
- Enhanced `ServiceControlDbContext` to include the new `MigrationCheckpoints` DbSet.
- Updated `EfCoreExtensions` to provide an `UpsertCheckpoint` method for managing checkpoint persistence.
- Added unit tests for `EFMigrationCheckpointStore` and `MigrationCheckpointTable` to ensure correct functionality.
- Implemented transaction handling in `MigrationCheckpointTransactionTests` to verify checkpoint behavior during commits and rollbacks.
- Updated RavenDB integration to support embedded server readiness checks.
@warwickschroeder
warwickschroeder force-pushed the warwick/migration-engine-2 branch from 7dd5105 to a2f9ce8 Compare September 30, 2026 07:25
@warwickschroeder
warwickschroeder removed this pull request from stack #5898 October 9, 2026 01:45
…inery (#5911)

* Update migration documentation

* feat: run the required migration copy at host start

Adds the RavenDB source and the EF Core target for the KnownEndpoints and EndpointSettings categories, the startup checks that refuse an unsupported or unready migration, and the stall watchdog that stops a copy committing nothing for 30 minutes. Documents the migration contracts.

* test: cover the RavenDB to SQL migration end to end

Adds unit tests for the startup checks, the stall watchdog and the refusals, target and reader tests for both persisters, a SQL Server collation test for the endpoint settings key, and acceptance tests for a copy that is killed, restarted and opened on. Approves the two new migration settings.

* tests: ensure DB schema is setup before the host starts

* refactor: update BatchSizeFor method to return Task<int> for async compatibility across migration targets

* docs: update migration instructions and system design to clarify settings and behavior

* Refactor migration checks and improve error handling

* Refactor migration commands and error handling

* PR review
@warwickschroeder
warwickschroeder marked this pull request as ready for review October 9, 2026 09:53
@warwickschroeder
warwickschroeder merged commit 13eb6b7 into warwick/migration-engine-1 Oct 9, 2026
38 checks passed
@warwickschroeder
warwickschroeder deleted the warwick/migration-engine-2 branch October 9, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants