Skip to content

feat(TC-1274): add epoch_order (space, parent, child) index migration - #341

Merged
samgbafa merged 2 commits into
mainfrom
skgbafa/tc-1274-epoch-order-index
Oct 17, 2026
Merged

samgbafa merged 2 commits into
mainfrom
skgbafa/tc-1274-epoch-order-index

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

TC-1274. Adds the node migration that matches the production index Sam is building online, idx_epoch_order_space_parent_child ON epoch_order (space, parent, child). With it, main matches prod, and fresh databases get the index.

Behaviour

Database state at boot What m20261017_000000_epoch_order_space_index does
Fresh database Creates the index on an empty table. Instant.
PostgreSQL, a valid btree index of that name on (space, parent, child) already exists (operator-built) Reads pg_index and issues no DDL. It takes no lock on epoch_order, and the index keeps its OID.
PostgreSQL, an index of that name that is INVALID (failed CONCURRENTLY), or has a different shape (columns, method, predicate or expressions) Boot fails. Nothing is applied, and the error gives the DROP INDEX CONCURRENTLY / CREATE INDEX CONCURRENTLY fix.
Existing database without the index Builds it with a plain CREATE INDEX inside the boot migration transaction. The SHARE lock on epoch_order blocks writes for the whole build, and holds until every pending migration commits. Waiting for that lock is bounded by the 30 s lock_timeout from #316 (TC-1176). The build itself is not bounded.
MySQL Checks the name with has_index, then creates the index. MySQL has no CREATE INDEX IF NOT EXISTS, and sea-query silently drops the clause.

The pg_index pre-check exists because a bare CREATE INDEX IF NOT EXISTS still takes epoch_order's SHARE lock before it checks the name, and the migration transaction holds that lock until commit. The test below shows this: the naive if_not_exists version blocks behind an open writer.

down() keeps the index. It may be operator-built, and older releases run the same anti-join.

This does not copy m20261015's drop-then-create pattern.

Runbook (PostgreSQL, before upgrading any node that has history)

Run this in autocommit (psql, no BEGIN):

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);
SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;  -- must be t
-- if f: DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child; then rebuild

If you upgrade without it, the boot builds the index under a write-blocking lock.

Tests (RED first)

  • generated_create_index_statements: the exact PG, SQLite and MySQL DDL. This is MySQL's only coverage, because the full MySQL migration chain fails on init_tables with error 1074 (TC-1314, a known release caveat; I reproduced it on MySQL 8.4.7).
  • SQLite:
    • fresh database;
    • v1.20.6 upgrade (first 25 migrations, with seeded history);
    • pre-existing index kept (same rootpage).
  • PostgreSQL, through the production db::migrate entry point:
    • fresh database (valid, columns space,parent,child);
    • v1.20.6 upgrade with history;
    • operator CREATE INDEX CONCURRENTLY index kept, same OID, while a writer holds ROW EXCLUSIVE on epoch_order (finishes in under 5 s);
    • INVALID or wrong-column same-name index → DbErr naming the index and DROP INDEX CONCURRENTLY, migration not recorded.

RED evidence, native PG 18.6:

  • With a no-op stub, 5 of the 8 tests fail.
  • With a naive create_index(..if_not_exists()), postgres_operator_built_index_is_kept_without_blocking_writers fails (blocked behind the writer) and postgres_unusable_operator_index_fails_the_migration fails (the INVALID index is accepted).
  • The final version passes 8/8. cargo test -p tinycloud-core --lib migrations passes 18/18. cargo clippy -p tinycloud-core --all-targets -D warnings and fmt --check are clean.

Also: m20261015's lock-timeout test used Migrator::down(Some(1)) and so assumed it was the last migration. It now rolls back to just before itself.

No merge or deploy is implied.

Adds m20261017_000000_epoch_order_space_index, matching the index operators
build online with CREATE INDEX CONCURRENTLY. On PostgreSQL it reads pg_index
first and issues no DDL (so takes no SHARE lock on epoch_order) when a valid
btree index of that name on (space, parent, child) exists; an INVALID or
differently shaped index of that name fails the boot with the fix. Without
the index it builds it in the migration transaction. down() keeps the index.

The m20261015 lock-timeout test no longer assumes it is the last migration.
@samgbafa

Copy link
Copy Markdown
Contributor Author

APPROVE: see TC-1274 (head 4e371d756751c6fb6f6b162fdd3e1d0789466ddc). Review: https://linear.app/tinycloud-labs/issue/TC-1274/investigate-sam-owner-sql-read-stalls-across-node-hosts#comment-b2453ff1

samgbafa added a commit that referenced this pull request Oct 17, 2026
…st_keys (#326)

Orders after #341's m20261017_000000_epoch_order_space_index. No schema
change.
…ogress

The pg_index pre-check now requires indisvalid, indisready and indislive.
A same-name index whose CREATE INDEX CONCURRENTLY is still running fails
the boot with 'still being built ... wait', not the DROP INDEX fix. It is
found by its pg_stat_progress_create_index row: by index_relid, or, when
PostgreSQL hides the row's columns from this role, by that backend's SHARE
UPDATE EXCLUSIVE lock on epoch_order.

The m20261015 lock test starts from migrated_through(that migration)
instead of rolling back with down().
@samgbafa

Copy link
Copy Markdown
Contributor Author

APPROVE: see TC-1274 (delta 4e371d7..d21be15, head d21be15d3c4d16e21140a1862bec6b9493141744). Review: https://linear.app/tinycloud-labs/issue/TC-1274/investigate-sam-owner-sql-read-stalls-across-node-hosts#comment-b43b9571

@samgbafa
samgbafa merged commit 1cbeb7d into main Oct 17, 2026
22 checks passed
samgbafa added a commit that referenced this pull request Oct 17, 2026
TC-735's later kv_orphan_candidates migration made Migrator::down(1)
undo it instead of the cursor index. Identical to #341's fix.
samgbafa added a commit that referenced this pull request Oct 17, 2026
Rename the unshipped kv_orphan_candidates migration from
m20261015_020000 to m20261017_020000 so it sorts after main's
m20261017_000000_epoch_order_space_index, per the second-merger rule.
Take main's cursor-index lock test (migrated_through), dropping ours.
samgbafa added a commit that referenced this pull request Oct 17, 2026
…lit (#326)

- D1: a mirrored data key that doesn't authenticate is never ignored and
  never overwritten. It may be the genuine key under a generation that
  didn't load (a restore plus a failed boot read). restore_dek fails
  closed for that space (AtRestError::DekUnavailable, transient, reason
  dek_unavailable) instead of letting create_dek and mirror_dek replace
  it; mirror_dek never overwrites an existing object with other bytes;
  an oversized DEK object fails closed too. Test reproduces the review's
  D1 chain and checks the original key returns once the object reads.
- KeyringConfig::allow_resplit_before_verify: a holder change may
  re-split at boot only with the flag, no pin, an unverified active
  escrow and no data key ever made; otherwise EscrowHoldersChanged.
- Merges main (#341's m20261017_000000_epoch_order_space_index and its
  migrated_through m20261015 test).
samgbafa added a commit that referenced this pull request Oct 17, 2026
…ig (#327)

- FS mirror get opens with O_NOFOLLOW | O_NONBLOCK and requires a
  regular file (fstat on the handle) before reading, so a FIFO under an
  escrow or record name can't hang the boot (test).
- at_rest.escrow.allow_resplit_before_verify passes through to the
  keyring's guarded pre-ceremony re-split.
- Merges #326 (D1 fail-closed data keys, main with #341).
- Docs: the re-split flag, dek_unavailable, the never-overwrite DEK rule.
samgbafa added a commit that referenced this pull request Oct 17, 2026
…ory (#343)

* feat(TC-1274): maintain epoch_head so writes stop scanning epoch history

transact found a space's head epochs (a new epoch's parents) with an
epoch/epoch_order anti-join over the space's whole history on every write,
under the space's sequence lock. It now reads them from a new epoch_head
table (PK (space, id)) and records the new epoch as the space's only head in
the same transaction: one UPDATE in the usual one-row case.

The cache is trusted only when its newest seq is the space's newest
(MAX(event_order.seq) + 1, which transact already computes). Every epoch,
including one appended by a release that does not write epoch_head, outranks
all existing ones under the same lock, so a stale cache is always detected;
that space falls back to the anti-join once and the write repairs it. Heads
are taken in id byte order; nothing re-derives an epoch hash from its
parents.

m20261018_000000_epoch_head creates only the empty table, with no foreign
keys, so it takes no lock on existing tables. A gentle online backfill
([epoch_head_backfill], on by default at 2 spaces/s, pausing at least as long
as each space took, stopping at its first error) fills spaces written before
the table existed; writes are correct without it.

The TC-951 round-trip budgets rise by the measured cache write: sign-in
75 -> 77, kv put 37 -> 38. Also folds in two #341 review nits: the
in-progress check's pg_locks is limited to the current database, and a
failed index plus a non-building SHARE UPDATE EXCLUSIVE holder still gets
the DROP advice.

* fix(TC-1274): epoch_head review delta: doc reflow, rollback note, disabled pacing

- Reflow the epoch_head module doc so no line starts with '+ 1' (clippy
  doc_lazy_continuation).
- CHANGELOG: rolling back to v1.20.x first needs the seaql_migrations rows
  of m20261017 and m20261018 deleted (sea-orm-migration refuses unknown
  applied versions); keep the index and the table.
- A disabled backfill boots whatever its pacing fields say; an enabled one
  still needs spaces_per_second and progress_every of at least 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant