Test rolling 5.x upgrades and a one-node rollback across a mixed-version cluster

[kriszyp]

⊙ Problem

harper-pro has no test where two 5.x versions replicate to each other. The only cross-version tests are 4.x → 5 (cloneFromLegacy, legacyCopyBaseline, replicationTopology "Replicate data from a legacy node"). A rolling minor upgrade — each node restarted on a new binary over its old data while its peers still run the old one — and a one-node rollback are untested, though past rolling-upgrade bugs (blob bytes lost, typed-structure ids diverging between nodes) lived exactly there.

❓ Your call: Requirement as specified, yes, with one correction to the task's premise: main's package.json is 5.3.1, not 5.4. So 5.2.15 → this build is the real minor upgrade (it runs the 5.3.0 upgrade directive, and the rollback crosses the confirmable minor-downgrade gate), while 5.3.1 → this build is a same-version binary swap that still crosses every replication change merged since 5.3.1. When main is bumped to 5.4, add the latest 5.3.x to the job's previous-release matrix.

It found a real defect: #1007 (filed P1, milestone v5.3). This build refuses a base copy to any peer that does not advertise safeCopyAudit — every published 5.x release — when a table has no numeric update-timestamp attribute; the old peer's subscription closes 1008 and retries forever, so it never receives the upgraded node's writes. #848 introduced the gate and is already on v5.3, so the next 5.3.x patch ships it. In a rolling upgrade it bites an old node that has no resume cursor for an upgraded node: one that joins mid-upgrade, or whose only contact was a base copy not yet flushed when the peer switched binaries (how it surfaced: 1 of the first 12 local runs).

❓ Your call: How the suite treats #1007 — updated after rebasing onto main, which now contains #1025 (closed #1007: a registration_info version probe backs a peer lacking safeCopyAudit as long as it reports a published 5.x version). The main flow still writes live from every node right after connecting, so no peer holds only a base-copy cursor when a node switches binaries. The last step's fresh node join now asserts convergence unconditionally (it used to branch on whether the previous release advertised safeCopyAudit and pin a refusal-log wait otherwise — that wait is now unreachable, since #1025 fixed the refusal). An independent review round found that unconditional assertion can pass via the already-rolled-back peer (nodeC) instead of exercising #1025 on this build's nodes, masking a future regression in #1025's fix; closed by wiring legacyCopyBaseline.test.mjs's real-binary mode (HARPER_PRO_PREVIOUS_VERSION_PATH) into the mixed-version CI job, whose own two-node topology makes the current build the only possible historical-data supplier. Planning review on that follow-up: Framing-Verdict: chosen-approach-sound.

💡 Solution

A new suite, mixedVersionRollingUpgrade.test.mjs, runs once per previous install listed in HARPER_PRO_PREVIOUS_VERSION_PATHS. Three nodes A, B, C:

Step	Binaries	Writes	Then asserts
seed	A, B, C on old	each node writes alone, then B and C add_node A; then each writes again, live	full mesh, exact convergence
1 of 3	C → this build	A (old) while C is down; then A (old) and C (new) concurrently	C caught up; exact convergence
2 of 3	B → this build	A (old) while B is down; then A (old) and B (new)	same
3 of 3	A → this build	B (new) while A is down; then A	same
cold restart	all stop, all start on this build	B, after the restart converges	nothing lost; still replicates
rollback	C → old, CONFIRM_DOWNGRADE=yes	A (new) while C is down; then C (old) and A (new)	C rejoins; exact convergence
join	fresh node D on old joins A (after C rolled back to old)	A	D converges, regardless of whether the old release advertises safeCopyAudit (#1007, fixed by #1025)

Exact convergence means that on every node, within a bounded 120 s poll (no fixed sleeps): the id set and every field equal the test's own expected map, not merely each other; the category secondary index returns exactly the expected ids for every category ever written; every blob reads back byte-equal over GET /MixedVersionItem/<id>.payload; and every 20 KB (file-backed) blob is present in the node's own blob files, because an HTTP read could have been served from a peer. Each write batch inserts records in shapes that vary from batch to batch, updates two records another batch wrote (moving them to a new index category and attaching a new blob), and deletes two, so inserts, updates and deletes all cross the version boundary in both directions. Two tables: an audited table with an indexed attribute and a Blob, and a randomAccessFields: true table for the typed-struct encoding. Every start asserts the script the process was launched from and the version it reports (5.3.1 and this build report the same version); every stop must exit cleanly on SIGTERM within 30 s (exit code 0), not be killed or crash. Once a step fails, the later ones skip instead of each burning its own timeout.

CI: a new job, run-mixed-version-tests, runs only this file, one matrix entry per previous release (5.2.15, 5.3.1). It installs the release with --ignore-scripts, then rebuilds re2 in that install — without it the old releases' waf component fails to load, so they would not be running as shipped. A guard step fails the job if either test file, the install, or re2's binary is missing, since a literal nonexistent test path exits 0 with zero tests run (verified empirically) — the same false-green class the comment above it already called out, now also covering the legacyCopyBaseline.test.mjs step added below. In the six cluster shards the file skips, so they pay nothing.

⚖️ Alternatives

Planning review (before any code): Framing-Verdict: chosen-approach-sound. Its in-frame findings were adopted: writes while each node is down, seeding on isolated nodes so their typed-structure dictionaries diverge, binary identity from the spawn arguments, the local blob-file check, and a CI guard against an empty install list.

• Core-only single-node upgrade and rollback test (the companion harper task): no replication, mesh, or blob streaming in core; it stays a companion, not a replacement.
• Golden replication frames from 5.2: cover encoding only, not the in-place storage upgrade, upgrade directives, system-table replication across versions, or rejoin after rollback.
• Point replicationTopology's legacy test at a 5.2 install: a fresh node joining, with no in-place upgrade, no mixed-mesh writes from a new node, and no rollback.
• Install the old releases in all six cluster shards (as harperdb@4 is): two installs × six shards for one file, inside shards already budgeted at 25 minutes.

❓ Your call: Exact pins (5.2.15, 5.3.1) rather than resolving the latest patch of each line when the job runs: reproducible CI, at the cost of a one-line matrix edit per release you want covered. Floating would track customers' latest patch automatically but let an npm publish turn CI red with no change here.

❓ Your call: The job runs on every pull request (two matrix entries, ~3 minutes of tests each plus setup), so a change that breaks old/new replication is caught before merge rather than on the nightly run. Restricting it to schedule and pushes to main/release branches is one if: line.

❓ Your call: Keeping the 5.3.1 suite, which only swaps binaries (no upgrade directive, no downgrade gate while main is 5.3.1). It costs one CI job of ~3 minutes; it is the path a 5.3.1 cluster takes onto the next 5.3.x patch, and it reproduces #1007 just as the 5.2.15 suite does.

❓ Your call: The oracle accepts either log replay or a full copy as the way a returning node catches up. The node logs from every local run show each restarted node resuming from its per-peer cursor (direct resume cursor), but the test does not assert it; doing so needs a fixture endpoint into internal cursor state that also works on the old releases.

❓ Your call: Every stop is a clean SIGTERM. A hard kill at the version boundary (crash recovery across a binary swap) is not covered; it is additive later.

❓ Your call: Release target: v5.3. Review flagged the PR milestone (v5.4) against the evidence (this PR's changed code is also live on v5.3, and it overlaps #1025, which is milestoned v5.3) — leaving the milestone as-is per instruction (never change a milestone just to silence the check); flagging here for whoever sets it.

🔧 Changes

• integrationTests/cluster/mixedVersionRollingUpgrade.test.mjs — the suite; its parts are linked from Solution above.
• integrationTests/cluster/fixture-mixed-version/ — config.yaml + schema.graphql for the two tables, copied into each node's components before its first start (not setupHarperWithFixture, which discards the pre-assigned hostname that replication.securePort needs).
• .github/workflows/integration-tests.yaml — the run-mixed-version-tests job. It sits between run-integration-tests and run-copy-gap-runtime-regression because that stretch of the file is identical on main and v5.3, so the milestone's cherry-pick applies cleanly. Next to run-cluster-tests, its first position, the pick conflicted on v5.3: that branch lacks #983's run-replication-pool-regression job and has a different run-cluster-tests name line. Job order has no effect on how the workflow runs. Added on rebase: a second step runs legacyCopyBaseline.test.mjs against the job's already-installed real previous release — this job is the only place in the whole workflow that installs one, so it's the only place that test's real-binary mode can run instead of its default capability-omission simulation. Review caught that the existing guard step didn't cover this new step's own test path, so it now checks both files.
• replication/DESIGN.md — a row for the suite in the integration-test table: when to bump the releases, and (updated on rebase) that the join step converges unconditionally now that Allow base copies to published 5.x peers during rolling upgrades #1025 fixed Rolling upgrade: a node on this build refuses base copies to published 5.x peers (no safeCopyAudit), so they never receive its writes #1007. The rest of that table's diff is prettier re-padding the column plus restoring a residencyConfirmation.test.mjs row I'd accidentally dropped mid-rebase, caught by review.

✅ Verification

• Local, both previous releases, final code: 3 runs × 14 tests, 3/3 pass, 155–164 s each on 4 cores (CI's runner size). Earlier iterations: 9 more full runs, one of which surfaced Rolling upgrade: a node on this build refuses base copies to published 5.x peers (no safeCopyAudit), so they never receive its writes #1007.
• It can fail: with dist patched (not committed) to drop the replicated record c-down-file on receive, the 5.2.15 suite fails at "1 of 3 upgraded" after the 120 s bound — 127.0.0.142: MixedVersionItem c-down-file is missing, the record old A wrote while C was down never reached C after its upgrade — the 5 later steps skip, exit 1.
• HARPER_PRO_PREVIOUS_VERSION_PATHS=/nonexistent/... fails the file (ENOENT, exit 1); unset, the file reports one skipped suite (exit 0).
• Rolling upgrade: a node on this build refuses base copies to published 5.x peers (no safeCopyAudit), so they never receive its writes #1007's 2-node repro (this build + 5.2.15, then + 5.3.1; a scratch script, not committed): the old node has not received the upgraded node's record after 45 s; this build logs Cannot verify replication baseline without numeric update timestamps and closes 1008 on every retry.
• From the node logs: B and A, the second and third nodes to upgrade, logged system.hdb_* already exists; skipping create — replication had created the 5.3.0 system tables on them while they still ran 5.2.15, the path core/upgrade/directives/5-3-0.ts documents. After the re2 rebuild no node logs Could not load component.
• npm run test:unit: 1792 passing (post-rebase). npm run lint and npm run typecheck pass. Prettier applied to the changed files.
• Rebase onto main (this round): confirmed the join step's old branch would time out against the real previous-release binaries before fixing it (ran both 5.2.15 and 5.3.1 suites pre-fix: both hung 30s on the now-unreachable refusal log), then confirmed 7/7 pass on both post-fix. Ran legacyCopyBaseline.test.mjs locally against the real 5.2.15 binary (both rocksdb/lmdb engines, ~58s) to validate the new CI step before pushing.

🤖 Generated with Claude Code

https://claude.ai/code/session_017Hbc5pizrkHrpvfWTCnjWK

_{🤖 Generated by Claude Code (Claude Opus 5.5); posted via @kriszyp.}

Release target: v5.3 — changed code is also on v5.3; overlapping #1025 milestoned v5.3
Related PRs: #1025 overlaps (now merged to main; fixes #1007, see the updated Problem/Solution above), #1026 overlaps (carries #1025's fix for #1007 into the v5.3 release line; no shared changed lines with this PR), #983 overlaps (both edit the jobs in .github/workflows/integration-tests.yaml; no shared behavior), #956 overlaps (both edit replication/DESIGN.md's Tests table; its dropTableOfflinePeer.test.mjs row survives the rebase's re-merge, no shared behavior), #1032 overlaps (both edit DESIGN.md's Tests table; its residencyConfirmation.test.mjs row survives the rebase's re-merge, no shared behavior)
Complexity: medium

Origin — the dispatch brief this PR was written from

Add an integration test for mixed-version 5.x clusters in HarperFast/harper-pro, covering a rolling minor upgrade 5.2 -> 5.3 (and rollback). Today the only cross-version replication coverage is 4.x <-> 5 (cloneFromLegacy, legacyCopyBaseline, replicationTopology 'Replicate data from a legacy node', all via HARPER_LEGACY_VERSION_PATH = harperdb@4). Scenario: start a 3-node cluster on @harperfast/harper-pro@5.2.15 (latest 5.2.x), seed data (plain, indexed, audit-enabled table; include a blob attribute), then roll the nodes one at a time to the current build on the SAME data dirs (kill, restart with current binary). At each intermediate state (1 of 3 upgraded, 2 of 3 upgraded) write on BOTH an old-version and a new-version node and assert bidirectional replication converges on every node (record sets equal, indexed search resolves, blobs readable). After all nodes are upgraded, assert convergence and that a cold restart keeps it. Then roll ONE node back to 5.2.15 (CONFIRM_DOWNGRADE=yes) and assert it rejoins and replication still converges both ways. Also run from @harperfast/harper-pro@5.3.1 -> current if cheap (main is 5.4, so 5.3 -> 5.4 is the N-1 path on main).

Acceptance

New suite runs (not skipped) in harper-pro CI from 5.2.15 and passes, covering: partial-upgrade bidirectional replication at 1/3 and 2/3 upgraded, full-upgrade convergence + cold restart, and one-node rollback rejoin. PR body shows it can fail (e.g. assertions catch a deliberately dropped replicated write). Uses bounded polling, not fixed sleeps. No production code changes unless a small real defect fix; otherwise file an issue.

_{Dispatch: task harper-pro-mixed-version-replication · queued by kris-session · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=15; full=8 @ 5fd8435}

_{Review-Attention: study ~20m (sensitive: integration-tests.yaml; decisions: convergence-only-oracle, disjoint-concurrent-writes, rollback-as-contract, pinned-release-matrix, runs-on-every-pr) @ 5fd8435}

[github-actions]

Release cherry-pick v5.3: cancelled

Cherry-pick branch cherry-pick/v5.3/pr-1008 was deleted — this PR no longer targets v5.3 (milestone is now v5.4).