test(storage): synthetic reproducer for the ART-index corruption

[Fl0p]

What

A synthetic reproducer for the ART-index corruption that took production down on 2026-09-28, plus the recovery page's verdict on the half of the forensic volume's release condition that nobody could evaluate.

internal/storage/art_repro_test.go rebuilds the trigger from the schema, with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone:

1. spans as it stood at schema version 2 (no user_id) with that era's three ART indexes, filled with spans spread over 60 days, i.e. across the 30-day retention boundary;
2. storage.Open, so the real schema.sql runs the real ALTER TABLE spans ADD COLUMN user_id VARCHAR against a filled, indexed table and then builds idx_spans_user_id on that added column;
3. a bulk UPDATE writes the added column, then cotel's own retention roll-up supplies the large replace transaction — the INSERT INTO daily_usage and the DELETE FROM spans are one transaction, so it is the real mutation and not an imitation;
4. checkpoint, close, reopen, checkpoint, ingest another batch, repeat.

The repetition and the restart are the test, not scaffolding: the damage accumulates invisibly, and this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts the reopen succeeds, both checkpoints fold, all four indexes survive, and an indexed user_id = ? agrees with the same predicate written so the index cannot serve it — the last one covering the silent form of this damage, a wrong answer rather than an error.

Result

Against the engine the tree links today (duckdb-go/v2 v2.10506.0, SELECT version() → v1.5.6), on linux/arm64 under golang:1.24:

Parameters	Result
20 000 spans, 2 cycles (the defaults)	did not reproduce, 4.7 s
200 000 spans, 6 cycles — 8 opens, 14 checkpoints, table peaking at 407 763 rows	did not reproduce, 3 m 40 s

The docs state this as a measurement, not an exoneration: two negative runs at one scale on one architecture, against a production failure that took weeks of ingest and an upstream report of the same class on 1.5.5 at ~19 GB. The release condition now reads not shown to reproduce at our scale on 1.5.6, which is strictly weaker than fixed, and a clean run does not release the volume.

Decision worth a second opinion

The ticket allowed a long/optional build tag "if it is slow". At the defaults it costs 4.7 s, so it is not tagged — it runs in go test ./..., which means a driver or engine bump exercises it automatically. COTEL_ART_REPRO_SPANS / COTEL_ART_REPRO_CYCLES scale it up for a deliberate hunt. If you would rather it never ran in CI, a //go:build artrepro tag is a one-line change.

Not done, deliberately

• Nothing published upstream. The reproducer is publishable by construction, but sending it is a board decision.
• The forensic volume was not touched, mounted or read.
• No attempt to fix the defect or move the engine.

Verified

• CGO_ENABLED=1 go vet ./... — clean
• CGO_ENABLED=1 go test ./... — all packages green, internal/storage 11.0 s
• npm run build in docs/ — VitePress site builds (the Docs workflow runs on this PR)
• the two reproducer runs in the table above

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 469d3496-054b-456f-92a6-cabbb2a473a0

📥 Commits

Reviewing files that changed from the base of the PR and between 363da1f and efd4054.

📒 Files selected for processing: 4

• CHANGELOG.md
• README.md
• docs/operations/duckdb-recovery.md
• internal/storage/art_repro_test.go

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}