test(storage): synthetic reproducer for the ART-index corruption

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2, without `user_id` and with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts the reopen succeeds, both checkpoints fold, all four indexes survive, and an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it, which is the silent form of the damage: a wrong answer rather than an error. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.7 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))
+
 ### Changed
 - `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))
 

README.md

@@ -555,6 +555,8 @@ from the network at startup instead of bundling it).
 | `SNAPSHOT_MIRROR_KEEP` | `56` | Read by `scripts/pull-snapshots.sh`: how many newest complete snapshots the mirror keeps. Never below one, and never a deletion made because the source stopped having a snapshot. |
 | `SNAPSHOT_HELPER_IMAGE` | `debian:bookworm-slim` | Read by `scripts/pull-snapshots.sh`: the image the throwaway read-only container runs. It needs `tar`, `find` and `sha256sum`; cotel's own runtime base already has them and is already on the deploy host. |
 | `SNAPSHOT_SSH_TIMEOUT` | `30` | Read by `scripts/pull-snapshots.sh`: ssh connect timeout and keepalive interval in seconds, so a tick against a dead host fails rather than hangs. |
+| `COTEL_ART_REPRO_SPANS` | `20000` | Read by the ART-corruption reproducer test, not by the binary: how many synthetic spans it builds before the migration under test. The default keeps the test inside the ordinary `go test ./...` budget; raise it when hunting the defect. See [the reproducer](docs/operations/duckdb-recovery.md#the-synthetic-reproducer). |
+| `COTEL_ART_REPRO_CYCLES` | `2` | Read by the ART-corruption reproducer test, not by the binary: how many replace-transaction-then-restart cycles it runs. The damage accumulates across cycles, so this is the other knob to raise. |
 | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. |
 
 ## Architecture

docs/operations/duckdb-recovery.md

@@ -344,16 +344,97 @@ release conditions have to hold, and each one has a trap:
   That would satisfy the words "the issue is resolved" while the defect is still
   live, so it does not release this volume. Read the closing commit, not the state.
 - **A DuckDB in cotel shown not to hit it.** Today cotel links
-  `duckdb-go/v2 v2.10506.0` (DuckDB 1.5.6) and the crash was on 1.1.3; nobody has
-  demonstrated either way, so this half is currently unevaluable rather than
-  satisfied. Only a reproducer settles it, which is the same reason a version bump
-  is the natural moment to build one.
+  `duckdb-go/v2 v2.10506.0` (DuckDB 1.5.6) and the crash was on 1.1.3. This half
+  used to be unevaluable — nobody had tried either way. It is now measured, and the
+  measurement is negative rather than exonerating: see [The synthetic
+  reproducer](#the-synthetic-reproducer) below. A run that does not reproduce does
+  not release this volume; a run that does is an incident.
 
 Never attach this volume, or anything extracted from it, to a public upstream issue:
 it is production telemetry with real user ids, session ids and span attributes. An
 upstream contribution has to be a synthetic reproducer built from the schema above,
 and publishing one is an external-facing decision, not an operational one.
 
+### The synthetic reproducer
+
+`TestARTIndexSurvivesAddedColumnAndBulkReplace` in
+`internal/storage/art_repro_test.go` runs the sequence above against whatever engine
+the tree links, built from the schema and nothing else — every value is derived from
+the row number, so it carries no user id, session id or span attribute that belonged
+to anyone, and it can be run anywhere. It is in the ordinary test set, not behind a
+tag, because the moment it most needs to run is a driver or engine bump, and that is
+exactly the moment nobody remembers an optional test.
+
+One pass is:
+
+1. build `spans` as it stood at schema version 2 — every column it has today except
+   `user_id` — with that era's three ART indexes, and fill it with spans spread over
+   60 days, i.e. across the 30-day raw-retention boundary;
+2. open it with `storage.Open`, so the real `schema.sql` runs the real
+   `ALTER TABLE spans ADD COLUMN user_id VARCHAR` against a filled, indexed table and
+   then builds `idx_spans_user_id` on that added column;
+3. write the added column for every row with a bulk `UPDATE`, then run cotel's own
+   retention roll-up as the large replace transaction — the `INSERT INTO daily_usage`
+   and the `DELETE FROM spans` are one transaction, so this is the real mutation and
+   not an imitation of it;
+4. checkpoint, close, reopen, checkpoint again, ingest a further batch, and repeat.
+
+Each pass asserts four things, and they fail for different reasons:
+
+| Assertion | What a failure means |
+|---|---|
+| the reopen succeeds | the 1.1.3 `abort()` inside `duckdb_open_ext`, which 1.5.6 raises as a Go error instead |
+| both checkpoints fold | the index fold refused — the same signal the startup and shutdown checkpoints give in production, and the earliest honest warning there is |
+| all four indexes are present after the restart | an index was lost rather than damaged |
+| an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it | the silent form of this damage: a wrong answer, which no checkpoint and no open reports |
+
+Size and repetitions are tunable, and turning them up is the first thing to do when
+the verdict is "did not reproduce":
+
+```bash
+COTEL_ART_REPRO_SPANS=200000 COTEL_ART_REPRO_CYCLES=6 \
+  CGO_ENABLED=1 go test -timeout 60m -v -run TestARTIndex ./internal/storage/
+```
+
+There is no Go toolchain on the Pi, so a run there goes through the image:
+
+```bash
+docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp/h -e GOCACHE=/go/.cache/go-build \
+  -e CGO_ENABLED=1 -e COTEL_ART_REPRO_SPANS=200000 -e COTEL_ART_REPRO_CYCLES=6 \
+  -v "$HOME/go":/go -v "$PWD":/src -w /src golang:1.24 \
+  go test -timeout 60m -v -run TestARTIndex ./internal/storage/
+```
+
+#### What has been run, and what it does not say
+
+On 2026-10-13, on linux/arm64 under `golang:1.24`, against the engine the tree links
+today — `duckdb-go/v2 v2.10506.0`, with `SELECT version()` answering `v1.5.6`:
+
+| Parameters | Result |
+|---|---|
+| 20 000 spans, 2 cycles (the defaults, so `go test ./...` runs this) | did not reproduce, 4.7 s |
+| 200 000 spans, 6 cycles — 8 opens, 14 checkpoints, the table peaking at 407 763 rows | did not reproduce, 3 m 40 s |
+
+**This is not a demonstration that 1.5.6 is free of the defect, and must not be
+quoted as one.** It is two negative runs at one scale on one architecture. Three
+things bound what they cover:
+
+- The production file that was damaged held ~34 700 spans in 147 MB and took weeks
+  of ingest and roll-ups after the migration before an open aborted. The runs above
+  compress that into minutes; whatever the real trigger needs may need wall-clock or
+  a write pattern this sequence does not have.
+- duckdb#25360 reports the same class of fault on 1.5.5 against a ~19 GB database,
+  three orders of magnitude larger than anything run here. Sheer size may be part of
+  the trigger.
+- The damage accumulates invisibly. A pass that ends clean proves the index folded
+  and read back correctly *that time*, which is the strongest statement available
+  and is still not absence.
+
+So the second release condition now reads: *not shown to reproduce at our scale on
+1.5.6*, which is strictly weaker than *fixed*. Re-run the reproducer — heavier than
+the defaults — whenever the linked engine moves, and record the result here next to
+these rows rather than replacing them.
+
 ### The leftovers are not the backup
 
 The three volumes a recovery leaves look like redundancy and are not. Two of them