test(storage): start the ART reproducer from the real pre-v10 schema

CHANGELOG.md

@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2, without `user_id` and with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts the reopen succeeds, both checkpoints fold, all four indexes survive, and an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it, which is the silent form of the damage: a wrong answer rather than an error. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.7 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))
+- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2 — without `user_id`, and *with* the mid-table `duration_ms` generated column, so that the one ALTER production ran on a long-lived filled table (v9 → v10, six weeks before the abort) runs here too instead of passing over a column the fixture never had, and so the indexes are built over the logical/physical column skew they were built over in production — with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts that `duration_ms` was there before the migration and is gone after it (a `DROP COLUMN IF EXISTS` that no-ops must not be able to come back unnoticed), that the reopen succeeds, that both checkpoints fold — a present-but-empty `.wal` counts as folded, so an engine that truncates rather than unlinks is not reported as this defect — that all four indexes survive, that a bare `user_id = <constant>` folded into the scan agrees with the same predicate wrapped in an expression, which is not, and that re-inserting a stored `span_id` is still rejected while an unused one is accepted. The last two are the silent form of the damage: a wrong answer rather than an error. They are phrased that way because no `SELECT` over `spans` is planned as an ART index scan on this engine — measured, including on the primary key and on a textbook unique match with the thresholds raised — so an “indexed probe agrees with an unindexed probe” assertion would have been two sequential scans agreeing; the pushdown pair is what [ADR-0013](docs/decisions/0013-spans-has-no-derived-columns.md)'s wrong answers actually came out of, and unique-constraint enforcement is the one path that provably traverses an ART index. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.1 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md))
 
 ### Changed
 - `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))

docs/operations/duckdb-recovery.md

@@ -368,25 +368,51 @@ exactly the moment nobody remembers an optional test.
 One pass is:
 
 1. build `spans` as it stood at schema version 2 — every column it has today except
-   `user_id` — with that era's three ART indexes, and fill it with spans spread over
-   60 days, i.e. across the 30-day raw-retention boundary;
+   `user_id`, plus the mid-table `duration_ms` generated column that only v10
+   removed — with that era's three ART indexes, and fill it with spans spread over
+   60 days, i.e. across the 30-day raw-retention boundary. Both differences carry a
+   migration, and the second one carries the migration that matters: `DROP COLUMN IF
+   EXISTS duration_ms` over a column the fixture never had is a silent no-op, and
+   v10 is the ALTER production ran on a long-lived filled table six weeks before the
+   abort, where v2 → v3 ran the day after the database was created. While the column
+   is there the table also carries the logical/physical column skew of
+   [ADR-0013](../decisions/0013-spans-has-no-derived-columns), so the indexes are
+   built over the layout production built them over;
 2. open it with `storage.Open`, so the real `schema.sql` runs the real
-   `ALTER TABLE spans ADD COLUMN user_id VARCHAR` against a filled, indexed table and
-   then builds `idx_spans_user_id` on that added column;
+   `ALTER TABLE spans ADD COLUMN user_id VARCHAR` against a filled, indexed table,
+   drops `duration_ms` out of the middle of it, and then builds `idx_spans_user_id`
+   on the added column;
 3. write the added column for every row with a bulk `UPDATE`, then run cotel's own
    retention roll-up as the large replace transaction — the `INSERT INTO daily_usage`
    and the `DELETE FROM spans` are one transaction, so this is the real mutation and
    not an imitation of it;
 4. checkpoint, close, reopen, checkpoint again, ingest a further batch, and repeat.
 
-Each pass asserts four things, and they fail for different reasons:
+Each pass asserts the following, and they fail for different reasons:
 
 | Assertion | What a failure means |
 |---|---|
+| `duration_ms` is there before the migration and gone after it | the pass never exercised v10's `ALTER TABLE spans DROP COLUMN` — either the fixture stopped being pre-v10, in which case the drop was a no-op, or the drop ran and left the column |
 | the reopen succeeds | the 1.1.3 `abort()` inside `duckdb_open_ext`, which 1.5.6 raises as a Go error instead |
-| both checkpoints fold | the index fold refused — the same signal the startup and shutdown checkpoints give in production, and the earliest honest warning there is |
+| both checkpoints fold | the index fold refused — the same signal the startup and shutdown checkpoints give in production, and the earliest honest warning there is. A `.wal` file that is present but zero bytes counts as folded, so an engine that truncates rather than unlinks is not reported as this defect |
 | all four indexes are present after the restart | an index was lost rather than damaged |
-| an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it | the silent form of this damage: a wrong answer, which no checkpoint and no open reports |
+| a bare `user_id = <constant>`, which the optimizer folds into the scan, agrees with the same predicate wrapped in an expression, which it does not | the silent form of this damage: a wrong answer, which no checkpoint and no open reports. It is [ADR-0013](../decisions/0013-spans-has-no-derived-columns)'s symptom exactly — a pushed-down equality answered against the wrong column offset, matching nothing |
+| re-inserting a stored `span_id` is rejected, and inserting an unused one is accepted | the primary key's unique ART index has lost its own entries, or gained phantom ones. This is the only assertion in the list that provably traverses an ART index — see below |
+
+**The equality pair compares two sequential scans, not an index against a scan.** On
+the linked engine, no `SELECT` over `spans` is planned as an ART index scan: `EXPLAIN`
+answers `SEQ_SCAN` / `Type: Sequential Scan` for `user_id = <constant>` and for a
+bound `user_id = ?` alike, for an equality on the primary key, and for the textbook
+case — a unique match in a 100 000-row table with a fresh index on exactly the probed
+column — with `index_scan_max_count` and `index_scan_percentage` raised to admit
+everything, and with and without the `storage_compatibility_version` pin cotel opens
+with. So an assertion phrased as *indexed probe agrees with unindexed probe* would be
+two sequential scans agreeing with each other. What the pair does compare is filter
+pushdown against no pushdown, which is the path ADR-0013's wrong answers came out of;
+the test asserts that the two probes still plan differently, so the comparison cannot
+quietly degenerate into one path checked twice. The unique-constraint probe is the
+check that reaches an ART index, because conflict detection consults the primary key's
+index on the way in.
 
 Size and repetitions are tunable, and turning them up is the first thing to do when
 the verdict is "did not reproduce":
@@ -410,13 +436,23 @@ docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp/h -e GOCACHE=/go/.cache/go-b
 On 2026-10-13, on linux/arm64 under `golang:1.24`, against the engine the tree links
 today — `duckdb-go/v2 v2.10506.0`, with `SELECT version()` answering `v1.5.6`:
 
-| Parameters | Result |
-|---|---|
-| 20 000 spans, 2 cycles (the defaults, so `go test ./...` runs this) | did not reproduce, 4.7 s |
-| 200 000 spans, 6 cycles — 8 opens, 14 checkpoints, the table peaking at 407 763 rows | did not reproduce, 3 m 40 s |
+| Fixture | Parameters | Result |
+|---|---|---|
+| pre-v3: `duration_ms` absent, so the only ALTER exercised was v2 → v3 and v10's `DROP COLUMN` was a silent no-op | 20 000 spans, 2 cycles | did not reproduce, 4.7 s |
+| pre-v3, as above | 200 000 spans, 6 cycles — 8 opens, 14 checkpoints, the table peaking at 407 763 rows | did not reproduce, 3 m 40 s |
+| pre-v10: `duration_ms` present and dropped by the migration under test, the indexes built over the column skew | 20 000 spans, 2 cycles (the defaults, so `go test ./...` runs this) | did not reproduce, 4.1 s |
+| pre-v10, as above | 200 000 spans, 6 cycles — 8 opens, 14 checkpoints, the table peaking at 407 956 rows | did not reproduce, 2 m 58 s |
+
+The first two rows are kept because they are what was run, not because they cover
+what they were taken to cover: their fixture never had `duration_ms`, so the v10
+`ALTER TABLE spans DROP COLUMN` did nothing, and the only ALTER those two runs put
+under an indexed filled table was v2 → v3 — which in production ran the day after
+the database was created, on an almost empty table. The bottom two rows are the same
+two sizes over the corrected fixture, so they are the ones that exercised the ALTER
+production ran six weeks before the abort. No run has reproduced anything.
 
 **This is not a demonstration that 1.5.6 is free of the defect, and must not be
-quoted as one.** It is two negative runs at one scale on one architecture. Three
+quoted as one.** It is four negative runs at two scales on one architecture. Three
 things bound what they cover:
 
 - The production file that was damaged held ~34 700 spans in 147 MB and took weeks