diff --git a/CHANGELOG.md b/CHANGELOG.md index 3358024..55e8345 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- A synthetic reproducer for the ART-index corruption that took production down in September 2026: `TestARTIndexSurvivesAddedColumnAndBulkReplace` in `internal/storage/art_repro_test.go`, built from the schema with no production bytes — every value is derived from the row number, so it carries no user id, session id or span attribute that belonged to anyone. It rebuilds the trigger rather than a statement from it: `spans` as it stood at schema version 2, without `user_id` and with that era's three ART indexes, filled across the 30-day retention boundary; `storage.Open` then runs the real `schema.sql`, which adds `user_id VARCHAR` to the filled indexed table and puts `idx_spans_user_id` on that added column; a bulk `UPDATE` writes the column, cotel's own retention roll-up supplies the large replace transaction (the `INSERT INTO daily_usage` and the `DELETE FROM spans` are one transaction, so it is the real mutation and not an imitation); then checkpoint, close, reopen, checkpoint, ingest more, repeat. The repetition is the point — the damage accumulates and production ran for weeks after its migration before an open aborted — and so is the restart: this class of fault is only ever reported by the checkpoint that folds the index or the open that reads it back. Each pass asserts the reopen succeeds, both checkpoints fold, all four indexes survive, and an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it, which is the silent form of the damage: a wrong answer rather than an error. It runs in the ordinary test set rather than behind a build tag, because the moment it most needs to run is a driver or engine bump and that is exactly when an optional test is forgotten; `COTEL_ART_REPRO_SPANS` and `COTEL_ART_REPRO_CYCLES` scale it past the defaults (20 000 spans, 2 cycles, 4.7 s). Against the engine the tree links today — `duckdb-go/v2 v2.10506.0`, `SELECT version()` → `v1.5.6` — it did **not** reproduce, at the defaults or at 200 000 spans over 6 cycles. That measures the half of the forensic volume's release condition that was previously unevaluable, and measures it negative rather than exonerating: upstream reports the same class of fault on 1.5.5 against a ~19 GB database, the file that was damaged here took weeks of ingest to get there, and two clean runs at our scale are not absence ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md)) + ### Changed - `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md)) diff --git a/README.md b/README.md index 9b1ddb2..7733c2b 100644 --- a/README.md +++ b/README.md @@ -555,6 +555,8 @@ from the network at startup instead of bundling it). | `SNAPSHOT_MIRROR_KEEP` | `56` | Read by `scripts/pull-snapshots.sh`: how many newest complete snapshots the mirror keeps. Never below one, and never a deletion made because the source stopped having a snapshot. | | `SNAPSHOT_HELPER_IMAGE` | `debian:bookworm-slim` | Read by `scripts/pull-snapshots.sh`: the image the throwaway read-only container runs. It needs `tar`, `find` and `sha256sum`; cotel's own runtime base already has them and is already on the deploy host. | | `SNAPSHOT_SSH_TIMEOUT` | `30` | Read by `scripts/pull-snapshots.sh`: ssh connect timeout and keepalive interval in seconds, so a tick against a dead host fails rather than hangs. | +| `COTEL_ART_REPRO_SPANS` | `20000` | Read by the ART-corruption reproducer test, not by the binary: how many synthetic spans it builds before the migration under test. The default keeps the test inside the ordinary `go test ./...` budget; raise it when hunting the defect. See [the reproducer](docs/operations/duckdb-recovery.md#the-synthetic-reproducer). | +| `COTEL_ART_REPRO_CYCLES` | `2` | Read by the ART-corruption reproducer test, not by the binary: how many replace-transaction-then-restart cycles it runs. The damage accumulates across cycles, so this is the other knob to raise. | | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. | ## Architecture diff --git a/docs/operations/duckdb-recovery.md b/docs/operations/duckdb-recovery.md index 9a4fcd6..24eea04 100644 --- a/docs/operations/duckdb-recovery.md +++ b/docs/operations/duckdb-recovery.md @@ -344,16 +344,97 @@ release conditions have to hold, and each one has a trap: That would satisfy the words "the issue is resolved" while the defect is still live, so it does not release this volume. Read the closing commit, not the state. - **A DuckDB in cotel shown not to hit it.** Today cotel links - `duckdb-go/v2 v2.10506.0` (DuckDB 1.5.6) and the crash was on 1.1.3; nobody has - demonstrated either way, so this half is currently unevaluable rather than - satisfied. Only a reproducer settles it, which is the same reason a version bump - is the natural moment to build one. + `duckdb-go/v2 v2.10506.0` (DuckDB 1.5.6) and the crash was on 1.1.3. This half + used to be unevaluable — nobody had tried either way. It is now measured, and the + measurement is negative rather than exonerating: see [The synthetic + reproducer](#the-synthetic-reproducer) below. A run that does not reproduce does + not release this volume; a run that does is an incident. Never attach this volume, or anything extracted from it, to a public upstream issue: it is production telemetry with real user ids, session ids and span attributes. An upstream contribution has to be a synthetic reproducer built from the schema above, and publishing one is an external-facing decision, not an operational one. +### The synthetic reproducer + +`TestARTIndexSurvivesAddedColumnAndBulkReplace` in +`internal/storage/art_repro_test.go` runs the sequence above against whatever engine +the tree links, built from the schema and nothing else — every value is derived from +the row number, so it carries no user id, session id or span attribute that belonged +to anyone, and it can be run anywhere. It is in the ordinary test set, not behind a +tag, because the moment it most needs to run is a driver or engine bump, and that is +exactly the moment nobody remembers an optional test. + +One pass is: + +1. build `spans` as it stood at schema version 2 — every column it has today except + `user_id` — with that era's three ART indexes, and fill it with spans spread over + 60 days, i.e. across the 30-day raw-retention boundary; +2. open it with `storage.Open`, so the real `schema.sql` runs the real + `ALTER TABLE spans ADD COLUMN user_id VARCHAR` against a filled, indexed table and + then builds `idx_spans_user_id` on that added column; +3. write the added column for every row with a bulk `UPDATE`, then run cotel's own + retention roll-up as the large replace transaction — the `INSERT INTO daily_usage` + and the `DELETE FROM spans` are one transaction, so this is the real mutation and + not an imitation of it; +4. checkpoint, close, reopen, checkpoint again, ingest a further batch, and repeat. + +Each pass asserts four things, and they fail for different reasons: + +| Assertion | What a failure means | +|---|---| +| the reopen succeeds | the 1.1.3 `abort()` inside `duckdb_open_ext`, which 1.5.6 raises as a Go error instead | +| both checkpoints fold | the index fold refused — the same signal the startup and shutdown checkpoints give in production, and the earliest honest warning there is | +| all four indexes are present after the restart | an index was lost rather than damaged | +| an indexed `user_id = ?` agrees with the same predicate written so the index cannot serve it | the silent form of this damage: a wrong answer, which no checkpoint and no open reports | + +Size and repetitions are tunable, and turning them up is the first thing to do when +the verdict is "did not reproduce": + +```bash +COTEL_ART_REPRO_SPANS=200000 COTEL_ART_REPRO_CYCLES=6 \ + CGO_ENABLED=1 go test -timeout 60m -v -run TestARTIndex ./internal/storage/ +``` + +There is no Go toolchain on the Pi, so a run there goes through the image: + +```bash +docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp/h -e GOCACHE=/go/.cache/go-build \ + -e CGO_ENABLED=1 -e COTEL_ART_REPRO_SPANS=200000 -e COTEL_ART_REPRO_CYCLES=6 \ + -v "$HOME/go":/go -v "$PWD":/src -w /src golang:1.24 \ + go test -timeout 60m -v -run TestARTIndex ./internal/storage/ +``` + +#### What has been run, and what it does not say + +On 2026-10-13, on linux/arm64 under `golang:1.24`, against the engine the tree links +today — `duckdb-go/v2 v2.10506.0`, with `SELECT version()` answering `v1.5.6`: + +| Parameters | Result | +|---|---| +| 20 000 spans, 2 cycles (the defaults, so `go test ./...` runs this) | did not reproduce, 4.7 s | +| 200 000 spans, 6 cycles — 8 opens, 14 checkpoints, the table peaking at 407 763 rows | did not reproduce, 3 m 40 s | + +**This is not a demonstration that 1.5.6 is free of the defect, and must not be +quoted as one.** It is two negative runs at one scale on one architecture. Three +things bound what they cover: + +- The production file that was damaged held ~34 700 spans in 147 MB and took weeks + of ingest and roll-ups after the migration before an open aborted. The runs above + compress that into minutes; whatever the real trigger needs may need wall-clock or + a write pattern this sequence does not have. +- duckdb#25360 reports the same class of fault on 1.5.5 against a ~19 GB database, + three orders of magnitude larger than anything run here. Sheer size may be part of + the trigger. +- The damage accumulates invisibly. A pass that ends clean proves the index folded + and read back correctly *that time*, which is the strongest statement available + and is still not absence. + +So the second release condition now reads: *not shown to reproduce at our scale on +1.5.6*, which is strictly weaker than *fixed*. Re-run the reproducer — heavier than +the defaults — whenever the linked engine moves, and record the result here next to +these rows rather than replacing them. + ### The leftovers are not the backup The three volumes a recovery leaves look like redundancy and are not. Two of them diff --git a/internal/storage/art_repro_test.go b/internal/storage/art_repro_test.go new file mode 100644 index 0000000..eab2e85 --- /dev/null +++ b/internal/storage/art_repro_test.go @@ -0,0 +1,355 @@ +// Synthetic reproducer for the ART-index corruption that took production down +// on 2026-09-28 and still has no upstream fix. It is built from the schema +// alone — no production bytes — so it can be run anywhere and, if the board +// ever decides to, published. +// +// The trigger, as reported upstream and as it happened here, is a sequence +// rather than a single statement: a long-lived table carrying ART indexes gets +// a VARCHAR column added by ALTER TABLE, an index is built on that added +// column, and large replace transactions then mutate it. The damage does not +// surface at write time, and it accumulates — production ran for weeks after +// the migration before an open aborted. So the sequence here is a loop, and +// every pass through it ends in a full stop and start: the checkpoint that +// folds the index on stop, and the open that reads it back, are the only two +// places this class of damage is ever reported. +// +// Size and repetitions are tunable, and turning them up is the first thing to +// try when the verdict is "did not reproduce" — a negative result at one size +// is evidence, not a proof of absence: +// +// COTEL_ART_REPRO_SPANS=200000 COTEL_ART_REPRO_CYCLES=6 \ +// CGO_ENABLED=1 go test -timeout 60m -v -run TestARTIndex ./internal/storage/ + +package storage + +import ( + "context" + "database/sql" + "fmt" + "os" + "path/filepath" + "strconv" + "testing" + "time" +) + +// defaultReproSpans / defaultReproCycles keep the test inside the ordinary +// `go test ./...` budget so every engine or driver bump runs it without anyone +// remembering to. The production table that was damaged held ~34 700 spans, so +// a deliberate hunt should raise these well past the defaults. +const ( + defaultReproSpans = 20_000 + defaultReproCycles = 2 +) + +// reproSpanDays spreads the generated spans over more than the 30-day raw +// retention window, so each replace transaction has both rows to roll up and +// rows to keep. A roll-up that moves nothing exercises nothing. +const reproSpanDays = 60 + +// reproUsers is how many distinct values the index on the added column gets. +// Low cardinality over many rows is what gives the ART long shared prefixes and +// deep nodes, i.e. the structure the defect damages. +const reproUsers = 12 + +// preUserIDSchema is `spans` as it stood at schema version 2: every column the +// table has today except user_id, with the three ART indexes of that era. The +// point is to reach the ALTER from a table that was built and filled without +// the column, which is the state production was in. +const preUserIDSchema = ` +CREATE TABLE schema_version ( + version INTEGER PRIMARY KEY, + applied_at TIMESTAMPTZ DEFAULT now() +); +INSERT INTO schema_version (version) VALUES (1), (2); + +CREATE TABLE spans ( + trace_id VARCHAR NOT NULL, + span_id VARCHAR NOT NULL PRIMARY KEY, + parent_span_id VARCHAR, + name VARCHAR NOT NULL, + start_time TIMESTAMPTZ NOT NULL, + end_time TIMESTAMPTZ NOT NULL, + service_name VARCHAR, + session_id VARCHAR, + model VARCHAR, + tool_name VARCHAR, + status_code TINYINT DEFAULT 0, + input_tokens INTEGER, + output_tokens INTEGER, + cache_read_tokens INTEGER, + cache_write_tokens INTEGER, + cost_usd DOUBLE, + attributes JSON, + resource_attrs JSON, + ingested_at TIMESTAMPTZ DEFAULT now() +); + +CREATE INDEX idx_spans_session_id ON spans(session_id); +CREATE INDEX idx_spans_start_time ON spans(start_time); +CREATE INDEX idx_spans_name ON spans(name); +` + +// seedSpans inserts a batch in one statement. Values are derived from the row +// number, never from a captured payload: the reproducer must carry no user id, +// session id or span attribute that ever belonged to anyone. Parameters are +// (now, span-days, first row number, last row number). +const seedSpans = ` +INSERT INTO spans ( + trace_id, span_id, parent_span_id, name, + start_time, end_time, service_name, session_id, model, tool_name, + status_code, input_tokens, output_tokens, + cache_read_tokens, cache_write_tokens, cost_usd, + attributes, resource_attrs) +SELECT + 'trace-' || (i // 7)::VARCHAR, + 'span-' || i::VARCHAR, + CASE WHEN i % 7 = 0 THEN NULL ELSE 'span-' || ((i // 7) * 7)::VARCHAR END, + 'span.name.' || (i % 17)::VARCHAR, + ts, + ts + INTERVAL 250 MILLISECOND, + 'synthetic-producer', + 'session-' || (i // 500)::VARCHAR, + 'model-' || (i % 3)::VARCHAR, + CASE WHEN i % 5 = 0 THEN NULL ELSE 'tool-' || (i % 11)::VARCHAR END, + CASE WHEN i % 23 = 0 THEN 2 ELSE 0 END, + i % 4096, i % 512, i % 8192, i % 256, (i % 97) / 1000.0, + '{"synthetic":' || i::VARCHAR || '}', + '{"synthetic.resource":' || (i % 3)::VARCHAR || '}' +FROM ( + SELECT i, CAST(? AS TIMESTAMPTZ) - to_seconds((i % ?) * 86400 + (i * 37) % 86400) AS ts + FROM range(?, ?) t(i) +) s +` + +// populateAddedColumn writes the added VARCHAR column for every row that has no +// value yet. This is the mutation the defect needs: a bulk write through an ART +// index that was built on a column the table did not originally have. +var populateAddedColumn = fmt.Sprintf( + `UPDATE spans SET user_id = 'user-' || (hash(span_id) %% %d)::VARCHAR WHERE user_id IS NULL`, + reproUsers) + +func envCount(t *testing.T, key string, def int) int { + t.Helper() + raw := os.Getenv(key) + if raw == "" { + return def + } + n, err := strconv.Atoi(raw) + if err != nil || n < 1 { + t.Fatalf("%s=%q: want a positive integer", key, raw) + } + return n +} + +// openRaw opens the file with the same DSN Open builds, so the on-disk format +// of the seeded database matches what cotel would have written. +func openRaw(t *testing.T, path string) *sql.DB { + t.Helper() + db, err := sql.Open("duckdb", path+"?storage_compatibility_version="+StorageCompatibilityVersion) + if err != nil { + t.Fatalf("open raw: %v", err) + } + db.SetMaxOpenConns(1) + return db +} + +func mustExec(t *testing.T, db *sql.DB, what, query string, args ...any) { + t.Helper() + start := time.Now() + if _, err := db.Exec(query, args...); err != nil { + t.Fatalf("%s: %v", what, err) + } + t.Logf("%s: ok in %s", what, time.Since(start).Round(time.Millisecond)) +} + +func indexNames(t *testing.T, ro *ReadDB) []string { + t.Helper() + rows, err := ro.Query(`SELECT index_name FROM duckdb_indexes() ORDER BY index_name`) + if err != nil { + t.Fatalf("duckdb_indexes(): %v", err) + } + defer rows.Close() + var names []string + for rows.Next() { + var n string + if err := rows.Scan(&n); err != nil { + t.Fatalf("scan index name: %v", err) + } + names = append(names, n) + } + if err := rows.Err(); err != nil { + t.Fatalf("iterate indexes: %v", err) + } + return names +} + +// TestARTIndexSurvivesAddedColumnAndBulkReplace runs the production sequence +// end to end against whatever engine the tree currently links, and fails if the +// database stops opening, an index goes missing, a checkpoint refuses to fold, +// or an indexed equality starts disagreeing with the same predicate written so +// the index cannot serve it. The last check is the one that catches the quiet +// variant of this damage: a wrong answer rather than an error. +func TestARTIndexSurvivesAddedColumnAndBulkReplace(t *testing.T) { + spans := envCount(t, "COTEL_ART_REPRO_SPANS", defaultReproSpans) + cycles := envCount(t, "COTEL_ART_REPRO_CYCLES", defaultReproCycles) + path := filepath.Join(t.TempDir(), "art-repro.duckdb") + ctx := context.Background() + now := time.Now().UTC() + + // Phase 1 — a long-lived, indexed, pre-migration database. + var engine string + seed := openRaw(t, path) + if err := seed.QueryRow("SELECT version()").Scan(&engine); err != nil { + t.Fatalf("version(): %v", err) + } + t.Logf("engine %s: %d spans over %d days, then %d replace/restart cycles", engine, spans, reproSpanDays, cycles) + mustExec(t, seed, "pre-migration schema", preUserIDSchema) + mustExec(t, seed, fmt.Sprintf("seed %d spans", spans), seedSpans, now, reproSpanDays, 0, spans) + mustExec(t, seed, "seed checkpoint", "CHECKPOINT") + if err := seed.Close(); err != nil { + t.Fatalf("close seed: %v", err) + } + requireNoWAL(t, path, "after seeding") + + // Phase 2 — the migration, as cotel performs it on the next start: + // schema.sql adds user_id VARCHAR to the filled table, then builds an ART + // index on that added column. + db, err := Open(path) + if err != nil { + t.Fatalf("REPRODUCED: the open that runs the migration failed: %v", err) + } + defer func() { + if db != nil { + db.Close() + } + }() + if v, ok := appliedSchemaVersion(db.rw); !ok || v < 10 { + t.Fatalf("schema did not migrate: applied version %d (recorded=%v)", v, ok) + } + t.Logf("indexes after migration: %v", indexNames(t, db.ReadOnly())) + + // The startup checkpoint cotel runs immediately after schema apply: the + // earliest point at which a migration that damaged an index says so. + if err := db.Checkpoint(ctx); err != nil { + t.Fatalf("REPRODUCED: startup checkpoint right after the migration failed: %v", err) + } + + // Phase 3 — the loop production actually ran: fill the added column, fold + // the batch away in one large replace transaction, stop, start, repeat. + batch := spans / 2 + if batch < 1 { + batch = 1 + } + for cycle := 1; cycle <= cycles; cycle++ { + mustExec(t, db.rw, fmt.Sprintf("cycle %d: populate added column", cycle), populateAddedColumn) + + before := countSpans(t, db) + start := time.Now() + if err := db.RollupAndPurge(DefaultRetention); err != nil { + t.Fatalf("cycle %d: roll-up and purge: %v", cycle, err) + } + after := countSpans(t, db) + if after == before { + t.Fatalf("cycle %d: replace transaction deleted nothing (%d spans, all inside the %d-day window?)", + cycle, before, DefaultRetention.RawDays) + } + t.Logf("cycle %d: replace transaction %d spans -> %d in %s", + cycle, before, after, time.Since(start).Round(time.Millisecond)) + + // The stop. The shutdown checkpoint folds the mutated index into the + // main file; this is where the 2026-09-28 damage first became visible. + if err := db.Checkpoint(ctx); err != nil { + t.Fatalf("REPRODUCED: cycle %d: shutdown checkpoint after the replace transaction failed: %v", cycle, err) + } + if err := db.Close(); err != nil { + t.Fatalf("cycle %d: close: %v", cycle, err) + } + db = nil + requireNoWAL(t, path, fmt.Sprintf("after the cycle %d shutdown checkpoint", cycle)) + + // The start. On 1.1.3 the equivalent open aborted inside + // duckdb_open_ext; a Go error here is the same defect reported sanely. + db, err = Open(path) + if err != nil { + t.Fatalf("REPRODUCED: cycle %d: reopen after the replace transaction failed: %v", cycle, err) + } + if err := db.Checkpoint(ctx); err != nil { + t.Fatalf("REPRODUCED: cycle %d: startup checkpoint after reopen failed: %v", cycle, err) + } + assertIndexesIntact(t, db, after) + + // Fresh ingest after the restart, so the next cycle mutates an index + // that has already been folded and read back once. + if cycle < cycles { + lo := cycle * spans + mustExec(t, db.rw, fmt.Sprintf("cycle %d: ingest %d more spans", cycle, batch), + seedSpans, now, reproSpanDays, lo, lo+batch) + } + } + + t.Logf("did not reproduce on %s after %d cycles: every open succeeded, both checkpoints folded "+ + "on every cycle, all four indexes survived, and indexed probes agreed with unindexed ones", + engine, cycles) +} + +// assertIndexesIntact is the verdict after a restart: the four indexes exist, +// no row was lost, and the index on the added column still answers correctly. +func assertIndexesIntact(t *testing.T, db *DB, wantSpans int) { + t.Helper() + want := []string{"idx_spans_name", "idx_spans_session_id", "idx_spans_start_time", "idx_spans_user_id"} + got := indexNames(t, db.ReadOnly()) + if fmt.Sprint(got) != fmt.Sprint(want) { + t.Fatalf("after reopen: indexes %v, want %v", got, want) + } + if n := countSpans(t, db); n != wantSpans { + t.Fatalf("after reopen: %d spans, want %d", n, wantSpans) + } + assertIndexedProbesAgree(t, db) +} + +func countSpans(t *testing.T, db *DB) int { + t.Helper() + var n int + if err := db.ReadOnly().QueryRow("SELECT count(*) FROM spans").Scan(&n); err != nil { + t.Fatalf("count spans: %v", err) + } + return n +} + +// assertIndexedProbesAgree compares an equality on the added column — which the +// ART index on it can serve — against the same predicate wrapped in an +// expression, which it cannot. A damaged or mis-addressed index shows up as a +// disagreement; a wrong count here is this defect's silent form, and neither a +// checkpoint nor an open reports it. +func assertIndexedProbesAgree(t *testing.T, db *DB) { + t.Helper() + ro := db.ReadOnly() + for u := 0; u < reproUsers; u++ { + want := fmt.Sprintf("user-%d", u) + var indexed, scanned int + if err := ro.QueryRow("SELECT count(*) FROM spans WHERE user_id = ?", want).Scan(&indexed); err != nil { + t.Fatalf("indexed probe %s: %v", want, err) + } + if err := ro.QueryRow("SELECT count(*) FROM spans WHERE user_id || '' = ?", want).Scan(&scanned); err != nil { + t.Fatalf("unindexed probe %s: %v", want, err) + } + if indexed != scanned { + t.Fatalf("REPRODUCED: user_id = %q matched %d rows through the index, %d by scan", want, indexed, scanned) + } + if indexed == 0 { + t.Fatalf("probe %s matched nothing by either path; the fixture built no entries for it", want) + } + } +} + +func requireNoWAL(t *testing.T, path, when string) { + t.Helper() + info, err := os.Stat(path + ".wal") + if err == nil { + t.Fatalf("%s: %s is still present (%d bytes); the checkpoint did not fold", when, path+".wal", info.Size()) + } + if !os.IsNotExist(err) { + t.Fatalf("stat wal %s: %v", when, err) + } +}