From d26a2f049800444ed4a4d4d1e2eb2489bd4efa95 Mon Sep 17 00:00:00 2001 From: Wayland Date: Tue, 13 Oct 2026 16:15:03 +0200 Subject: [PATCH 1/2] build(compose): tag the local build cotel-local:latest instead of the published name `image:` next to `build:` is the name compose tags the local build with, so reusing `ghcr.io/flopsstuff/cotel:latest` there meant the published name was being applied to an artifact that never came from the registry. The deploy is `docker compose build --pull` then `up -d` - and `--pull` refreshes the Dockerfile's base images, not the app image - so production's container reported `Image=ghcr.io/flopsstuff/cotel:latest` for a binary built on its own host: built-here and pulled-from-registry were the same string, the published artifact was never consumed, and a stray `docker pull` of that name would have replaced the running service's binary at its next restart. Production still builds its own image; only the name it reuses changes. Making production consume the published image would break merge-equals-deploy, since `release.yml` moves `:latest` on explicit dispatch only, and is a separate decision. The operations pages now name both images where the distinction matters: the recovery procedure's version-matching step asks `cotel-local:latest` on a host that builds its own image, and both pages record that `ghcr.io/flopsstuff/cotel:latest` is a multi-arch index whose `linux/arm64` entry runs natively, which also retires a stale "the published image is amd64 only" claim next to the snapshot-mirror restore. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 3 +++ README.md | 9 ++++++++ docker-compose.yml | 8 ++++++- docs/operations/duckdb-recovery.md | 11 +++++++++- docs/operations/duckdb-snapshots.md | 34 +++++++++++++++++++---------- 5 files changed, 51 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e257860..3358024 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed +- `docker compose build` tags its result `cotel-local:latest` instead of `ghcr.io/flopsstuff/cotel:latest`. Compose treats `image:` next to `build:` as the name to tag the local build with, so the published name was being reused for an artifact that never came from the registry: on the production host — where the deploy is `docker compose build --pull` followed by `up -d`, and `--pull` refreshes the Dockerfile's base images rather than the app image — `Image=ghcr.io/flopsstuff/cotel:latest` named a locally built binary, and built-here and pulled-from-registry were the same string. Three things follow from splitting them. You can read which artifact a host is running off the name. The thing the release pipeline publishes, and which `release.yml` moves only on an explicit dispatch, is no longer shadowed by every merge to `main`. And `docker pull ghcr.io/flopsstuff/cotel:latest` on the production host stops being able to swap the running service's binary at its next restart for content no deploy step validated — it now lands on a name nothing there runs. Production still *builds* its image, deliberately: a pull-only production would mean a merge to `main` no longer changes production, which is a separate decision from this cleanup and would need its own ADR. The operations pages name both images where the distinction matters — the recovery procedure's "match the CLI to the version linked into cotel" step asks `cotel-local:latest` on a host that builds its own image — and they also record that `ghcr.io/flopsstuff/cotel:latest` is now a multi-arch index whose `linux/arm64` entry runs natively on a Pi or Apple Silicon, so the documented `docker run --entrypoint /usr/local/bin/cotel … --db-query` and `--db-import` commands are honest on arm64 without a local build ([docs/operations/duckdb-recovery.md](docs/operations/duckdb-recovery.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md)) + ## [0.4.0] - 2026-10-13 ### Added diff --git a/README.md b/README.md index 211c142..9b1ddb2 100644 --- a/README.md +++ b/README.md @@ -515,6 +515,15 @@ docker compose up CGO_ENABLED=1 go test ./... ``` +`docker compose build` tags its result **`cotel-local:latest`**, which is not any +`ghcr.io/flopsstuff/cotel:*` name on purpose: compose builds this image rather +than pulling it (that is what the deploy does on the production host too), and +if the local build carried the published name you could no longer tell which of +the two artifacts a host was running — nor would `docker pull` of the published +name be safe, since it would quietly replace the running service's binary at the +next restart. So on any host, `cotel-local:latest` is what was built there and +`ghcr.io/flopsstuff/cotel:latest` is what the registry published. + The runtime image is ≈ 294 MB. Most of that is the statically linked DuckDB engine (it was 213 MB on the DuckDB 1.1.3 driver, which loaded the ICU extension from the network at startup instead of bundling it). diff --git a/docker-compose.yml b/docker-compose.yml index 14257b9..70f53ff 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,7 +1,13 @@ services: cotel: build: . - image: ghcr.io/flopsstuff/cotel:latest + # Deliberately NOT ghcr.io/flopsstuff/cotel:latest. With `build:` present, + # `image:` is the name compose tags the local build with, so reusing the + # published name would leave built-here and pulled-from-registry + # indistinguishable on the host, and a stray `docker pull` of that name + # would then replace this service's binary at the next restart with content + # no deploy step validated. + image: cotel-local:latest ports: - "4318:4318" # OTLP/HTTP ingest - "8080:8080" # Dashboard diff --git a/docs/operations/duckdb-recovery.md b/docs/operations/duckdb-recovery.md index 8485fb5..ca7ee14 100644 --- a/docs/operations/duckdb-recovery.md +++ b/docs/operations/duckdb-recovery.md @@ -83,6 +83,15 @@ docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-probe:/data \ `--entrypoint` is not optional here: the image's entrypoint script starts the server and passes extra arguments through, so without it the flag is swallowed and you get a *running cotel* writing to the volume instead of a one-shot query. +**Which image to ask.** `ghcr.io/flopsstuff/cotel:latest` is the registry's published release, and since the release pipeline publishes a multi-arch index (`linux/arm64` + `linux/amd64`, [ADR-0003](../decisions/0003-release-policy)) the command above runs as-is on an arm64 host — a Pi or Apple Silicon — with nothing but a `docker pull`. That is the right image when you want *the released* engine version. It is **not** necessarily the one that wrote the damaged file: the production deploy builds on its host and tags the result `cotel-local:latest`, and `:latest` in the registry moves only when someone cuts a release. On a host that builds its own image, ask `cotel-local:latest` instead, and treat a disagreement between the two answers as the thing to resolve before you touch the file: + +```bash +docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-probe:/data \ + cotel-local:latest --db-query "SELECT version()" +``` + +The two names are deliberately distinct so that this question has an answer at all; they used to be the same string, which made "which binary is running here" unanswerable on the production host. + `--db-query` opens the database **read-only**, so it is safe to run against the probe copy. If the file is too damaged to open even read-only, read the pin out of the source tree instead: ```bash @@ -150,7 +159,7 @@ SELECT max(start_time), max(ingested_at) FROM spans; -- newest data sti SELECT index_name FROM duckdb_indexes() ORDER BY index_name; -- all four back ``` -Then exit the CLI and confirm the file opens with the cotel binary — the round trip is the real test, because a version-mismatched CLI write would show up here and nowhere earlier: +Then exit the CLI and confirm the file opens with the cotel binary — the round trip is the real test, because a version-mismatched CLI write would show up here and nowhere earlier. Ask the binary that will actually serve the file, which on a host that builds its own image means `cotel-local:latest` (see step 4): ```bash docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-probe:/data \ diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 1edafd6..0661932 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -150,13 +150,20 @@ needs nothing but the image already on the host: no DuckDB CLI, no version matching by hand (which is the step of the recovery procedure most able to destroy a file). -Published releases push a multi-arch image (`linux/amd64` and `linux/arm64`) — -see [ADR-0003](../decisions/0003-release-policy). On an arm64 host (Pi, Apple -Silicon) `docker pull ghcr.io/flopsstuff/cotel:latest` is enough; you do not -need `docker compose build` just to get a binary that can import. If pull still +`ghcr.io/flopsstuff/cotel:latest` is the registry's published release and a real +multi-arch index (`linux/amd64` + `linux/arm64`) — see +[ADR-0003](../decisions/0003-release-policy) — so it is safe to `docker run` on +an arm64 host (Pi, Apple Silicon) straight from a `docker pull`; you do not need +`docker compose build` just to get a binary that can import. If pull still reports `no matching manifest for linux/arm64`, you are on a release older than -that change — build once with `docker compose build` and proceed with the local -tag. +that change — build once with `docker compose build`, which tags its result +`cotel-local:latest`, and use that name in the commands below. + +Those two names are never the same artifact: `cotel-local:latest` is what this +host built, `ghcr.io/flopsstuff/cotel:latest` is what the registry published, +and the production deploy runs the former. For a restore either will do — the +import only needs a cotel binary for this platform — but when it matters which +one you asked, the name now tells you. Two rules the flag enforces rather than documents: the target database file must be **empty or absent** (the snapshot's `schema.sql` issues plain `CREATE TABLE`, @@ -337,12 +344,15 @@ docker run --rm --entrypoint /usr/local/bin/cotel \ absolute `/snapshots//...` paths, which is also why each mirrored directory keeps the source's name exactly. -**That command does not run on the Pi.** `--db-import` needs a cotel binary for -the host's platform and the published image is amd64 only, so the proof -available on the Pi is byte-identity - `sha256` per file and the manifest, both -checked on every pull - rather than a restore. Restore from the mirror wherever -an amd64 cotel binary exists, by copying the directory there; restorability of -the artifact itself is established at the source, with the drill in +**That command now runs on the Pi too.** `--db-import` needs a cotel binary for +the host's platform, and the published image stopped being amd64-only: `docker +pull ghcr.io/flopsstuff/cotel:latest` on the Pi resolves the `linux/arm64` entry +of the index and the binary inside it executes natively. What has not changed is +the proof the mirror itself offers on every tick - byte-identity, `sha256` per +file plus the manifest - which is what the status line reports; an actual +restore from the mirror on the Pi is now possible but is not something the pull +does for you. Restorability of the artifact itself is still established at the +source, with the drill in [ADR-0023](../decisions/0023-production-database-snapshots). ## What snapshots do not cover From 5422991aa3e9179ed40a34b7eb66697c51b30c69 Mon Sep 17 00:00:00 2001 From: Daedalus Date: Tue, 13 Oct 2026 16:26:48 +0200 Subject: [PATCH 2/2] docs(operations): ask the serving binary by name in both verify steps Two verification steps say to use the binary that will serve the file and then name the published image in the command. On a host that builds its own image that is now a different artifact from the one that will serve it, so the step asks a different question than the one it claims to answer - the recovery page's round trip said `cotel-local:latest` in prose and `ghcr.io/...` in the command block directly beneath it. Both now name `cotel-local:latest` in the command and say which name to substitute on a host that pulls instead of builds. Co-Authored-By: Daedalus Co-Authored-By: Claude Opus 5 --- docs/operations/duckdb-recovery.md | 4 ++-- docs/operations/duckdb-snapshots.md | 13 +++++++++---- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/operations/duckdb-recovery.md b/docs/operations/duckdb-recovery.md index ca7ee14..15feb66 100644 --- a/docs/operations/duckdb-recovery.md +++ b/docs/operations/duckdb-recovery.md @@ -159,11 +159,11 @@ SELECT max(start_time), max(ingested_at) FROM spans; -- newest data sti SELECT index_name FROM duckdb_indexes() ORDER BY index_name; -- all four back ``` -Then exit the CLI and confirm the file opens with the cotel binary — the round trip is the real test, because a version-mismatched CLI write would show up here and nowhere earlier. Ask the binary that will actually serve the file, which on a host that builds its own image means `cotel-local:latest` (see step 4): +Then exit the CLI and confirm the file opens with the cotel binary — the round trip is the real test, because a version-mismatched CLI write would show up here and nowhere earlier. Ask the binary that will actually serve the file: on a host that builds its own image, which production does, that is `cotel-local:latest` (see step 4); on a host that pulls, substitute `ghcr.io/flopsstuff/cotel:latest`. ```bash docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-probe:/data \ - ghcr.io/flopsstuff/cotel:latest --db-query "SELECT count(*) FROM spans" + cotel-local:latest --db-query "SELECT count(*) FROM spans" ``` No `cotel.duckdb.wal` should remain in the probe volume after the CLI exits cleanly. diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 0661932..40346bc 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -206,15 +206,20 @@ make a new one rather than trying to clean it up: docker volume rm cotel-data-restore-probe && docker volume create cotel-data-restore-probe ``` -**Step 3 - verify with the same binary that will serve it.** +**Step 3 - verify with the same binary that will serve it.** Which name that is +depends on the host: `cotel-local:latest` where the image is built locally, as +production does, and `ghcr.io/flopsstuff/cotel:latest` where it is pulled. The +distinction matters at this step and not at step 2 - what this step catches is a +file the *serving* binary cannot open, so asking any other binary answers a +different question. ```bash docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-restore-probe:/data \ - ghcr.io/flopsstuff/cotel:latest --db-query "SELECT count(*) FROM spans" + cotel-local:latest --db-query "SELECT count(*) FROM spans" docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-restore-probe:/data \ - ghcr.io/flopsstuff/cotel:latest --db-query "SELECT max(start_time) FROM spans" + cotel-local:latest --db-query "SELECT max(start_time) FROM spans" docker run --rm --entrypoint /usr/local/bin/cotel -v cotel-data-restore-probe:/data \ - ghcr.io/flopsstuff/cotel:latest --db-query "SELECT count(*) FROM duckdb_indexes()" + cotel-local:latest --db-query "SELECT count(*) FROM duckdb_indexes()" ``` The secondary indexes are **rebuilt from the data** by the import rather than