SEP-1871: Repin both pins to the current builds, and own pmm-server's start period

[yyyyyyyan]

Summary

The side-car now obtains its own Grafana service-account token at container start (landed on main as part of SEP-1871). Three harness changes follow.

Side-car repin. sep-sidecar pinned …:29d29624, a main-line tag predating the mint. Repinned to …:95a70382, the main-line commit that added it — currently both main's tip and the newest published tag. Verified on the published artifact rather than on the commit, in the style the harness README's "Which image to pin" section already prescribes:

Property	Observed
helper present	grafana_service_account.py in the app root
state directory	the side-car's state dir, mode 700, owned 1001:1001
fan-out function	export_grafana_token in settings-env.sh
reads secrets from a directory	4 SECRETS_DIR references in settings-env.sh
carries a HEALTHCHECK	CMD healthcheck.sh, start period 150s

sep-state volume. The side-car persists the minted token mode 0600 under its state directory and re-reads it on the next start, so a restart needs no Grafana admin credential. The volume extends that across a container recreate. Without it every recreate mints a further token on the same service account, and since minted tokens are asked not to expire, the ones earlier containers resolved stay valid in Grafana until an operator deletes them.

pmm-server repin, and owning its start period. PMM_FB_TAG moves from PR-4500-882b6ba to PR-4500-73298a4, in both places the default is spelled. That tag was derived from a tree still containing PMM-15280 and PMM-15331, both closed unmerged on 2026-08-20, and could not bring the pair up at all — pmm-server exited on the PMM_SEP_POSTGRES_PASSWORD that #1383 stopped seeding, then on /srv/sep not being writable, since that build wrote /srv/sep-secrets and did not ship the directory #1382 mounts.

Losing PMM-15331 also lost the widened start period it carried. The image ships 25 s with 3 retries at 4 s, so pmm-server is marked unhealthy around 37 s while a cold start here takes ~72 s to first pass readyz — and because sep-sidecar depends on service_healthy, compose aborted the dependent rather than waiting. This was reproducible on both green feature builds. compose.yaml now sets a 300 s start period of its own, restoring what the build used to provide, and docker compose up -d completes in one invocation again. That start period is the only healthcheck field it sets: Docker merges the healthcheck field by field, so the probe and the remaining timings keep tracking whatever the pinned image ships rather than a copy that goes stale across repins.

sep-sidecar keeps condition: service_healthy. The README's standing advice was to fall back to service_started when repinning to a build without the gate; that trades an aborted bring-up for a side-car that wins the race and exits on a missing SECRET_KEY. Health still means the four secret files exist — the start period is what gives a cold start room to reach that point.

The doc claims the repin invalidates are corrected alongside it: the caveat citing PMM-15331 as the source of the start period, the count of files removed when PMM_ENABLE_SEP is unset, which is four now that the two Grafana token names are gone, and four statements that PMM generates or owns every SEP secret — true while PMM published the Grafana token, and now false by exactly that one credential. Those are scoped to the four files SEP reads from disk, and the token gets its own bullet recording that the side-car owns it while a build publishing the two canonical names still outranks the mint. compose.yaml's own header carried two of them, alongside a Run ./bootstrap.sh first line the degating below it had just retired.

The repin checklist gains the property this branch makes load-bearing. "Which image to pin" listed two things a pin must carry; the sep-state volume adds a third, because an image without the mint passes both existing checks, comes up healthy, and mounts the volume over a directory nothing writes to — with inert Grafana auth as the only symptom, and nothing logged. The old pin this PR replaces was exactly such an image, so the checklist meant to catch it now probes for the mint helper and the state directory. The bring-up section's token paragraph, which credited PMM with minting and publishing the token, now describes the side-car's own mint, the volume that persists it, and the fact that a token arriving through SECRETS_DIR still outranks it.

bootstrap.sh is now genuinely mysql-only. It generates three sep-mysql test-fixture passwords and nothing else — PMM provisions every secret the pair needs — but compose.yaml guarded them with ${VAR:?run ./bootstrap.sh first}, and Compose interpolates every service at parse time regardless of active profiles. So a guard on this profile-gated service made the script a prerequisite of the two-service bring-up too: with the generated dotenv absent, docker compose config exited 1 on both profiles. The three are now passed through unset. This removes a duplicated guard rather than weakening one — mysql-entrypoint.sh defines require_secrets with the same three names and the same message, and calls it unconditionally at top level before any use, so a mysql run without bootstrap.sh still fails loudly, at container start instead of at parse.

Tested

Measured on this branch against PR-4500-73298a4, and separately against PR-4500-7dcd575, with no overrides beyond a host port remap (an unrelated pmm-server holds 8443 on this machine). Torn down with down -v after each run.

• With the generated dotenv absent entirely, docker compose config exits 0 on both profiles, and docker compose up -d reaches sep-sidecar healthy with four secret files, a token minted mode 600 owned 1001:1001, and one sep service account in Grafana.
• docker compose up -d completes in one invocation, ~72 s, both containers healthy. Without the start-period override the same command aborts with dependency failed to start: container … is unhealthy.
• The side-car's secrets directory holds exactly four files — the three database passwords and SECRET_KEY — and no Grafana token, which is the "no token configured anywhere" precondition.
• Grafana shows exactly one service account: id=2 name=sep role=Admin tokens=1.
• The minted token is persisted mode 600 owned 1001:1001 in the sep-state volume.
• All five supervised programs — sep, inventory, tasks, celery-worker, celery-beat — carry both canonical names, each equal to the persisted token, read from each process's own /proc environ.
• Grafana-backed sign-in end to end through PMM's nginx: /graph/login 200 → session exchange → SEP bearer → /sep/api/apps/ 200 with four apps.
• The PMM-facing path round-trips: from inside celery-worker's own environment, PMM.api_key resolves and an authenticated GET /graph/api/org/users returns one org user. That is the call that previously no-opped silently.
• Restarting the side-car reuses the persisted token byte-for-byte; Grafana still reports one account with one token.
• Neither canonical token name nor GF_SECURITY_ADMIN_* appears in docker inspect output.

Left for QA in its own environment:

• Recreate the side-car (docker compose up -d --force-recreate sep-sidecar) and confirm the token survives, which is what the sep-state volume buys over a plain restart.
• Confirm a PMM build that does publish the two Grafana token names into the secrets directory still outranks the mint — the side-car should make no Grafana call and use the published token.
• Bring up with --profile mysql and confirm the sep-mysql build picks up the same PMM_FB_TAG default.

Note for anyone repeating this: a docker exec into the side-car starts from the image's declared environment and never sees PID 1's exports, so probing the token that way reports it unset even when every supervised program has it. That is the design working — inheritance is also why it stays out of docker inspect — but it reads like a failure. Read a supervised child's /proc/<pid>/environ instead.

Known limitations

• Feature-build tags are a moving target: PMM-15216, PMM-15293 and PMM-15294 are still open and their branches get rewritten without notice. If a bring-up starts failing with nothing apparently changed on either side, suspect a silent re-derivation of PMM-15205-sep-fb before looking at SEP.

Checklist

• New/modified functions have type hints and rST docstrings (N/A — compose and docs only)
• New tests added for new features or bug fixes (N/A — the mint's tests live on main)
• Database migrations generated if models changed (N/A for this change)
• User-facing changes documented (README, inline help, UI text)
• Configuration changes documented with examples

[Copilot]

Pull request overview

Updates the PMM feature-build harness to support side-car-managed Grafana tokens while preserving reliable startup ordering.

Changes:

• Repins PMM and SEP container images.
• Persists the side-car’s Grafana token in a named volume.
• Extends PMM startup health timing and updates harness documentation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
sidecar/pmm-fb/README.md	Documents token minting, persistence, and startup behavior.
sidecar/pmm-fb/compose.yaml	Repins images, adds token state storage, and overrides PMM health timing.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.