Skip to content

fix(store): being served old docker images and not detecting a new one#409

Open
OisinKyne wants to merge 3 commits intomainfrom
oisin/image-determinism
Open

fix(store): being served old docker images and not detecting a new one#409
OisinKyne wants to merge 3 commits intomainfrom
oisin/image-determinism

Conversation

@OisinKyne
Copy link
Copy Markdown
Contributor

@OisinKyne OisinKyne commented May 3, 2026

Summary

You weren't overthinking — your diagnosis was exactly right.
With :latest tags, the embedded K8s manifest is byte-identical
across binary versions, so kubectl apply reports "unchanged"
and the cached pod keeps serving the old image forever. New
users updating their binary saw stale pods.

Changes (188 lines added, 3 changed)

internal/images/images.go (new) — Resolve(repo) policy helper:

  • OBOL_DEVELOPMENT=true → repo:latest (matches
    buildAndImportLocalImages flow)
  • GitCommit empty/unknown/dev → repo:latest (binaries built
    without ldflags)
  • GitDirty=true → repo:latest (no published image for a dirty
    commit)
  • Otherwise → repo: (matches CI's
    type=sha,format=short tag)

internal/images/images_test.go (new) — 6 tests covering all
branches plus case-insensitive OBOL_DEVELOPMENT.

cmd/obol/sell.go — demo-server image now goes through
images.Resolve.

internal/tunnel/tunnel.go — storefront image now goes through
images.Resolve.

.github/workflows/docker-publish-x402.yml — added
type=sha,format=short,prefix= so the binary's short-SHA
GitCommit resolves. Long SHA stays for Trivy compat.

.github/workflows/docker-publish-storefront.yml — added both
short AND long SHA tags. (Storefront workflow had the same
MANIFEST_UNKNOWN Trivy bug x402 had — fixed in passing.)

How this fixes the user's bug

After this:

  1. CI publishes obol-stack-public-storefront:f2c0508 (short
    SHA) on every commit to main
  2. The release binary is built with version.GitCommit=f2c0508
    injected via ldflags
  3. When the binary's CreateStorefront runs, it produces a
    manifest with image: ...storefront:f2c0508
  4. On the next binary upgrade (different commit → different
    SHA), the manifest changes → kubectl detects diff → rolls the
    pod → fresh image pulled

Dev mode (OBOL_DEVELOPMENT=true) still uses :latest and the
local-build/k3d-import path — unchanged behavior.

Known follow-ups (deferred, called out for transparency)

  1. Embedded YAML templates still use :latest for x402-buyer,
    x402-verifier, serviceoffer-controller. Same problem, but these
    go through helmfile not Go — fixing requires
    --state-values-set imageTag=... plumbing in stack.go + template
    gotmpl interpolation. The TODOs in x402.yaml/llm.yaml already
    flag this.
  2. CI race on tagged releases: release.yml and the image
    workflows trigger on the same tags: v* push and run in
    parallel. If image jobs are slower, a freshly-installed binary
    briefly references an image that hasn't been pushed →
    ImagePullBackOff. Fix is needs: ordering in release.yml. Likely
    worth it before any wider rollout.

Want me to tackle either follow-up next, or push these and
validate against the user who saw the bug?

Base automatically changed from oisin/extra-hermes-behaviour to main May 3, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant