Skip to content

images: mirror digest-pinned indexes byte for byte - #391

Open
mpuig wants to merge 2 commits into
kernel:mainfrom
mpuig:fix/mirror-index-digest
Open

images: mirror digest-pinned indexes byte for byte#391
mpuig wants to merge 2 commits into
kernel:mainfrom
mpuig:fix/mirror-index-digest

Conversation

@mpuig

@mpuig mpuig commented Aug 11, 2026

Copy link
Copy Markdown

Problem

A Dockerfile that pins its base image by digest almost always pins the multi-arch index digest — it's the digest docker pull and Docker Hub print. MirrorBaseImage resolves that reference to the platform image and pushes it under the index digest, which the local registry correctly refuses:

push to local registry: PUT …/v2/library/python/manifests/sha256:9b4929a7…:
unexpected status code 400 Bad Request: digest mismatch:
expected sha256:9b4929a7…, got sha256:1e58d36e…

Because the mirror failure is only a WARN, the build proceeds and fails later with an unrelated error (create builder instance: image is required on v0.3.0), so the root cause is invisible in the API response. Net effect: every digest-pinned FROM is unbuildable — and supply-chain-pinned Dockerfiles are exactly the ones that hit it.

Reproduced on v0.3.0 on a GitHub-hosted ubuntu runner with FROM python:3.13-alpine@sha256:9b4929a7…; journal evidence above.

Fix

An index named by digest is mirrored as the index itself (remote.WriteIndex — children first, then the index manifest), so the destination stores content whose digest is exactly the one the caller pinned.

Unchanged behavior:

  • tag references still mirror only the platform-resolved image (the storage-saving path);
  • a digest reference naming a plain (single-platform) manifest already round-trips and is untouched.

The push logic moves into pushMirrored so the property is testable against in-memory registries (pkg/registry, as in imagepush/manager_test.go): the new tests pin a random two-manifest index by digest, mirror it, and prove the pinned digest resolves at the destination — which is the assertion the old code could not pass.

Testing

  • go test ./lib/images/ green (new + existing tests)
  • go build ./... green

Found while building barista on hypeman; happy to adjust if you'd rather handle index-pinned refs differently (e.g. refusing them loudly instead of mirroring all platforms).


Note

Medium Risk
Changes core image mirroring push semantics for digest-pinned refs (more data copied for indexes) but scope is limited to lib/images with targeted tests; builds depending on mirroring are fixed rather than broken.

Overview
Fixes digest-pinned FROM lines that reference a multi-arch index digest (the usual shape from docker pull). Mirroring no longer resolves to a single platform image and pushes it under the pinned index digest, which registries reject with digest mismatch.

MirrorBaseImage now fetches via remote.Get and delegates to pushMirrored: digest references that name an index are copied with remote.WriteIndex so the local registry serves content at exactly the pinned digest. Tag references still mirror only the platform-resolved image to save storage.

Adds httptest registry tests for pushMirrored covering index-by-digest round-trip and unchanged tag-based single-image mirroring.

Reviewed by Cursor Bugbot for commit 066b356. Bugbot is set up for automated code reviews on this repo. Configure here.

A Dockerfile that pins its base image by digest almost always pins the
multi-arch *index* digest -- it is the digest `docker pull` and Docker Hub
print. MirrorBaseImage resolved that reference to the platform image and
pushed it under the index digest, which the local registry correctly
refuses:

    push to local registry: PUT .../v2/library/python/manifests/sha256:9b4929a7...:
    unexpected status code 400 Bad Request: digest mismatch:
    expected sha256:9b4929a7..., got sha256:1e58d36e...

The mirror failure is only a WARN, so the build then proceeds and fails
later with an unrelated error -- every digest-pinned FROM is unbuildable,
and supply-chain-pinned Dockerfiles are exactly the ones that hit it.

An index named by digest is now mirrored as the index itself (children
first, then the index manifest), so the destination stores content whose
digest is exactly the one the caller pinned. Tag references keep the
existing platform-image behavior, which is what saves storage; a digest
reference that names a plain manifest is also unchanged.

The push logic moves into pushMirrored so the property is testable against
in-memory registries: the new tests pin a random two-manifest index by
digest, mirror it, and prove the pinned digest resolves at the destination.
mpuig added a commit to mpuig/barista.sh that referenced this pull request Aug 11, 2026
§8 and §9 went upstream as fix PRs (kernel/hypeman#391, #393), §10 and §11
as issues (#394, #395). Each finding now names the workaround it retires
when the upstream fix ships, so the CI hacks have their deletion triggers
on record rather than in memory.
@sjmiller609
sjmiller609 requested a review from chruffins August 11, 2026 21:01

@chruffins chruffins left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for putting up this PR! it's just missing some test coverage but looks good otherwise

Comment thread lib/images/mirror_test.go

// The storage-saving path is unchanged: a tag reference mirrors only the
// (platform-resolved) image, not the whole index.
func TestPushMirroredByTagStillMirrorsImage(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the tag test uses a single-platform image, so it doesn’t check that tagged multi-platform indexes resolve to just the requested platform image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants