Skip to content

fix: preserve sub-manifest digests during container cleanup (#254) - #257

Merged
david-waltermire merged 3 commits into
metaschema-framework:developfrom
david-waltermire:fix/issue-254-container-cleanup
Apr 23, 2026
Merged

fix: preserve sub-manifest digests during container cleanup (#254)#257
david-waltermire merged 3 commits into
metaschema-framework:developfrom
david-waltermire:fix/issue-254-container-cleanup

Conversation

@david-waltermire

@david-waltermire david-waltermire commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Closes #254.

Summary

docker/finch pull ghcr.io/metaschema-framework/oscal-cli:latest fails with FATA[0000] failed to copy: ... content at .../manifests/sha256:... not found. Verified against the current registry state: the :latest index (digest sha256:f304f679…) is intact, but all four of its referenced sub-manifests (amd64 image, arm64 image, and the two attestation manifests) return 404.

Root cause

.github/workflows/container-cleanup.yml runs daily with snok/container-retention-policy, filter image-tags: "!latest !v*". That filter keeps images tagged latest or v*, but GHCR's packages API reports each multi-platform push as:

  • one tagged OCI image index (e.g. :latest), and
  • several untagged package versions — the per-platform sub-manifests and the attestation manifests the index references

The action cannot infer which untagged versions are "children" of a protected index — this limitation is called out explicitly in the action's README. The untagged children match the !latest !v* deletion filter, age past the 7-day cut-off, and are pruned, leaving the tagged index dangling.

As a secondary issue, :latest was being set on every push to main (flavor: latest=${{ github.ref == 'refs/heads/main' }}). That means :latest could jump to a fresh image at any time, and the previous :latest's sub-manifests — once the tag moved off them — were immediately in the delete pool.

Fix

Two changes, applied together:

  1. container-cleanup.yml — add a pre-step that enumerates every version currently tagged latest or v* via the GHCR packages API, fetches each index manifest from the OCI registry, extracts every referenced sub-manifest digest, and passes the deduped list through skip-shas. This is the mitigation the action's own docs recommend.

  2. container.yml + build.yml — switch the latest tag from "applied on every main push" to flavor: latest=auto, and add tags: v* to the build workflow's push triggers. latest=auto publishes :latest on tagged-release pushes only, so the tag tracks an immutable release instead of the tip of main. This also aligns :latest with the existing !latest !v* protection — the tag and the version it points to are both retained.

Notes

  • The currently broken :latest in GHCR will not be repaired by this PR; the dangling sub-manifests are already gone. It will be replaced on the next tagged release build, which now publishes :latest automatically.
  • Tested the enumeration pre-step locally: against the current registry it correctly identifies the four sub-manifest digests that :latest index references (the same four currently returning 404). Once a fresh :latest is published, those digests will be live and this skip-list will prevent the cleanup from repeating the break.

Test plan

  • CI builds pass on this PR
  • After merge, trigger Container Image Cleanup with dry-run: true via workflow_dispatch and confirm the log line reports a non-zero digest count and that no protected digests appear in the deletion plan
  • Next tagged release (post-merge) publishes a fresh :latest; pulling ghcr.io/metaschema-framework/oscal-cli:latest succeeds

Summary by CodeRabbit

  • Chores
    • CI now runs for pushes of version-style tags (e.g., v*), so tagged releases trigger builds and tests.
    • Container cleanup preserves images referenced by protected tags by detecting and skipping their underlying manifest digests during pruning.
    • Docker image metadata tagging now uses semver-derived tag patterns and lets the tagging tool automatically decide when to apply the "latest" tag.

@coderabbitai

coderabbitai Bot commented Apr 23, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 33b737b2-9687-44bf-b3af-04cbec93192e

📥 Commits

Reviewing files that changed from the base of the PR and between e792bd6 and 19b4dc7.

📒 Files selected for processing (1)
  • .github/workflows/container.yml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/container.yml

📝 Walkthrough

Walkthrough

Build workflow now runs for pushed tags matching "v*". Container cleanup is parameterized (REGISTRY, OWNER, IMAGE_NAME) and computes protected sub-manifest digests from GHCR (latest, v*) to skip during retention. Container publish metadata now uses latest=auto.

Changes

Cohort / File(s) Summary
Build trigger
\.github/workflows/build.yml
Added push.tags: - 'v*' so the build workflow also triggers on pushed version-like tags.
Container cleanup
\.github/workflows/container-cleanup.yml
Parameterised registry values via env (REGISTRY, OWNER, IMAGE_NAME); added pre-cleanup steps that list GHCR tags (latest, v*), fetch manifests with manifest-list Accept header, extract and deduplicate .manifests[].digest values, and pass them as skip-shas to the retention action.
Container publish tagging
\.github/workflows/container.yml
Updated docker/metadata-action tag generation: removed direct tag-from-ref rule and added semver tag patterns; changed latest handling to latest=auto (auto-assign latest).

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Actions as GitHub Actions
    participant GHCR as GHCR API
    participant Retention as Container Retention Action

    Actions->>GHCR: List tags matching `latest` and `v*`
    GHCR-->>Actions: Return tag list
    Actions->>GHCR: Fetch each tag's manifest (Accept: application/vnd...+json)
    GHCR-->>Actions: Return manifest JSONs (include .manifests[].digest)
    Actions->>Actions: Extract & dedupe digests → emit `skip-shas`
    Actions->>Retention: Invoke retention action with `skip-shas`
    Retention-->>Actions: Perform cleanup, skipping provided sub-manifests
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Poem

🐇
I nibbled YAML under moonlit light,
Gathered shas to keep them safe at night,
Skipped the crumbs that pruning meant to take,
Now images rest without a quake.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix: preserve sub-manifest digests during container cleanup' directly describes the main change—ensuring container sub-manifests are protected during cleanup operations.
Linked Issues check ✅ Passed The code changes directly address issue #254: they preserve sub-manifest digests by enumerating protected tags and passing their digests to skip-shas, preventing 404 errors when pulling containers.
Out of Scope Changes check ✅ Passed All changes are in scope: container-cleanup.yml adds the sub-manifest enumeration logic, container.yml fixes latest tagging behavior, and build.yml adds tag push triggers—all directly addressing the root causes in issue #254.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/container-cleanup.yml (1)

36-62: Well-designed protection logic with good edge-case handling.

The script correctly handles failure scenarios (missing tags, unavailable manifests) using || true and jq null coalescing. The output clearly logs the protection count for observability.

Minor nit: Line 61 has inconsistent quoting for $tags_file (unquoted) compared to other usages (quoted). While mktemp paths don't contain spaces, consistent quoting is good practice.

🔧 Optional: consistent variable quoting
-        echo "Protecting $count sub-manifest digest(s) across $(wc -l < $tags_file | tr -d ' ') tag(s)"
+        echo "Protecting $count sub-manifest digest(s) across $(wc -l < "$tags_file" | tr -d ' ') tag(s)"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/container-cleanup.yml around lines 36 - 62, The unquoted
file redirection using $tags_file in the echo that prints the protection count
is inconsistent; update uses of the tags_file variable in command
substitutions/redirections to be quoted (e.g., change occurrences like $(wc -l <
$tags_file | tr -d ' ') to $(wc -l < "$tags_file" | tr -d ' ')) so "$tags_file"
is quoted everywhere (notably where the "Protecting $count ..." message is
constructed) to follow the quoting used for sort -u "$tags_file" -o
"$tags_file".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In @.github/workflows/container-cleanup.yml:
- Around line 36-62: The unquoted file redirection using $tags_file in the echo
that prints the protection count is inconsistent; update uses of the tags_file
variable in command substitutions/redirections to be quoted (e.g., change
occurrences like $(wc -l < $tags_file | tr -d ' ') to $(wc -l < "$tags_file" |
tr -d ' ')) so "$tags_file" is quoted everywhere (notably where the "Protecting
$count ..." message is constructed) to follow the quoting used for sort -u
"$tags_file" -o "$tags_file".

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d53bc429-7070-4ee1-b4cb-36fe4a909a35

📥 Commits

Reviewing files that changed from the base of the PR and between 4154bc8 and d98d7c3.

📒 Files selected for processing (3)
  • .github/workflows/build.yml
  • .github/workflows/container-cleanup.yml
  • .github/workflows/container.yml

@david-waltermire david-waltermire linked an issue Apr 23, 2026 that may be closed by this pull request
…ma-framework#254)

Pulls of ghcr.io/metaschema-framework/oscal-cli:latest fail with
"manifest ... not found". The daily snok/container-retention-policy run
uses `image-tags: "!latest !v*"`, which keeps the tagged OCI index but
does not recognise the index's untagged per-platform and attestation
sub-manifests as linked. Those sub-manifests age out on the 7-day
cut-off and get deleted, leaving the index pointing at nothing.

Two changes:

* Add a pre-step that lists the currently-tagged `latest` and `v*`
  images, fetches each index manifest from GHCR, and passes every
  referenced sub-manifest digest into `skip-shas` — the workaround
  documented in snok/container-retention-policy's README.
* Move `latest` from "applied on every main push" to `latest=auto` and
  add `tags: v*` to build.yml's push triggers. `latest` now tracks the
  highest semver tag instead of the tip of main, which matches the
  pattern the retention policy protects and keeps it immutable between
  releases.

The currently broken `:latest` in GHCR will be replaced on the next
tagged release build.

Closes metaschema-framework#254
@david-waltermire
david-waltermire force-pushed the fix/issue-254-container-cleanup branch from db7dc5c to e792bd6 Compare April 23, 2026 06:53
Replace type=ref,event=tag (which kept the v-prefix literally) with
type=semver patterns. Pushing v3.2.0 now yields container tags 3.2.0,
3.2, and 3, alongside the existing latest (from latest=auto) and the
commit-SHA tag. This matches the conventional 'majors/minors/patch'
tag set consumers expect.
@david-waltermire
david-waltermire merged commit fb13883 into metaschema-framework:develop Apr 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Container manifest issue?

1 participant