remotecache: fix silently dropped cache link - #7048
Open
mispencer wants to merge 1 commit into
Open
Conversation
addItemToStorage (import) and marshalItem (export) both build their output
by recursively walking an item's own dependencies, memoizing each item's
storage entry / record slot so a shared dependency is only resolved once.
Both allocated that memoized entry only *after* the recursive walk
finished, using a sentinel ("" / -1) to mark an item as already in
progress in the meantime.
When an item is reachable through more than one path - which happens in
practice once two unrelated ops coincidentally produce byte-identical
content, since their cache records can end up several levels deep in each
other's dependency chains despite having distinct provenance - a second
path can revisit an item while the first path is still in the middle of
resolving it. Before this fix, that reentrant call found only the
in-progress sentinel, got back nil/-1, and its caller silently skipped
registering the link - with no error, no warning. Which links survived
depended on Go's randomized per-process map iteration order (cc.leaves(),
and the multi-candidate alternatives at a single input slot), so
reconstructing the exact same chain from the exact same bytes could
non-deterministically drop a link on some runs and not others.
Both an item's final id (computed by computeIDs before addItemToStorage
runs) and its record's array index (known as soon as its slot in
state.records is reserved) are available up front, before any recursion.
Allocate and register the entry immediately instead of after, so a
reentrant call gets back the same, real (if not yet fully populated)
entry and can append its link to it successfully - nothing is silently
dropped anymore, regardless of traversal order.
This also makes addItemToStorage's separate `visited` map and the
k.byItem "" in-progress sentinel (and its "invalid loop" error branch)
redundant: k.byItem/k.byID alone now correctly memoize both same-call and
cross-call revisits, so `visited` was removed.
Verified with cache/remotecache/v1/cachestorage_test.go using a real
`--cache-to type=local,mode=max` export captured from a repro build
(testdata/cyclic-merge-chain.json): reparsing it into a fresh
NewCacheKeyStorage dropped the affected link on ~60-80% of iterations
against the pre-fix implementation (closely matching the failure rate
reported in the field) and 0% after this change. Also verified end-to-end
against real buildkitd builds (docker-container driver, both
type=registry and type=local cache backends, fresh builder per
iteration): 0 spurious cache misses across 95+ iterations post-fix versus
a 60-100% failure rate before it, including a wider variant with more
parallel merge points and a byte-for-byte content comparison between a
fresh build and a cache-imported one.
Signed-off-by: Spencer G. Jones <spencer.jones2@tylertech.com>
mispencer
force-pushed
the
fix/remotecache-cycle-link-loss
branch
from
August 18, 2026 17:15
5fe3b63 to
f896197
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
addItemToStorage(import,cache/remotecache/v1/cachestorage.go) andmarshalItem(export,cache/remotecache/v1/utils.go) both build theiroutput by recursively walking an item's own dependencies, memoizing each
item's storage entry / record slot so a shared dependency is only resolved
once. Both allocated that memoized entry only after the recursive walk
finished, using a sentinel (
""/-1) to mark an item as already inprogress in the meantime.
When an item is reachable through more than one path — which happens in
practice whenever two unrelated ops coincidentally produce byte-identical
content, since their cache records can end up several levels deep in each
other's dependency chains despite having distinct provenance — a second
path can revisit an item while the first path is still in the middle of
resolving it. Before this fix, that reentrant call found only the
in-progress sentinel, got back
nil/-1, and its caller silently skippedregistering the link — with no error, no warning. Which links survived
depended on Go's randomized per-process map iteration order (
cc.leaves(),and the multi-candidate alternatives at a single input slot), so
reconstructing the exact same chain from the exact same bytes could
non-deterministically drop a link on some runs and not others.
An item's final id (computed by
computeIDsbeforeaddItemToStorageruns) and a record's array index (known as soon as its slot in
state.recordsis reserved) are both available up front, before anyrecursion. This PR allocates and registers the entry immediately instead
of after, so a reentrant call gets back the same, real (if not yet fully
populated) entry and can append its link to it successfully — nothing is
silently dropped anymore, regardless of traversal order.
This also makes
addItemToStorage's separatevisitedmap and thek.byItem""in-progress sentinel (and its"invalid loop"errorbranch) redundant:
k.byItem/k.byIDalone now correctly memoize bothsame-call and cross-call revisits, so
visitedwas removed.Related issues
I believe this is the same underlying mechanism as #2279 ("Docker
BuildKit caching w/ --cache-from fails (roughly 50% rate)") — that issue's
own investigation (multi-platform builds where two platforms coincidentally
produce a byte-identical blob for one step) traced the corruption to
before the upload/export stage and found a layer graph with links missing
exactly where two branches reconverge on identical content, which matches
what I found here. I haven't reproduced #2279's specific multi-platform
trigger, so I'm not marking this as closing it, but I'd appreciate it if
anyone still hitting that issue could try this branch against their repro.
Likely related, same subsystem, same "identical content confuses cache
chain dedup" family: #1876, #2973, #3009, #3188, #2822, #2996, #2383.
Test plan
cache/remotecache/v1/cachestorage_test.gousing a real--cache-to type=local,mode=maxexport captured from a repro build(
testdata/cyclic-merge-chain.json): reparsing it into a freshNewCacheKeyStoragedropped the affected link on ~60-80% ofiterations against the pre-fix implementation (closely matching the
failure rate reported in the field) and 0% after this change.
buildkitdbuilds(
docker-containerdriver, bothtype=registryandtype=localcache backends, fresh builder per iteration): 0 spurious cache
misses across 95+ iterations post-fix versus a 60-100% failure rate
before it, including a wider variant with more parallel merge points
and a byte-for-byte content comparison between a fresh build and a
cache-imported one.
go build ./...,go vet ./cache/remotecache/v1/...,gofmt -s,and the full existing test suite (
cache/remotecache/v1,solver) all pass.