Memoize include and extends.file loads within a single Load call - #904
Merged
Merged
Conversation
Cache include models per Load and extends.file bases per loadYamlModel, so a file reachable through several include paths or extended by several services is only parsed and expanded once. Listener events are now only emitted for include/extends declared in the config files passed to the loader. Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
ndeloof
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ApplyIncludere-parses and recursively re-expands an included file once perinclude path that reaches it. When the same file is reachable through more
than one path (a "diamond" include graph), this is exponential: a depth-24
doubling graph loads the leaf 2²⁴ ≈ 16.7M times. Similarly, an
extends.filebase is fully re-loaded (read, parse, interpolate, path-resolve) once per
extending service: 50 services extending
common.yamlparse it 50 times.This is the same problem as #886, which showed a real-world ~80-service
monorepo federation taking ~55s in
docker compose config.Why not #886's approach
#886 memoizes includes while preserving the Listener contract "one event per
occurrence" by recording and replaying listener events on cache hits. As
noted in review, Docker Compose always registers a listener, so the replay
path is the production path — and event replay is itself exponential:
3·(2^depth−1)events on a doubling graph, measured at ~1.8GB peak heap atdepth 20 on that branch. The memoization would never be effective in practice.
Fix
include/extendsevents are only emittedfor declarations in the config files passed to the loader, not for
declarations inside included or extended files. This relaxation is backed
by a product decision: for usage metrics, only the
include/extendsusage declared in the user's own config files needs to be accounted for —
there is no requirement to track occurrences discovered recursively while
expanding them. The only known consumer of these events is Docker
Compose's telemetry counters, whose current per-occurrence counts are
already inflated non-deterministically by diamond re-expansion. Resource
metrics (services, networks, …) are computed from the final merged project
and are unaffected.
Loadcall, carried by the context. The keycovers every input that determines the model: resolved paths, working
directory, project directory and effective environment (length-prefixed
SHA-256). A pristine deep copy is stored; every consumer gets its own copy.
extends.filememoization, scoped perloadYamlModelcall (fixedinterpolation/environment within that scope), keyed by resolved path +
working directory. Services are deep-copied per consumer because resolving
an extends chain writes merged services back into the base map.
Cycle detection is unaffected: both checks run before any cache lookup, and
only successfully-expanded (cycle-free) subtrees are ever cached.
Benchmarks
CPU:
extends.filesame baseMemory (total allocated during load):
Both caches live only as context values on the load call stack: they are
released as soon as
Loadreturns (post-GC retained heap measured identicalto main in all scenarios). Worst-case retention during a load is linear in
unique included content, bounded by the size of the final model — never
per-occurrence.
Tests
TestIncludeDiamondDedup: depth-24 diamond with a registered listener,guarded by a 20s timer — fails fast if memoization regresses.
TestIncludeListenerTopLevelOnly, updatedextends_test.goexpectations:pin the new listener contract.
TestIncludeSameFileDistinctEnv: same file included twice with differentenv_filemust not share a cache entry.TestExtendsFileNameCollision,TestExtendsCacheServesIsolatedCopies:guard cache-entry isolation against the extends write-back.
BenchmarkIncludeDiamond,BenchmarkExtendsFile.Closes #886 — credit to @bonitao for the diagnosis and the diamond
regression-test approach, both reused here.