Skip to content

Memoize include and extends.file loads within a single Load call - #904

Merged
ndeloof merged 1 commit into
compose-spec:mainfrom
glours:perf/memoize-include-extends
Jul 30, 2026
Merged

Memoize include and extends.file loads within a single Load call#904
ndeloof merged 1 commit into
compose-spec:mainfrom
glours:perf/memoize-include-extends

Conversation

@glours

@glours glours commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Problem

ApplyInclude re-parses and recursively re-expands an included file once per
include path that reaches it. When the same file is reachable through more
than one path (a "diamond" include graph), this is exponential: a depth-24
doubling graph loads the leaf 2²⁴ ≈ 16.7M times. Similarly, an extends.file
base is fully re-loaded (read, parse, interpolate, path-resolve) once per
extending service: 50 services extending common.yaml parse it 50 times.

This is the same problem as #886, which showed a real-world ~80-service
monorepo federation taking ~55s in docker compose config.

Why not #886's approach

#886 memoizes includes while preserving the Listener contract "one event per
occurrence" by recording and replaying listener events on cache hits. As
noted in review, Docker Compose always registers a listener, so the replay
path is the production path — and event replay is itself exponential:
3·(2^depth−1) events on a doubling graph, measured at ~1.8GB peak heap at
depth 20 on that branch. The memoization would never be effective in practice.

Fix

  1. Listener contract change: include/extends events are only emitted
    for declarations in the config files passed to the loader, not for
    declarations inside included or extended files. This relaxation is backed
    by a product decision: for usage metrics, only the include/extends
    usage declared in the user's own config files needs to be accounted for —
    there is no requirement to track occurrences discovered recursively while
    expanding them. The only known consumer of these events is Docker
    Compose's telemetry counters, whose current per-occurrence counts are
    already inflated non-deterministically by diamond re-expansion. Resource
    metrics (services, networks, …) are computed from the final merged project
    and are unaffected.
  2. Include memoization, per Load call, carried by the context. The key
    covers every input that determines the model: resolved paths, working
    directory, project directory and effective environment (length-prefixed
    SHA-256). A pristine deep copy is stored; every consumer gets its own copy.
  3. extends.file memoization, scoped per loadYamlModel call (fixed
    interpolation/environment within that scope), keyed by resolved path +
    working directory. Services are deep-copied per consumer because resolving
    an extends chain writes merged services back into the base map.

Cycle detection is unaffected: both checks run before any cache lookup, and
only successfully-expanded (cycle-free) subtrees are ever cached.

Benchmarks

CPU:

scenario before after
diamond include, depth 10 235.5 ms 3.8 ms
diamond include, depth 24 (+ listener) timeout (2²⁴ loads) 0.03 s
40 services × extends.file same base 9.2 ms 5.5 ms

Memory (total allocated during load):

scenario before after
diamond include, depth 10 161.8 MB 8.6 MB
worst case: 60 distinct includes × 8 services, zero cache hits 174.5 MB 174.9 MB (+0.2%)

Both caches live only as context values on the load call stack: they are
released as soon as Load returns (post-GC retained heap measured identical
to main in all scenarios). Worst-case retention during a load is linear in
unique included content, bounded by the size of the final model — never
per-occurrence.

Tests

  • TestIncludeDiamondDedup: depth-24 diamond with a registered listener,
    guarded by a 20s timer — fails fast if memoization regresses.
  • TestIncludeListenerTopLevelOnly, updated extends_test.go expectations:
    pin the new listener contract.
  • TestIncludeSameFileDistinctEnv: same file included twice with different
    env_file must not share a cache entry.
  • TestExtendsFileNameCollision, TestExtendsCacheServesIsolatedCopies:
    guard cache-entry isolation against the extends write-back.
  • BenchmarkIncludeDiamond, BenchmarkExtendsFile.

Closes #886 — credit to @bonitao for the diagnosis and the diamond
regression-test approach, both reused here.

Cache include models per Load and extends.file bases per loadYamlModel,
so a file reachable through several include paths or extended by several
services is only parsed and expanded once. Listener events are now only
emitted for include/extends declared in the config files passed to the
loader.

Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
@glours
glours requested a review from ndeloof as a code owner July 29, 2026 16:41
@ndeloof
ndeloof merged commit d70c053 into compose-spec:main Jul 30, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants