diff --git a/plans/tecnix-target-eval-caching/explainer.md b/plans/tecnix-target-eval-caching/explainer.md index 1f37036f4..25218af3a 100644 --- a/plans/tecnix-target-eval-caching/explainer.md +++ b/plans/tecnix-target-eval-caching/explainer.md @@ -202,6 +202,24 @@ This is the precise sense in which the system works across commits. After a comm The same property makes the system fail-safe. A row is never believed on the strength of its key; it is believed only when proven. Format changes, corruption, and storage defects therefore all degrade into cache misses, never into wrong answers. In practice, the difference between a cold run and a warm run is the difference between roughly a minute of full tracked evaluation and well under a second — several orders of magnitude, with the gap consisting precisely of "all of evaluation" versus "fingerprint checks and output construction." +### 4.2 Synthetic keys: parts of the manifest + +Not every closure key is a file. A target that looks up one zone in `.meta/manifest.json` depends on that zone's entry, not on the whole file, and recording the file would invalidate every such target whenever any zone is added or renumbered. The manifest builtins therefore record *synthetic* keys, each naming one observation of the manifest. This is the complete list: + +| key | recorded by | fingerprint changes exactly when | +|---|---|---| +| `.meta/manifest.json#` | `builtins.tectonixManifestEntry zonePath` | the JSON value of that zone's entry (any field, not only `id`) changes, appears, or disappears | +| `.meta/manifest.json#keys` | `builtins.tectonixManifestKeys` | the set of zone paths (manifest keys) changes | +| `.meta/manifest.json#id/` | `builtins.tectonixManifestIdToPath zoneId` | a different zone path, or none, is the first in sorted order whose entry has the id `zoneId` | + +The builtins read the manifest through the same accessor as every other tracked read (§7), so a value and the fingerprint of its key always describe the same manifest. The grammar is exact: + +- A key is synthetic if and only if it begins with `.meta/manifest.json#`. Every other key is a repo-relative path, even one that contains `#`; splitting arbitrary keys at `#` is wrong. +- The fragment after that prefix is `keys`, or `id/` followed by a zone id (everything after `id/`, verbatim), or a zone path, which begins with `//`. `tectonixManifestEntry` given an argument that does not begin with `//` records the whole file, `.meta/manifest.json`, so the three forms never overlap. +- A synthetic key depends on the file `.meta/manifest.json`. A consumer of closure keys (for example, one mapping changed files to affected targets) that cannot interpret a fragment must treat any change to that file as affecting the key; one that can compares the named observation of the manifest before and after the change. + +Reading the whole manifest (the `unsafeTectonixInternalManifest*` builtins, or `builtins.readFile` of the file) still records the plain path `.meta/manifest.json`. + --- ## 5. A Worked Example @@ -327,7 +345,7 @@ Clean paths are served from git's object store — no checkout is required, and In a worldtree sandbox (`tectonix-worldtree-socket` set) there is no git repository to read; the clean tree is instead the daemon's immutable FUSE projection of the pinned commit, with repo-relative paths mapped through the committed manifest to per-zone views. The fingerprint vocabulary is unchanged: directories read their exact committed tree oid from the projection's `user.worldtree.tree-oid` xattr, and regular files read the daemon's `user.worldtree.blob-oid` xattr when it is served — one O(1) metadata read each. Symlinks (which cannot carry user xattrs; `getxattr` would follow the link and answer for its target, a different git object) and files under daemons that do not serve blob oids fall back to hashing their bytes as git blobs — a blob oid is a pure function of content — memoized in memory for the accessor's lifetime, which the immutable projection makes sound: each unique file is hashed at most once per evaluation. Because every mechanism emits identical fingerprint strings, closures produced under one backend validate under the other. Two caveats follow from the projection being zone-granular: committed paths outside every visible zone do not exist in this view and observe as `absent`, and zone-ancestor directories are synthesized with a composite `worldtree-union:` fingerprint outside the git vocabulary (their listing genuinely differs from the full git tree, so cross-backend cache misses there are correct, not conservative). -The evaluator's own caches require care, since any of them could silently absorb a read. Each — the file-evaluation cache, the import-resolution cache, the source-to-store copy cache — either maintains a separate tracked-domain instance or replays its provenance on a hit. The general rule: *any cache capable of skipping a physical read must replay the provenance of that read.* +The evaluator's own caches require care, since any of them could silently absorb a read. Each — the file-evaluation cache, the import-resolution cache, the source-to-store copy cache, the `builtins.tecnixMemoize` tables — maintains a separate tracked-domain instance, replays its provenance on a hit, or both. The general rule: *any cache capable of skipping a physical read must replay the provenance of that read.* The failure policy throughout is to fail closed. If `git status` fails, evaluation raises an error rather than assuming a clean tree, because the dirty overlay is load-bearing for closure validity. If a path cannot be fingerprinted, evaluation raises an error rather than emitting a partial closure. If evaluation reaches something the closure format cannot represent, it raises an error rather than under-tracking. Every failure mode resolves to a cache miss or a visible error; none resolves to a plausible wrong answer. diff --git a/src/libexpr-tests/meson.build b/src/libexpr-tests/meson.build index 289660047..b1fae20fd 100644 --- a/src/libexpr-tests/meson.build +++ b/src/libexpr-tests/meson.build @@ -68,6 +68,7 @@ sources = files( 'search-path.cc', 'tecnix-dependency-tracking.cc', 'tecnix-eval-cache-row.cc', + 'tecnix-manifest-fingerprint.cc', 'tecnix-worldtree.cc', 'tectonix.cc', 'trivial.cc', diff --git a/src/libexpr-tests/tecnix-manifest-fingerprint.cc b/src/libexpr-tests/tecnix-manifest-fingerprint.cc new file mode 100644 index 000000000..c7b6d999f --- /dev/null +++ b/src/libexpr-tests/tecnix-manifest-fingerprint.cc @@ -0,0 +1,110 @@ +#include +#include +#include + +#include + +#include "nix/expr/tecnix/eval-cache.hh" +#include "nix/util/memory-source-accessor.hh" + +namespace nix { + +namespace { + +/** A source tree whose only file is `.meta/manifest.json` with the given content. */ +ref treeWithManifest(std::string manifest) +{ + auto accessor = make_ref(); + accessor->addFile(CanonPath(".meta/manifest.json"), std::move(manifest)); + return accessor; +} + +/** Fingerprint one synthetic manifest path, with a fresh cache like a fresh evaluation. */ +std::optional fingerprint(ref accessor, std::string_view syntheticPath) +{ + DependencyFingerprintCache cache; + return dependencyFingerprint(accessor, syntheticPath, cache); +} + +constexpr auto twoZones = R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000002" } })"; +constexpr auto threeZones = + R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000002" }, "//zones/c": { "id": "W-000003" } })"; +constexpr auto bRenumbered = R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000009" } })"; + +} // namespace + +TEST(TecnixManifestFingerprint, EntryFingerprintIgnoresOtherEntries) +{ + auto entryA = ".meta/manifest.json#//zones/a"; + auto before = fingerprint(treeWithManifest(twoZones), entryA); + ASSERT_TRUE(before); + EXPECT_EQ(fingerprint(treeWithManifest(threeZones), entryA), before); + EXPECT_EQ(fingerprint(treeWithManifest(bRenumbered), entryA), before); +} + +TEST(TecnixManifestFingerprint, EntryFingerprintFollowsItsOwnEntry) +{ + auto entryB = ".meta/manifest.json#//zones/b"; + EXPECT_NE(fingerprint(treeWithManifest(twoZones), entryB), fingerprint(treeWithManifest(bRenumbered), entryB)); + EXPECT_NE( + fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#//zones/a"), + fingerprint(treeWithManifest(twoZones), entryB)); +} + +TEST(TecnixManifestFingerprint, AbsentEntryHasItsOwnFingerprint) +{ + auto entryC = ".meta/manifest.json#//zones/c"; + auto absent = fingerprint(treeWithManifest(twoZones), entryC); + ASSERT_TRUE(absent); + EXPECT_NE(fingerprint(treeWithManifest(threeZones), entryC), absent); +} + +TEST(TecnixManifestFingerprint, KeyFingerprintTracksTheKeySetOnly) +{ + auto keys = ".meta/manifest.json#keys"; + auto before = fingerprint(treeWithManifest(twoZones), keys); + ASSERT_TRUE(before); + EXPECT_NE(fingerprint(treeWithManifest(threeZones), keys), before); + // Renumbering a zone changes no key. + EXPECT_EQ(fingerprint(treeWithManifest(bRenumbered), keys), before); +} + +TEST(TecnixManifestFingerprint, KeyFingerprintIgnoresKeyOrder) +{ + auto reordered = R"({ "//zones/b": { "id": "W-000002" }, "//zones/a": { "id": "W-000001" } })"; + EXPECT_EQ( + fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys"), + fingerprint(treeWithManifest(reordered), ".meta/manifest.json#keys")); +} + +TEST(TecnixManifestFingerprint, IdFingerprintTracksWhereTheIdPoints) +{ + auto idOfB = ".meta/manifest.json#id/W-000002"; + auto before = fingerprint(treeWithManifest(twoZones), idOfB); + ASSERT_TRUE(before); + // Adding an unrelated zone leaves the lookup alone; renumbering removes the id. + EXPECT_EQ(fingerprint(treeWithManifest(threeZones), idOfB), before); + EXPECT_NE(fingerprint(treeWithManifest(bRenumbered), idOfB), before); +} + +TEST(TecnixManifestFingerprint, FollowsTheAccessorAcrossCaches) +{ + // The parsed manifest must not outlive the cache it was read for: another + // evaluation on the same thread fingerprints against its own commit. + auto entryB = ".meta/manifest.json#//zones/b"; + auto first = fingerprint(treeWithManifest(twoZones), entryB); + auto second = fingerprint(treeWithManifest(bRenumbered), entryB); + EXPECT_NE(first, second); + EXPECT_NE( + fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys"), + fingerprint(treeWithManifest(threeZones), ".meta/manifest.json#keys")); +} + +TEST(TecnixManifestFingerprint, NoManifestMeansNoFingerprint) +{ + EXPECT_EQ(fingerprint(make_ref(), ".meta/manifest.json#keys"), std::nullopt); + // ...and that must not leak into the next tree either. + EXPECT_TRUE(fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys")); +} + +} // namespace nix diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 49ce1ddf5..3fe5b530f 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -249,8 +249,6 @@ static Symbol getName(const AttrName & name, EvalState & state, Env & env) } } -static constexpr size_t BASE_ENV_SIZE = 140; - EvalMemory::EvalMemory() { assertGCInitialized(); @@ -338,10 +336,10 @@ EvalState::EvalState( , regexCache(makeRegexCache()) , tecnixData(std::make_unique()) #if NIX_USE_BOEHMGC - , baseEnvP(std::allocate_shared(traceable_allocator(), &mem.allocEnv(BASE_ENV_SIZE))) + , baseEnvP(std::allocate_shared(traceable_allocator(), &mem.allocEnv(baseEnvCapacity))) , baseEnv(**baseEnvP) #else - , baseEnv(mem.allocEnv(BASE_ENV_SIZE)) + , baseEnv(mem.allocEnv(baseEnvCapacity)) #endif , staticBaseEnv{std::make_shared(nullptr, nullptr)} , countCalls(getEnv("NIX_COUNT_CALLS").value_or("0") != "0") @@ -500,6 +498,15 @@ void EvalState::checkURI(const std::string & uri0) throw RestrictedPathError("access to URI '%s' is forbidden in restricted mode", uri0); } +void EvalState::checkBaseEnvHasRoom(std::string_view name) const +{ + if (baseEnvDispl >= baseEnvCapacity) + throw Error( + "cannot register the builtin '%s': the base environment has only %d slots; increase baseEnvCapacity in eval.hh", + name, + baseEnvCapacity); +} + Value * EvalState::addConstant(const std::string & name, Value & v, Constant info) { Value * v2 = allocValue(); @@ -529,6 +536,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info) } /* Install value the base environment. */ + checkBaseEnvHasRoom(name); staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl); baseEnv.values[baseEnvDispl++] = v; const_cast(getBuiltins().attrs())->push_back(Attr(symbols.create(name2), v)); @@ -598,6 +606,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp) if (primOp.internal) internalPrimOps.emplace(primOp.name, RootValue(v)); else { + checkBaseEnvHasRoom(primOp.name); staticBaseEnv->vars.emplace_back(envName, baseEnvDispl); baseEnv.values[baseEnvDispl++] = v; const_cast(getBuiltins().attrs())->push_back(Attr(symbols.create(primOp.name), v)); diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index 68d907d78..4ccdb7cea 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -859,8 +859,21 @@ public: private: + /** + * Number of slots in the base environment, and the capacity of the + * `builtins` attribute set that mirrors it. Every global constant and + * every primop takes one of each (see `addConstant` and `addPrimOp`), so + * registering more builtins than this would write past the end of both. + * `nix __dump-language` registers all of them, including the ones gated + * behind experimental features; keep some headroom above what it needs. + */ + static constexpr size_t baseEnvCapacity = 160; + unsigned int baseEnvDispl = 0; + /** Throw instead of writing past the end of the base environment. */ + void checkBaseEnvHasRoom(std::string_view name) const; + void createBaseEnv(const EvalSettings & settings); Value * addConstant(const std::string & name, Value & v, Constant info); diff --git a/src/libexpr/include/nix/expr/tecnix/source-accessors.hh b/src/libexpr/include/nix/expr/tecnix/source-accessors.hh index 7899883da..9f9451881 100644 --- a/src/libexpr/include/nix/expr/tecnix/source-accessors.hh +++ b/src/libexpr/include/nix/expr/tecnix/source-accessors.hh @@ -17,6 +17,12 @@ const std::string & getManifestContent(const EvalState & state); const nlohmann::json & getManifestJson(const EvalState & state); StorePath getLegacyTectonixZoneStorePath(EvalState & state, std::string_view zonePath); ref getTecnixRepoAccessor(EvalState & state); +/** + * `.meta/manifest.json` as the Tecnix repo accessor serves it (the view that + * also fingerprints Tecnix dependencies), parsed once per evaluation. The + * read records nothing: callers record the part of the manifest they observe. + */ +const nlohmann::json & getTecnixManifestJson(EvalState & state); /** Resolve a checkout's HEAD to a commit SHA (throws if it has none). */ std::string resolveCheckoutHeadRev(const std::string & checkoutPath); diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 17947b13c..4ea87adda 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -5370,7 +5370,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) Value v; /* `builtins' must be first! */ - v.mkAttrs(buildBindings(140).finish()); + v.mkAttrs(buildBindings(baseEnvCapacity).finish()); addConstant( "builtins", v, diff --git a/src/libexpr/primops/tecnix.cc b/src/libexpr/primops/tecnix.cc index 06f8035fa..21cf78389 100644 --- a/src/libexpr/primops/tecnix.cc +++ b/src/libexpr/primops/tecnix.cc @@ -16,6 +16,7 @@ #include "nix/expr/tecnix/eval-cache.hh" #include "nix/expr/tecnix/source-accessors.hh" #include "nix/store/store-api.hh" +#include "nix/util/finally.hh" #include "nix/util/strings.hh" #include "nix/util/util.hh" @@ -121,6 +122,157 @@ static RegisterPrimOp primop_tecnixInternalSourceDepsList({ .impl = prim_tecnixInternalSourceDepsList, }); +// ============================================================================ +// builtins.tecnixMemoize f k +// `builtins.tecnixMemoize f` is `f`, memoized: calling it with a string `k` +// returns `f k`, evaluated at most once per EvalState for that `f` and `k` +// (once under tracking, once without). Under tracking, that evaluation +// records its source accesses into a label stored alongside the value; every +// later tracked consumer records that label as a child of its own frame (and +// inherits the value's label through the value-copy hooks), so all consumers +// share the same tracked dependencies without re-evaluating `f k`. +// ============================================================================ + +/* Calls whose `f k` evaluation is in progress on this thread. Re-entering one + (zone A loading itself transitively) can only be a genuine cycle; the guard + turns it into a catchable error instead of unbounded recursion. The key + views borrow from the calls' own arguments, which outlive their entries. */ +static thread_local std::vector tecnixMemoizeInProgress; + +static void prim_tecnixMemoize(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + auto * function = args[0]; + auto key = state.forceStringNoCtx( + *args[1], pos, "while evaluating the key passed to a function memoized by builtins.tecnixMemoize"); + EvalTecnixMemoizeKey::Lookup lookup{function, key}; + + auto * trackingCtx = currentTecnixThreadState.trackingContext; + + /* A result computed outside tracking carries no label, and the lazy parts + of a tracked result must only ever be forced under tracking, so tracked + and untracked calls never share a table: a tracked call reusing an + untracked result would record none of its sources, and an untracked call + forcing part of a tracked result would leave that part unlabelled for + every later tracked consumer. Untracked results are still memoized, in + their own table: `f` is typically recursive (zone A loads zone B), so + building them afresh would re-walk everything they share. */ + auto & memoCache = + trackingCtx ? *state.tecnixEvalData().trackedTecnixMemoizeCache : *state.tecnixEvalData().tecnixMemoizeCache; + + Value * result = nullptr; + auto sourceDeps = emptyEvalSourceAccessSetId; + if (memoCache.cvisit(lookup, [&](const auto & entry) { + result = *entry.second.value; + sourceDeps = entry.second.sourceDeps; + })) { + // Replay into this consumer the sources the entry's evaluation read. + if (trackingCtx) + recordTrackedSourceAccessSetDependency(*trackingCtx, sourceDeps); + v = *result; + return; + } + + for (auto & inProgress : tecnixMemoizeInProgress) + if (EvalTecnixMemoizeKey::Equal{}(inProgress, lookup)) + // A value that needs itself: fail as infinite recursion does, which + // builtins.tryEval cannot catch. + state + .error("builtins.tecnixMemoize: circular evaluation detected for key '%s'", key) + .atPos(pos) + .debugThrow(); + + tecnixMemoizeInProgress.push_back(lookup); + Finally popInProgress([&]() { tecnixMemoizeInProgress.pop_back(); }); + + // Evaluate `f k` outside any map bucket lock, so re-entrant misses (zone A + // loading zone B) can insert their own entries without deadlock. Under + // tracking, scope the evaluation so its accesses intern into one reusable + // label that every later consumer records. + // + // `f` gets a fresh copy of the key, not `args[1]`: that value carries the + // label of however this caller computed `k` (already recorded for this + // caller by the force above), and `f` forcing it inside the scope would + // store that label with the shared entry and replay it into every later + // caller. The key is a string without context, so the copy is equal. + auto * keyArg = state.allocValue(); + keyArg->mkString(key, state.mem); + result = state.allocValue(); + if (trackingCtx) { + TrackedSourceDepsScope scope(*trackingCtx); + state.callFunction(*function, *keyArg, *result, pos); + state.forceValue(*result, pos); + sourceDeps = scope.finish(result); + } else { + state.callFunction(*function, *keyArg, *result, pos); + state.forceValue(*result, pos); + } + + // Insert, but if a parallel evaluation raced ahead, reuse its entry so the + // table keeps a single shared value. + memoCache.try_emplace_and_cvisit( + EvalTecnixMemoizeKey{function, std::string(key)}, + EvalTecnixMemoizeEntry{}, + [&](auto & entry) { + entry.second.function = RootValue(function); + entry.second.value = RootValue(result); + entry.second.sourceDeps = sourceDeps; + }, + [&](const auto & entry) { + result = *entry.second.value; + sourceDeps = entry.second.sourceDeps; + }); + + if (trackingCtx) + recordTrackedSourceAccessSetDependency(*trackingCtx, sourceDeps); + v = *result; +} + +static RegisterPrimOp primop_tecnixMemoize({ + .name = "__tecnixMemoize", + .args = {"f", "k"}, + .doc = R"( + Memoize the function `f`: `builtins.tecnixMemoize f` is a function `g` + such that `g k` is always `f k`, but each result is computed once and + then shared. Bind `g` once and call it wherever `f k` is needed. + + Results are keyed by `f` and `k`. `f` is identified by its value (the + same binding), not by what it computes: every `g` made from the same + value of `f` shares its results, and a different function (another + system's loader, say) never sees them, even for equal keys. `k` must be + a string without string context; two keys are equal when they are the + same string. `f` is called with a string equal to `k`. + + Results live as long as the evaluator (one `EvalState`) and are never + freed. A call whose `f k` throws stores nothing, so the next call + evaluates `f k` again. + + Under Tecnix source tracking, the evaluation of `f k` records the + sources it reads, and every tracked call that reuses the result records + them too, so each target depends on everything the shared value was + computed from. The sources a caller read to compute `k` count for that + caller only; they are never stored with the shared result. Tracked and + untracked calls keep separate results, each computed once: neither kind + of call is ever handed the other's, since an untracked result has no + sources recorded, and forcing parts of a tracked result outside tracking + would lose theirs. + + If computing `f k` calls `g k` again, evaluation fails as it does for + infinite recursion, and `builtins.tryEval` cannot catch the error. + + Example: + + ```nix + let + zone = builtins.tecnixMemoize (zonePath: import ./load-zone.nix zonePath); + in + [ (zone "//areas/tools/dev") (zone "//areas/tools/dev") ] + ``` + + evaluates `import ./load-zone.nix "//areas/tools/dev"` once. + )", + .impl = prim_tecnixMemoize, +}); + // ============================================================================ // Shared helpers for Tecnix target evaluation and dependency-path tracking. // ============================================================================ diff --git a/src/libexpr/primops/tectonix.cc b/src/libexpr/primops/tectonix.cc index 8925e3437..00f235813 100644 --- a/src/libexpr/primops/tectonix.cc +++ b/src/libexpr/primops/tectonix.cc @@ -107,6 +107,146 @@ static RegisterPrimOp primop_worldManifestInverted({ .impl = prim_worldManifestInverted, }); +// ============================================================================ +// builtins.tectonixManifestEntry zonePath +// Returns null or { id = "W-xxxxxx" } for a single zone, recording only that +// entry as a tracked dependency (synthetic path .meta/manifest.json#) +// instead of the whole manifest file. The doc below is the one statement of +// the synthetic-key grammar; eval-cache.cc fingerprints the keys. +// ============================================================================ +static void prim_tectonixManifestEntry(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + auto zonePath = state.forceStringNoCtx( + *args[0], pos, "while evaluating the 'zonePath' argument to builtins.tectonixManifestEntry"); + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) { + // Manifest keys start with `//`. Anything else could spell one of the + // other key forms (`keys`, `id/...`), so it records the whole file. + if (zonePath.starts_with("//")) + ctx->recordAccess(".meta/manifest.json#" + std::string(zonePath)); + else + ctx->recordAccess(".meta/manifest.json"); + } + auto & manifest = getTecnixManifestJson(state); + auto it = manifest.find(std::string(zonePath)); + if (it == manifest.end() || !it->is_object() || !it->contains("id") || !(*it).at("id").is_string()) { + v.mkNull(); + return; + } + auto zoneAttrs = state.buildBindings(1); + zoneAttrs.alloc("id").mkString((*it).at("id").get(), state.mem); + v.mkAttrs(zoneAttrs); +} + +static RegisterPrimOp primop_tectonixManifestEntry({ + .name = "__tectonixManifestEntry", + .args = {"zonePath"}, + .doc = R"( + Get a single zone's manifest entry as a Nix attrset `{ id = "W-xxxxxx"; }`, + or null if the manifest has no entry for `zonePath` (or its entry has no + string `id`). Reads `.meta/manifest.json` through the Tecnix repository + view (the evaluated rev plus the checkout's uncommitted changes) that + every tracked read and every dependency fingerprint uses. + + Under Tecnix source tracking, the manifest builtins record what they + read of `.meta/manifest.json` as synthetic dependency keys, not the + whole file. These are all of them: + + - `.meta/manifest.json#`, from this builtin: the entry for + `zonePath`. Its fingerprint changes when the entry's JSON value (any + field, not only `id`) changes, appears or disappears. `zonePath` is + recorded verbatim; manifest keys start with `//`, and an argument + that does not records the whole file, `.meta/manifest.json`, instead. + - `.meta/manifest.json#keys`, from `builtins.tectonixManifestKeys`: the + set of zone paths (manifest keys). + - `.meta/manifest.json#id/`, from + `builtins.tectonixManifestIdToPath`: which zone path has the id + `zoneId` (the first in sorted order), or that none has. `` is + everything after `id/`, verbatim. + + Only keys that begin with `.meta/manifest.json#` are synthetic; every + other dependency key is a repo-relative path, even one that contains + `#`. A synthetic key depends on the file `.meta/manifest.json`: a + consumer of dependency keys that cannot interpret the part after the + `#` must treat any change to that file as affecting the key. + + Example: `builtins.tectonixManifestEntry "//areas/tools/dev"` returns `{ id = "W-123456"; }`. + )", + .impl = prim_tectonixManifestEntry, +}); + +// ============================================================================ +// builtins.tectonixManifestKeys +// Returns the sorted list of zone paths (manifest keys), recording only the +// key set as a tracked dependency (synthetic path .meta/manifest.json#keys). +// ============================================================================ +static void prim_tectonixManifestKeys(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(".meta/manifest.json#keys"); + auto & manifest = getTecnixManifestJson(state); + std::vector keys; + keys.reserve(manifest.size()); + for (auto & [path, value] : manifest.items()) + keys.push_back(path); + std::sort(keys.begin(), keys.end()); + auto list = state.buildList(keys.size()); + for (size_t i = 0; i < keys.size(); i++) { + auto * val = state.allocValue(); + val->mkString(keys[i], state.mem); + list[i] = val; + } + v.mkList(list); +} + +static RegisterPrimOp primop_tectonixManifestKeys({ + .name = "__tectonixManifestKeys", + .args = {}, + .doc = R"( + Get the sorted list of zone paths (manifest keys). Records only the key + set as a Tecnix tracked dependency (synthetic path .meta/manifest.json#keys), + not the whole manifest file. Only world-wide folds that enumerate every + zone should use this; per-target resolution should use tectonixManifestEntry. + Reads the manifest through the same Tecnix repository view as + tectonixManifestEntry, whose documentation lists every synthetic key. + )", + .impl = prim_tectonixManifestKeys, +}); + +// ============================================================================ +// builtins.tectonixManifestIdToPath zoneId +// Returns null or the zone path for a zone ID, recording only that id lookup +// as a tracked dependency (synthetic path .meta/manifest.json#id/). +// ============================================================================ +static void prim_tectonixManifestIdToPath(EvalState & state, const PosIdx pos, Value ** args, Value & v) +{ + auto zoneId = state.forceStringNoCtx( + *args[0], pos, "while evaluating the 'zoneId' argument to builtins.tectonixManifestIdToPath"); + if (auto ctx = currentTecnixThreadState.trackingContext; ctx) + ctx->recordAccess(".meta/manifest.json#id/" + std::string(zoneId)); + auto & manifest = getTecnixManifestJson(state); + for (auto & [path, value] : manifest.items()) { + if (value.contains("id") && value.at("id").is_string() && value.at("id").get() == zoneId) { + v.mkString(path, state.mem); + return; + } + } + v.mkNull(); +} + +static RegisterPrimOp primop_tectonixManifestIdToPath({ + .name = "__tectonixManifestIdToPath", + .args = {"zoneId"}, + .doc = R"( + Get the zone path for a zone ID, or null if not found. Records only that + id lookup as a Tecnix tracked dependency (synthetic path + .meta/manifest.json#id/), not the whole manifest file. Reads the + manifest through the same Tecnix repository view as + tectonixManifestEntry, whose documentation lists every synthetic key. + Example: `builtins.tectonixManifestIdToPath "W-123456"` returns `"//areas/tools/dev"`. + )", + .impl = prim_tectonixManifestIdToPath, +}); + static std::string normalizeTrackedRepoPath(std::string_view path); // ============================================================================ diff --git a/src/libexpr/tecnix/eval-cache.cc b/src/libexpr/tecnix/eval-cache.cc index 0e9302625..411399c4d 100644 --- a/src/libexpr/tecnix/eval-cache.cc +++ b/src/libexpr/tecnix/eval-cache.cc @@ -29,6 +29,8 @@ #include #include #include +#include +#include "nix/util/hash.hh" #include namespace nix { @@ -47,6 +49,9 @@ struct DependencyFingerprintThreadLocalCache { uint64_t generation = 0; boost::unordered_flat_map, StringViewHash, std::equal_to<>> fingerprints; + /** Cached parsed manifest JSON for synthetic path fingerprinting (per-batch). */ + std::optional cachedManifest; + bool manifestChecked = false; DependencyFingerprintThreadLocalCache() { @@ -1349,10 +1354,48 @@ static DependencyFingerprintThreadLocalCache & getDependencyFingerprintThreadCac if (threadCache.generation != cache.generation) { threadCache.generation = cache.generation; threadCache.fingerprints.clear(); + // The parsed manifest belongs to the accessor this generation reads. + threadCache.cachedManifest.reset(); + threadCache.manifestChecked = false; } return threadCache; } +static std::optional manifestPathFingerprint(const nlohmann::json & manifest, std::string_view suffix) +{ + constexpr std::string_view keysToken = "keys"; + if (suffix == keysToken) { + // Fingerprint the sorted key list + std::vector keys; + keys.reserve(manifest.size()); + for (auto & [k, v] : manifest.items()) + keys.push_back(k); + std::sort(keys.begin(), keys.end()); + std::string concatenated; + for (auto & k : keys) + concatenated += k + "\n"; + auto hash = hashString(HashAlgorithm::SHA256, concatenated); + return "manifest-keys:" + hash.to_string(HashFormat::Base64, false); + } + constexpr std::string_view idPrefix = "id/"; + if (suffix.starts_with(idPrefix)) { + auto zoneId = suffix.substr(idPrefix.size()); + for (auto & [path, value] : manifest.items()) { + if (value.contains("id") && value.at("id").is_string() && value.at("id").get() == zoneId) + return "manifest-id:" + std::string(zoneId) + "=" + path; + } + return "manifest-id:" + std::string(zoneId) + "=absent"; + } + // suffix is a zone path like "//zone/path" + auto it = manifest.find(std::string(suffix)); + if (it == manifest.end()) + return "manifest-entry:" + std::string(suffix) + "=absent"; + // Entries are { "id": "W-xxxxxx" }; fingerprint from the entry JSON + auto entryStr = it->dump(); + auto hash = hashString(HashAlgorithm::SHA256, entryStr); + return "manifest-entry:" + std::string(suffix) + "=" + hash.to_string(HashFormat::Base64, false); +} + static const std::optional * dependencyFingerprintCached(ref accessor, std::string_view path, DependencyFingerprintCache & cache) { @@ -1362,7 +1405,26 @@ dependencyFingerprintCached(ref accessor, std::string_view path, if (it != threadCache.fingerprints.end()) return &it->second; - auto [_, fp] = accessor->getFingerprint(CanonPath(path)); + std::optional fp; + constexpr std::string_view manifestPrefix = ".meta/manifest.json#"; + if (path.starts_with(manifestPrefix)) { + // Synthetic manifest path: fingerprint from manifest content, not the + // filesystem. This lets per-entry tracking avoid whole-file invalidation. + if (!threadCache.manifestChecked) { + threadCache.manifestChecked = true; + try { + auto content = accessor->readFile(CanonPath(".meta/manifest.json")); + threadCache.cachedManifest = nlohmann::json::parse(content); + } catch (...) { + threadCache.cachedManifest = std::nullopt; + } + } + if (threadCache.cachedManifest) + fp = manifestPathFingerprint(*threadCache.cachedManifest, path.substr(manifestPrefix.size())); + } else { + auto [_, rawFp] = accessor->getFingerprint(CanonPath(path)); + fp = std::move(rawFp); + } auto inserted = threadCache.fingerprints.emplace(std::string(path), std::move(fp)).first; return &inserted->second; } diff --git a/src/libexpr/tecnix/eval-data.hh b/src/libexpr/tecnix/eval-data.hh index d4140875f..b29c6bc87 100644 --- a/src/libexpr/tecnix/eval-data.hh +++ b/src/libexpr/tecnix/eval-data.hh @@ -5,6 +5,7 @@ #include "nix/expr/root-value.hh" #include "nix/expr/tecnix/access-set-graph.hh" +#include #include #include #include @@ -14,6 +15,7 @@ #include #include #include +#include #include namespace nix { @@ -47,6 +49,74 @@ struct EvalTecnixModuleCacheEntry using EvalTecnixModuleCache = boost::concurrent_flat_map; +/** + * A `builtins.tecnixMemoize` table key: the memoized function, identified by + * the address of its value, and the key it was called with. `Lookup` is the + * non-owning form, so a hit does not copy the key. + */ +struct EvalTecnixMemoizeKey +{ + const Value * function; + std::string key; + + struct Lookup + { + const Value * function; + std::string_view key; + }; + + struct Hash + { + using is_transparent = void; + + std::size_t operator()(const Lookup & k) const noexcept + { + std::size_t seed = 0; + boost::hash_combine(seed, k.function); + boost::hash_combine(seed, k.key); + return seed; + } + + std::size_t operator()(const EvalTecnixMemoizeKey & k) const noexcept + { + return (*this)(Lookup{k.function, k.key}); + } + }; + + struct Equal + { + using is_transparent = void; + + bool operator()(const auto & a, const auto & b) const noexcept + { + return a.function == b.function && a.key == b.key; + } + }; +}; + +/** + * A memoized `builtins.tecnixMemoize` result: the shared finished value of + * `f k` plus, in the tracked table, the source-access-set label recorded the + * one time it was evaluated. Every later tracked consumer records + * `sourceDeps` as a child of its own frame (and inherits the value's label + * through the normal value-copy hooks), so the shared value's tracked + * dependencies propagate to all consumers without re-evaluating `f k`. + */ +struct EvalTecnixMemoizeEntry +{ + /** Roots `f`, so that while this entry exists no other value can be + allocated at `f`'s address and inherit the entry. */ + RootValue function; + RootValue value; + EvalSourceAccessSetId sourceDeps = emptyEvalSourceAccessSetId; +}; + +using EvalTecnixMemoizeCache = boost::concurrent_flat_map< + EvalTecnixMemoizeKey, + EvalTecnixMemoizeEntry, + EvalTecnixMemoizeKey::Hash, + EvalTecnixMemoizeKey::Equal>; + struct EvalState::TecnixEvalData { /** @@ -106,6 +176,11 @@ struct EvalState::TecnixEvalData mutable std::once_flag tecnixRepoAccessorFlag; mutable std::optional> tecnixRepoAccessor; + /** `.meta/manifest.json` as `tecnixRepoAccessor` serves it, parsed + (thread-safe via once_flag). */ + mutable std::once_flag tecnixManifestJsonFlag; + mutable std::unique_ptr tecnixManifestJson; + /** * Virtual store path where the Tecnix repo-wide accessor is lazily mounted. * All repo subtree store paths are subpaths of this mount. @@ -138,6 +213,20 @@ struct EvalState::TecnixEvalData */ const ref tecnixModuleCache = make_ref(); + /** + * Memoization tables for `builtins.tecnixMemoize`, keyed by the memoized + * function and the key. Each entry holds the single shared result of + * `f k` and, in the tracked table, its recorded source-access-set label, + * so the (potentially expensive) `f k` evaluation runs once per EvalState + * and every tracked consumer inherits the same tracked dependencies. + * Tracked and untracked calls use separate tables, like + * `trackedFileEvalCache` and `fileEvalCache`, so a result never crosses + * between them. Misses evaluate outside any bucket lock, so re-entrant + * misses (zone A loading zone B) never deadlock. + */ + const ref tecnixMemoizeCache = make_ref(); + const ref trackedTecnixMemoizeCache = make_ref(); + /** Lazy-initialized set of zone IDs in sparse checkout (thread-safe via once_flag) */ mutable std::once_flag tectonixSparseCheckoutRootsFlag; mutable std::set tectonixSparseCheckoutRoots; diff --git a/src/libexpr/tecnix/repo-accessor.cc b/src/libexpr/tecnix/repo-accessor.cc index 98ff3734f..05dddcfd3 100644 --- a/src/libexpr/tecnix/repo-accessor.cc +++ b/src/libexpr/tecnix/repo-accessor.cc @@ -324,6 +324,13 @@ struct TecnixSourceAccessor : SourceAccessor return (isDirty(path) ? disk : clean)->readFile(path, sink, sizeCallback); } + /** `readFile` without recording the access, for a caller that records + its own, finer-grained observation of the file instead. */ + std::string readFileUnrecorded(const CanonPath & path) + { + return (isDirty(path) ? disk : clean)->readFile(path); + } + std::string readLink(const CanonPath & path) override { if (dumpPathDepth == 0) @@ -762,6 +769,22 @@ ref getTecnixRepoAccessor(EvalState & state) return *tecnixData(state)->tecnixRepoAccessor; } +const nlohmann::json & getTecnixManifestJson(EvalState & state) +{ + std::call_once(tecnixData(state)->tecnixManifestJsonFlag, [&state]() { + // getTecnixRepoAccessor always builds a TecnixSourceAccessor. + auto & accessor = static_cast(*getTecnixRepoAccessor(state)); + auto content = accessor.readFileUnrecorded(CanonPath(".meta/manifest.json")); + try { + tecnixData(state)->tecnixManifestJson = std::make_unique(nlohmann::json::parse(content)); + } catch (nlohmann::json::parse_error & e) { + throw Error( + "cannot parse '.meta/manifest.json' at commit '%s': %s", requireTectonixGitSha(state), e.what()); + } + }); + return *tecnixData(state)->tecnixManifestJson; +} + StorePath mountTecnixRepoAccessor(EvalState & state) { std::call_once(tecnixData(state)->tecnixRepoMountFlag, [&state]() { diff --git a/tests/functional/tecnix/base-env.sh b/tests/functional/tecnix/base-env.sh new file mode 100755 index 000000000..fd12a148d --- /dev/null +++ b/tests/functional/tecnix/base-env.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Every builtin must fit the evaluator's fixed-size base environment. The manual +# build runs `nix __dump-language`, which registers all of them, including the +# ones gated behind experimental features. Too many builtins used to write past +# the end of the environment (a heap buffer overflow only a sanitizer noticed). + +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" + +nix __dump-language > "$TEST_ROOT/language.json" +jq -e 'length > 100' "$TEST_ROOT/language.json" > /dev/null \ + || fail "__dump-language should describe every global builtin" + +echo "base environment holds every builtin" diff --git a/tests/functional/tecnix/manifest-tracking.sh b/tests/functional/tecnix/manifest-tracking.sh new file mode 100755 index 000000000..dc307c9c7 --- /dev/null +++ b/tests/functional/tecnix/manifest-tracking.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# Per-entry manifest tracking and the memoized zone loader. +# +# A target that reads one zone's manifest entry (`tectonixManifestEntry`), the +# key set (`tectonixManifestKeys`) or one id (`tectonixManifestIdToPath`) +# depends on exactly that, not on the whole `.meta/manifest.json`. So adding an +# unrelated zone leaves such a target's eval-cache row valid. A function +# memoized with `tecnixMemoize` shares one evaluation, and the source +# dependencies recorded during it, among every consumer, but never shares a +# result computed outside tracking with a tracked target (or the other way +# round). + +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" + +assert_jq() { + local json="$1" + local filter="$2" + local message="$3" + if ! jq -e "$filter" >/dev/null <<< "$json"; then + echo "$json" >&2 + fail "$message" + fi +} + +WORLD="$TEST_ROOT/manifest-world" +createGitRepo "$WORLD" + +write_manifest() { + # $1: extra manifest lines, already comma-terminated + cat > "$WORLD/.meta/manifest.json" < "$WORLD/shared.txt" +echo "unrelated one" > "$WORLD/unrelated.txt" +# Two files naming zone a, one memo key read from each by its own target. +printf a > "$WORLD/key1.txt" +printf a > "$WORLD/key2.txt" + +cat > "$WORLD/resolve.nix" <<'EOF' +args: +let + drv = name: suffix: { + drvPath = "/nix/store/00000000000000000000000000000000-${name}-${suffix}.drv"; + outputName = "out"; + }; + entry = name: builtins.tectonixManifestEntry "//zones/${name}"; + # Two consumers of one memoized load: the load reads shared.txt once. + loaded = builtins.tecnixMemoize (_: builtins.readFile ./shared.txt) "shared"; + # Memoized lookups that every consumer calls afresh, so every consumer after + # the first is a memo hit rather than a force of one shared thunk. + zoneId = builtins.tecnixMemoize (name: (entry name).id); + zoneRecord = builtins.tecnixMemoize (name: { id = (entry name).id; }); +in +{ + allTargetNames = [ "a" "b" "keys" "id" "m1" "m2" "whole" "za1" "za2" "peek" "fill-record" "prefilled" "record-field" "agree" "not-a-zone" "key1" "key2" ]; + resolve = name: + if name == "whole" then drv name (toString (builtins.length (builtins.attrNames builtins.unsafeTectonixInternalManifest))) + else if name == "keys" then drv name (toString (builtins.length (builtins.tectonixManifestKeys))) + else if name == "id" then drv name ( + let path = builtins.tectonixManifestIdToPath "W-000001"; in if path == null then "absent" else path + ) + else if name == "m1" || name == "m2" then drv name (builtins.hashString "sha256" loaded) + else if name == "za1" || name == "za2" || name == "prefilled" then drv name (zoneId "a") + # Hands both memoized lookups, unforced, to whoever reads the value. + else if name == "peek" then drv name "peek" // { zoneId = zoneId "a"; recordId = (zoneRecord "a").id; } + # Fills the memoized record without forcing its field. + else if name == "fill-record" then builtins.seq (zoneRecord "a") (drv name "record") + else if name == "record-field" then drv name (zoneRecord "a").id + # What the manifest builtins report about zone a, beside what reading + # .meta/manifest.json itself shows. + else if name == "agree" then + let + manifest = builtins.fromJSON (builtins.readFile ./.meta/manifest.json); + id = manifest."//zones/a".id; + in drv name "agree" // { + seen = { + builtins = { inherit (entry "a") id; keys = builtins.tectonixManifestKeys; path = builtins.tectonixManifestIdToPath id; }; + file = { inherit id; keys = builtins.attrNames manifest; path = "//zones/a"; }; + }; + } + # Every manifest key starts with `//`; "keys" is not one. + else if name == "not-a-zone" then drv name (if builtins.tectonixManifestEntry "keys" == null then "none" else "some") + else if name == "key1" || name == "key2" then drv name (zoneId (builtins.readFile ./${name}.txt)) + else drv name (entry name).id; +} +EOF + +commit_world() { + git -C "$WORLD" add -A + git -C "$WORLD" commit -q -m "$1" + git -C "$WORLD" rev-parse HEAD +} + +REV1=$(commit_world "two zones") + +TARGETS='[ "a" "b" "keys" "id" "m1" "m2" "whole" "za1" "za2" ]' + +# Dependency sets (target -> path -> fingerprint) of every target at $1. +# Extra nix flags follow. +deps_at() { + local rev="$1" + shift + nix eval --json -v \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tectonix-git-dir "$WORLD/.git" \ + --option tectonix-git-sha "$rev" \ + "$@" \ + --expr "builtins.listToAttrs (map (r: { name = r.target; value = r.dependencies; }) (builtins.tecnixTargets { + gitDir = \"$WORLD/.git\"; resolver = \"resolve.nix\"; args = { system = \"test-system\"; }; + rev = \"$rev\"; targets = $TARGETS; includeDependencies = true; includeTargets = false; }))" +} + +nocache_deps_at() { + deps_at "$1" --option tecnix-eval-cache false +} + +CACHE_HOME="$TEST_ROOT/manifest-cache-home" +cached_deps_at() { + XDG_CACHE_HOME="$CACHE_HOME" deps_at "$1" --option tecnix-eval-cache true --pure-eval +} + +# `hits ` asserts each target was served from the cache; +# `misses` asserts each was re-evaluated. +hits() { + local err="$1"; shift + for t in "$@"; do + grepQuiet "dependency cache hit for '$t'" "$err" || fail "$t should be a cache hit ($err)" + done +} +misses() { + local err="$1"; shift + for t in "$@"; do + grepQuiet "dependency cache miss, evaluating '$t'" "$err" || fail "$t should be re-evaluated ($err)" + done +} + +echo "Testing what each target depends on..." +deps1=$(nocache_deps_at "$REV1") +assert_jq "$deps1" '.a | keys == [".meta/manifest.json#//zones/a", "resolve.nix"]' \ + "a should depend on its own manifest entry, not the whole manifest" +assert_jq "$deps1" '.b | keys == [".meta/manifest.json#//zones/b", "resolve.nix"]' \ + "b should depend on its own manifest entry, not the whole manifest" +assert_jq "$deps1" '.keys | keys == [".meta/manifest.json#keys", "resolve.nix"]' \ + "keys should depend on the key set only" +assert_jq "$deps1" '.id | keys == [".meta/manifest.json#id/W-000001", "resolve.nix"]' \ + "id should depend on one id lookup only" +assert_jq "$deps1" '[.a, .b, .keys, .id, .m1, .m2, .za1, .za2 | has(".meta/manifest.json")] | any | not' \ + "the per-entry targets should not depend on the whole manifest file" +assert_jq "$deps1" '.whole | has(".meta/manifest.json")' \ + "reading the whole manifest (builtins.unsafeTectonixInternalManifest) should still depend on the file" +assert_jq "$deps1" '.a[".meta/manifest.json#//zones/a"] != .b[".meta/manifest.json#//zones/b"]' \ + "entry fingerprints should differ between zones" +assert_jq "$deps1" '.m1 == .m2 and (.m1 | has("shared.txt"))' \ + "both consumers of a memoized load should inherit the load's dependencies" +assert_jq "$deps1" '[.za1, .za2 | keys == [".meta/manifest.json#//zones/a", "resolve.nix"]] | all' \ + "a memo hit should replay the dependencies recorded when its entry was filled" +# An argument that is not a zone path could spell another key form (here +# `#keys`, the key set), so it records the whole manifest instead. +not_a_zone=$(TARGETS='[ "not-a-zone" ]' nocache_deps_at "$REV1") +assert_jq "$not_a_zone" '."not-a-zone" | keys == [".meta/manifest.json", "resolve.nix"]' \ + "looking up a non-zone path should depend on the whole manifest" +# How a caller computed a memo key is its own dependency, never the shared +# entry's. Either target may fill the entry first, so check both. +via=$(TARGETS='[ "key1" "key2" ]' nocache_deps_at "$REV1") +assert_jq "$via" '(.key1 | keys) == [".meta/manifest.json#//zones/a", "key1.txt", "resolve.nix"] + and (.key2 | keys) == [".meta/manifest.json#//zones/a", "key2.txt", "resolve.nix"]' \ + "a memo caller should depend on the file it read the key from, and on no other caller's" + +echo "Testing the eval cache across commits..." +cached_deps_at "$REV1" > "$TEST_ROOT/cold.json" 2> "$TEST_ROOT/cold.err" +misses "$TEST_ROOT/cold.err" a b keys id m1 m2 whole za1 za2 +[[ "$(jq -S . < "$TEST_ROOT/cold.json")" == "$(jq -S . <<< "$deps1")" ]] \ + || fail "the cached evaluation should report the same dependencies" + +# Adding an unrelated zone changes the manifest file and the key set. Only the +# targets that enumerate the keys or read the whole file are invalidated. +write_manifest ' "//zones/c": { "id": "W-000003" },' +REV2=$(commit_world "add an unrelated zone") +cached_deps_at "$REV2" > /dev/null 2> "$TEST_ROOT/add-zone.err" +hits "$TEST_ROOT/add-zone.err" a b id m1 m2 za1 za2 +misses "$TEST_ROOT/add-zone.err" keys whole + +# Changing one zone's entry invalidates that zone's target and nothing else. +sed -i.bak 's/W-000002/W-000009/' "$WORLD/.meta/manifest.json" && rm "$WORLD/.meta/manifest.json.bak" +REV3=$(commit_world "change b's id") +cached_deps_at "$REV3" > "$TEST_ROOT/change-b.json" 2> "$TEST_ROOT/change-b.err" +hits "$TEST_ROOT/change-b.err" a id keys m1 m2 za1 za2 +misses "$TEST_ROOT/change-b.err" b whole +assert_jq "$(cat "$TEST_ROOT/change-b.json")" \ + ".b[\".meta/manifest.json#//zones/b\"] != $(jq '.b[".meta/manifest.json#//zones/b"]' <<< "$deps1")" \ + "b's recorded entry fingerprint should follow its entry" + +# An id lookup is invalidated when the id moves, and only then. +sed -i.bak 's/W-000001/W-000010/' "$WORLD/.meta/manifest.json" && rm "$WORLD/.meta/manifest.json.bak" +REV4=$(commit_world "renumber a") +cached_deps_at "$REV4" > /dev/null 2> "$TEST_ROOT/renumber-a.err" +hits "$TEST_ROOT/renumber-a.err" b keys m1 m2 +misses "$TEST_ROOT/renumber-a.err" a id whole za1 za2 + +# A memoized load's dependencies reach every consumer: changing what the load +# read invalidates both, not just whichever consumer evaluated it first. +echo "shared two" > "$WORLD/shared.txt" +REV5=$(commit_world "change the shared file") +cached_deps_at "$REV5" > /dev/null 2> "$TEST_ROOT/shared.err" +hits "$TEST_ROOT/shared.err" a b id keys whole za1 za2 +misses "$TEST_ROOT/shared.err" m1 m2 + +# A file nothing read leaves every row valid. +echo "unrelated two" > "$WORLD/unrelated.txt" +REV6=$(commit_world "change an unrelated file") +cached_deps_at "$REV6" > /dev/null 2> "$TEST_ROOT/unrelated.err" +hits "$TEST_ROOT/unrelated.err" a b keys id m1 m2 whole za1 za2 + +# A memoized result computed outside tracking must never reach a tracked +# target without its dependencies, in either direction: an untracked call that +# comes first, and an untracked call that forces part of an entry a tracked +# target filled. `peek` hands both lookups, unforced, to the top-level +# expression, which forces them outside tracking before the tracked targets +# run in the same evaluation. Extra nix flags follow the rev. +prefill_deps_at() { + local rev="$1" + shift + nix eval --json -v \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tectonix-git-dir "$WORLD/.git" \ + --option tectonix-git-sha "$rev" \ + "$@" \ + --expr "let + call = includeTargets: targets: builtins.tecnixTargets { + gitDir = \"$WORLD/.git\"; resolver = \"resolve.nix\"; args = { system = \"test-system\"; }; + rev = \"$rev\"; inherit targets includeTargets; includeDependencies = true; }; + peek = (builtins.head (call true [ \"peek\" ])).value; + outsideTracking = builtins.seq peek.zoneId (builtins.seq (call false [ \"fill-record\" ]) peek.recordId); + in builtins.seq outsideTracking (builtins.listToAttrs (map (r: { name = r.target; value = r.dependencies; }) + (call false [ \"prefilled\" \"record-field\" ])))" +} + +echo "Testing memoized results computed outside tracking..." +PREFILLED_DEPS='[.prefilled, ."record-field" | keys == [".meta/manifest.json#//zones/a", "resolve.nix"]] | all' +# First with tracking from includeDependencies alone, then through the eval +# cache, where a missing dependency turns into a stale warm hit. +assert_jq "$(prefill_deps_at "$REV6" --option tecnix-eval-cache false)" "$PREFILLED_DEPS" \ + "tracked targets should depend on the entry behind memoized values first touched outside tracking" + +PREFILL_CACHE_HOME="$TEST_ROOT/manifest-prefill-cache-home" +XDG_CACHE_HOME="$PREFILL_CACHE_HOME" prefill_deps_at "$REV6" --option tecnix-eval-cache true --pure-eval \ + > "$TEST_ROOT/prefill-cold.json" 2> "$TEST_ROOT/prefill-cold.err" +misses "$TEST_ROOT/prefill-cold.err" prefilled record-field +assert_jq "$(cat "$TEST_ROOT/prefill-cold.json")" "$PREFILLED_DEPS" \ + "cached tracked targets should depend on the entry behind memoized values first touched outside tracking" + +# The reviewer's reproduction: renumber zone a, and the warm run must +# re-evaluate instead of serving the old id. +sed -i.bak 's/W-000010/W-000011/' "$WORLD/.meta/manifest.json" && rm "$WORLD/.meta/manifest.json.bak" +REV7=$(commit_world "renumber a again") +XDG_CACHE_HOME="$PREFILL_CACHE_HOME" prefill_deps_at "$REV7" --option tecnix-eval-cache true --pure-eval \ + > /dev/null 2> "$TEST_ROOT/prefill-warm.err" +misses "$TEST_ROOT/prefill-warm.err" prefilled record-field + +# The manifest builtins read .meta/manifest.json through the repo accessor, +# like every file the resolver reads and like the fingerprints of their +# dependency keys. So evaluating an older rev from a checkout whose HEAD has +# moved on shows them that rev's manifest, not the working tree's. +echo "Testing the manifest builtins at an older rev of a checkout..." +seen=$(nix eval --json \ + --extra-experimental-features 'nix-command' \ + --option lazy-trees true \ + --option tecnix-eval-cache false \ + --expr "(builtins.tecnixTargets { gitDir = \"$WORLD/.git\"; resolver = \"resolve.nix\"; args = { system = \"test-system\"; }; + rev = \"$REV1\"; checkoutPath = \"$WORLD\"; targets = [ \"agree\" ]; }).agree.seen") +assert_jq "$seen" '.file.id == "W-000001"' "the resolver should read the evaluated rev's manifest" +assert_jq "$seen" '.builtins == .file' "the manifest builtins should see the manifest the resolver reads" + +echo "All manifest tracking tests passed!" diff --git a/tests/functional/tecnix/memoize.sh b/tests/functional/tecnix/memoize.sh new file mode 100755 index 000000000..a90525e21 --- /dev/null +++ b/tests/functional/tecnix/memoize.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# builtins.tecnixMemoize: `g = builtins.tecnixMemoize f` is `f`, with each +# `f k` evaluated once per evaluation and shared. What tracked calls record, and +# that tracked and untracked results stay apart, is in manifest-tracking.sh. + +# shellcheck disable=SC2016 +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" + +memo_eval() { + nix eval --raw --extra-experimental-features 'nix-command' --expr "$1" +} + +# How often `f` ran for `k`, per the `computing ` traces in file $1. +computed() { + grep -c "trace: computing $2\$" "$1" || true +} + +out=$(memo_eval 'builtins.tecnixMemoize (x: x + "!") "k"') +[[ "$out" == "k!" ]] || fail "g k should be f k, got '$out'" + +# Every call for a key after the first reuses its result; each key is computed once. +memo_eval 'let g = builtins.tecnixMemoize (x: builtins.trace "computing ${x}" (x + "!")); + in builtins.concatStringsSep "|" [ (g "k") (g "k") (g "k2") (g "k") ]' \ + > "$TEST_ROOT/once.out" 2> "$TEST_ROOT/once.err" +[[ "$(cat "$TEST_ROOT/once.out")" == "k!|k!|k2!|k!" ]] || fail "unexpected results: $(cat "$TEST_ROOT/once.out")" +[[ "$(computed "$TEST_ROOT/once.err" k)" == 1 ]] || fail "f should run once for 'k'" +[[ "$(computed "$TEST_ROOT/once.err" k2)" == 1 ]] || fail "f should run once for 'k2'" + +# Results belong to the function: another function never sees them, even for an equal key. +out=$(memo_eval 'let g1 = builtins.tecnixMemoize (x: x + "!"); g2 = builtins.tecnixMemoize (x: x + "?"); + in builtins.seq (g1 "k") "${g1 "k"}|${g2 "k"}"') +[[ "$out" == "k!|k?" ]] || fail "each function should get its own results, got '$out'" + +# Every g made from the same value of f shares them. +memo_eval 'let f = x: builtins.trace "computing ${x}" (x + "!"); g1 = builtins.tecnixMemoize f; g2 = builtins.tecnixMemoize f; + in "${g1 "k"}|${g2 "k"}"' > "$TEST_ROOT/same-f.out" 2> "$TEST_ROOT/same-f.err" +[[ "$(cat "$TEST_ROOT/same-f.out")" == "k!|k!" ]] || fail "unexpected results: $(cat "$TEST_ROOT/same-f.out")" +[[ "$(computed "$TEST_ROOT/same-f.err" k)" == 1 ]] || fail "memoizing the same f twice should share results" + +# Only strings are keys. +expectStderr 1 nix eval --extra-experimental-features 'nix-command' --expr 'builtins.tecnixMemoize (x: x) 1' \ + | grepQuiet "expected a string but found an integer" + +# A call that needs its own result fails like infinite recursion: tryEval can't catch it. +expectStderr 1 nix eval --extra-experimental-features 'nix-command' --expr 'let g = builtins.tecnixMemoize (k: g k); in builtins.tryEval (g "a")' \ + | grepQuiet "circular evaluation detected" + +# Re-entering for a different key is not a cycle. +out=$(memo_eval 'let g = builtins.tecnixMemoize (k: if k == "root" then g "child" else k + "-val"); in g "root"') +[[ "$out" == "child-val" ]] || fail "expected 'child-val', got '$out'" diff --git a/tests/functional/tecnix/meson.build b/tests/functional/tecnix/meson.build index 3b0b37e8a..540675a5b 100644 --- a/tests/functional/tecnix/meson.build +++ b/tests/functional/tecnix/meson.build @@ -2,9 +2,12 @@ suites += { 'name' : 'tecnix', 'deps' : [], 'tests' : [ + 'base-env.sh', 'builtins.sh', 'gc.sh', + 'manifest-tracking.sh', 'ingestion-stats.sh', + 'memoize.sh', ], 'workdir' : meson.current_source_dir(), }