Track manifest entries instead of the whole manifest; add tectonixMemo

plans/tecnix-target-eval-caching/explainer.md

@@ -202,6 +202,24 @@ This is the precise sense in which the system works across commits. After a comm
 
 The same property makes the system fail-safe. A row is never believed on the strength of its key; it is believed only when proven. Format changes, corruption, and storage defects therefore all degrade into cache misses, never into wrong answers. In practice, the difference between a cold run and a warm run is the difference between roughly a minute of full tracked evaluation and well under a second — several orders of magnitude, with the gap consisting precisely of "all of evaluation" versus "fingerprint checks and output construction."
 
+### 4.2 Synthetic keys: parts of the manifest
+
+Not every closure key is a file. A target that looks up one zone in `.meta/manifest.json` depends on that zone's entry, not on the whole file, and recording the file would invalidate every such target whenever any zone is added or renumbered. The manifest builtins therefore record *synthetic* keys, each naming one observation of the manifest. This is the complete list:
+
+| key | recorded by | fingerprint changes exactly when |
+|---|---|---|
+| `.meta/manifest.json#<zonePath>` | `builtins.tectonixManifestEntry zonePath` | the JSON value of that zone's entry (any field, not only `id`) changes, appears, or disappears |
+| `.meta/manifest.json#keys` | `builtins.tectonixManifestKeys` | the set of zone paths (manifest keys) changes |
+| `.meta/manifest.json#id/<zoneId>` | `builtins.tectonixManifestIdToPath zoneId` | a different zone path, or none, is the first in sorted order whose entry has the id `zoneId` |
+
+The builtins read the manifest through the same accessor as every other tracked read (§7), so a value and the fingerprint of its key always describe the same manifest. The grammar is exact:
+
+- A key is synthetic if and only if it begins with `.meta/manifest.json#`. Every other key is a repo-relative path, even one that contains `#`; splitting arbitrary keys at `#` is wrong.
+- The fragment after that prefix is `keys`, or `id/` followed by a zone id (everything after `id/`, verbatim), or a zone path, which begins with `//`. `tectonixManifestEntry` given an argument that does not begin with `//` records the whole file, `.meta/manifest.json`, so the three forms never overlap.
+- A synthetic key depends on the file `.meta/manifest.json`. A consumer of closure keys (for example, one mapping changed files to affected targets) that cannot interpret a fragment must treat any change to that file as affecting the key; one that can compares the named observation of the manifest before and after the change.
+
+Reading the whole manifest (the `unsafeTectonixInternalManifest*` builtins, or `builtins.readFile` of the file) still records the plain path `.meta/manifest.json`.
+
 ---
 
 ## 5. A Worked Example
@@ -327,7 +345,7 @@ Clean paths are served from git's object store — no checkout is required, and
 
 In a worldtree sandbox (`tectonix-worldtree-socket` set) there is no git repository to read; the clean tree is instead the daemon's immutable FUSE projection of the pinned commit, with repo-relative paths mapped through the committed manifest to per-zone views. The fingerprint vocabulary is unchanged: directories read their exact committed tree oid from the projection's `user.worldtree.tree-oid` xattr, and regular files read the daemon's `user.worldtree.blob-oid` xattr when it is served — one O(1) metadata read each. Symlinks (which cannot carry user xattrs; `getxattr` would follow the link and answer for its target, a different git object) and files under daemons that do not serve blob oids fall back to hashing their bytes as git blobs — a blob oid is a pure function of content — memoized in memory for the accessor's lifetime, which the immutable projection makes sound: each unique file is hashed at most once per evaluation. Because every mechanism emits identical fingerprint strings, closures produced under one backend validate under the other. Two caveats follow from the projection being zone-granular: committed paths outside every visible zone do not exist in this view and observe as `absent`, and zone-ancestor directories are synthesized with a composite `worldtree-union:` fingerprint outside the git vocabulary (their listing genuinely differs from the full git tree, so cross-backend cache misses there are correct, not conservative).
 
-The evaluator's own caches require care, since any of them could silently absorb a read. Each — the file-evaluation cache, the import-resolution cache, the source-to-store copy cache — either maintains a separate tracked-domain instance or replays its provenance on a hit. The general rule: *any cache capable of skipping a physical read must replay the provenance of that read.*
+The evaluator's own caches require care, since any of them could silently absorb a read. Each — the file-evaluation cache, the import-resolution cache, the source-to-store copy cache, the `builtins.tecnixMemoize` tables — maintains a separate tracked-domain instance, replays its provenance on a hit, or both. The general rule: *any cache capable of skipping a physical read must replay the provenance of that read.*
 
 The failure policy throughout is to fail closed. If `git status` fails, evaluation raises an error rather than assuming a clean tree, because the dirty overlay is load-bearing for closure validity. If a path cannot be fingerprinted, evaluation raises an error rather than emitting a partial closure. If evaluation reaches something the closure format cannot represent, it raises an error rather than under-tracking. Every failure mode resolves to a cache miss or a visible error; none resolves to a plausible wrong answer.
 

src/libexpr-tests/meson.build

@@ -68,6 +68,7 @@ sources = files(
   'search-path.cc',
   'tecnix-dependency-tracking.cc',
   'tecnix-eval-cache-row.cc',
+  'tecnix-manifest-fingerprint.cc',
   'tecnix-worldtree.cc',
   'tectonix.cc',
   'trivial.cc',

src/libexpr-tests/tecnix-manifest-fingerprint.cc

@@ -0,0 +1,110 @@
+#include <optional>
+#include <string>
+#include <string_view>
+
+#include <gtest/gtest.h>
+
+#include "nix/expr/tecnix/eval-cache.hh"
+#include "nix/util/memory-source-accessor.hh"
+
+namespace nix {
+
+namespace {
+
+/** A source tree whose only file is `.meta/manifest.json` with the given content. */
+ref<SourceAccessor> treeWithManifest(std::string manifest)
+{
+    auto accessor = make_ref<MemorySourceAccessor>();
+    accessor->addFile(CanonPath(".meta/manifest.json"), std::move(manifest));
+    return accessor;
+}
+
+/** Fingerprint one synthetic manifest path, with a fresh cache like a fresh evaluation. */
+std::optional<std::string> fingerprint(ref<SourceAccessor> accessor, std::string_view syntheticPath)
+{
+    DependencyFingerprintCache cache;
+    return dependencyFingerprint(accessor, syntheticPath, cache);
+}
+
+constexpr auto twoZones = R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000002" } })";
+constexpr auto threeZones =
+    R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000002" }, "//zones/c": { "id": "W-000003" } })";
+constexpr auto bRenumbered = R"({ "//zones/a": { "id": "W-000001" }, "//zones/b": { "id": "W-000009" } })";
+
+} // namespace
+
+TEST(TecnixManifestFingerprint, EntryFingerprintIgnoresOtherEntries)
+{
+    auto entryA = ".meta/manifest.json#//zones/a";
+    auto before = fingerprint(treeWithManifest(twoZones), entryA);
+    ASSERT_TRUE(before);
+    EXPECT_EQ(fingerprint(treeWithManifest(threeZones), entryA), before);
+    EXPECT_EQ(fingerprint(treeWithManifest(bRenumbered), entryA), before);
+}
+
+TEST(TecnixManifestFingerprint, EntryFingerprintFollowsItsOwnEntry)
+{
+    auto entryB = ".meta/manifest.json#//zones/b";
+    EXPECT_NE(fingerprint(treeWithManifest(twoZones), entryB), fingerprint(treeWithManifest(bRenumbered), entryB));
+    EXPECT_NE(
+        fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#//zones/a"),
+        fingerprint(treeWithManifest(twoZones), entryB));
+}
+
+TEST(TecnixManifestFingerprint, AbsentEntryHasItsOwnFingerprint)
+{
+    auto entryC = ".meta/manifest.json#//zones/c";
+    auto absent = fingerprint(treeWithManifest(twoZones), entryC);
+    ASSERT_TRUE(absent);
+    EXPECT_NE(fingerprint(treeWithManifest(threeZones), entryC), absent);
+}
+
+TEST(TecnixManifestFingerprint, KeyFingerprintTracksTheKeySetOnly)
+{
+    auto keys = ".meta/manifest.json#keys";
+    auto before = fingerprint(treeWithManifest(twoZones), keys);
+    ASSERT_TRUE(before);
+    EXPECT_NE(fingerprint(treeWithManifest(threeZones), keys), before);
+    // Renumbering a zone changes no key.
+    EXPECT_EQ(fingerprint(treeWithManifest(bRenumbered), keys), before);
+}
+
+TEST(TecnixManifestFingerprint, KeyFingerprintIgnoresKeyOrder)
+{
+    auto reordered = R"({ "//zones/b": { "id": "W-000002" }, "//zones/a": { "id": "W-000001" } })";
+    EXPECT_EQ(
+        fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys"),
+        fingerprint(treeWithManifest(reordered), ".meta/manifest.json#keys"));
+}
+
+TEST(TecnixManifestFingerprint, IdFingerprintTracksWhereTheIdPoints)
+{
+    auto idOfB = ".meta/manifest.json#id/W-000002";
+    auto before = fingerprint(treeWithManifest(twoZones), idOfB);
+    ASSERT_TRUE(before);
+    // Adding an unrelated zone leaves the lookup alone; renumbering removes the id.
+    EXPECT_EQ(fingerprint(treeWithManifest(threeZones), idOfB), before);
+    EXPECT_NE(fingerprint(treeWithManifest(bRenumbered), idOfB), before);
+}
+
+TEST(TecnixManifestFingerprint, FollowsTheAccessorAcrossCaches)
+{
+    // The parsed manifest must not outlive the cache it was read for: another
+    // evaluation on the same thread fingerprints against its own commit.
+    auto entryB = ".meta/manifest.json#//zones/b";
+    auto first = fingerprint(treeWithManifest(twoZones), entryB);
+    auto second = fingerprint(treeWithManifest(bRenumbered), entryB);
+    EXPECT_NE(first, second);
+    EXPECT_NE(
+        fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys"),
+        fingerprint(treeWithManifest(threeZones), ".meta/manifest.json#keys"));
+}
+
+TEST(TecnixManifestFingerprint, NoManifestMeansNoFingerprint)
+{
+    EXPECT_EQ(fingerprint(make_ref<MemorySourceAccessor>(), ".meta/manifest.json#keys"), std::nullopt);
+    // ...and that must not leak into the next tree either.
+    EXPECT_TRUE(fingerprint(treeWithManifest(twoZones), ".meta/manifest.json#keys"));
+}
+
+} // namespace nix

src/libexpr/eval.cc

@@ -249,8 +249,6 @@ static Symbol getName(const AttrName & name, EvalState & state, Env & env)
     }
 }
 
-static constexpr size_t BASE_ENV_SIZE = 140;
-
 EvalMemory::EvalMemory()
 {
     assertGCInitialized();
@@ -338,10 +336,10 @@ EvalState::EvalState(
     , regexCache(makeRegexCache())
     , tecnixData(std::make_unique<TecnixEvalData>())
 #if NIX_USE_BOEHMGC
-    , baseEnvP(std::allocate_shared<Env *>(traceable_allocator<Env *>(), &mem.allocEnv(BASE_ENV_SIZE)))
+    , baseEnvP(std::allocate_shared<Env *>(traceable_allocator<Env *>(), &mem.allocEnv(baseEnvCapacity)))
     , baseEnv(**baseEnvP)
 #else
-    , baseEnv(mem.allocEnv(BASE_ENV_SIZE))
+    , baseEnv(mem.allocEnv(baseEnvCapacity))
 #endif
     , staticBaseEnv{std::make_shared<StaticEnv>(nullptr, nullptr)}
     , countCalls(getEnv("NIX_COUNT_CALLS").value_or("0") != "0")
@@ -500,6 +498,15 @@ void EvalState::checkURI(const std::string & uri0)
     throw RestrictedPathError("access to URI '%s' is forbidden in restricted mode", uri0);
 }
 
+void EvalState::checkBaseEnvHasRoom(std::string_view name) const
+{
+    if (baseEnvDispl >= baseEnvCapacity)
+        throw Error(
+            "cannot register the builtin '%s': the base environment has only %d slots; increase baseEnvCapacity in eval.hh",
+            name,
+            baseEnvCapacity);
+}
+
 Value * EvalState::addConstant(const std::string & name, Value & v, Constant info)
 {
     Value * v2 = allocValue();
@@ -529,6 +536,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info)
         }
 
         /* Install value the base environment. */
+        checkBaseEnvHasRoom(name);
         staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl);
         baseEnv.values[baseEnvDispl++] = v;
         const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(name2), v));
@@ -598,6 +606,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
     if (primOp.internal)
         internalPrimOps.emplace(primOp.name, RootValue(v));
     else {
+        checkBaseEnvHasRoom(primOp.name);
         staticBaseEnv->vars.emplace_back(envName, baseEnvDispl);
         baseEnv.values[baseEnvDispl++] = v;
         const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(primOp.name), v));

src/libexpr/include/nix/expr/eval.hh

@@ -859,8 +859,21 @@ public:
 
 private:
 
+    /**
+     * Number of slots in the base environment, and the capacity of the
+     * `builtins` attribute set that mirrors it. Every global constant and
+     * every primop takes one of each (see `addConstant` and `addPrimOp`), so
+     * registering more builtins than this would write past the end of both.
+     * `nix __dump-language` registers all of them, including the ones gated
+     * behind experimental features; keep some headroom above what it needs.
+     */
+    static constexpr size_t baseEnvCapacity = 160;
+
     unsigned int baseEnvDispl = 0;
 
+    /** Throw instead of writing past the end of the base environment. */
+    void checkBaseEnvHasRoom(std::string_view name) const;
+
     void createBaseEnv(const EvalSettings & settings);
 
     Value * addConstant(const std::string & name, Value & v, Constant info);

src/libexpr/include/nix/expr/tecnix/source-accessors.hh

@@ -17,6 +17,12 @@ const std::string & getManifestContent(const EvalState & state);
 const nlohmann::json & getManifestJson(const EvalState & state);
 StorePath getLegacyTectonixZoneStorePath(EvalState & state, std::string_view zonePath);
 ref<SourceAccessor> getTecnixRepoAccessor(EvalState & state);
+/**
+ * `.meta/manifest.json` as the Tecnix repo accessor serves it (the view that
+ * also fingerprints Tecnix dependencies), parsed once per evaluation. The
+ * read records nothing: callers record the part of the manifest they observe.
+ */
+const nlohmann::json & getTecnixManifestJson(EvalState & state);
 
 /** Resolve a checkout's HEAD to a commit SHA (throws if it has none). */
 std::string resolveCheckoutHeadRev(const std::string & checkoutPath);

src/libexpr/primops.cc

@@ -5370,7 +5370,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
     Value v;
 
     /* `builtins' must be first! */
-    v.mkAttrs(buildBindings(140).finish());
+    v.mkAttrs(buildBindings(baseEnvCapacity).finish());
     addConstant(
         "builtins",
         v,