CAS avoid manifest get on promote

docs/en/antalya/cas/architecture/manifests-and-refs.md

@@ -55,6 +55,20 @@ before any owner transition names it. Rewriting a part therefore writes a **new*
 the **same** blobs and moves the ref in one ref-log record — a repoint, covered in full on the
 [part-lifecycle page](/antalya/cas/architecture/part-lifecycle#repoint).
 
+The writer retains a compact record after its own successful `stageManifest`: the confirmed object
+incarnation and the blob dependencies in the order the caller passed the entries. The encoded body
+sorts entries by path, so this order can differ from a decoded body. Only the order of dependency
+checks and `BlobReuseAdopt` events depends on it. The matching `promote` uses this record without
+reading the immutable body back. Promotion of a manifest not staged by that `PartWriteTxn` still
+performs an exact `GET` and validates `RefMatchesBody` and `ManifestNamespaceMatches` fail-closed.
+
+The record only shows that the body existed when the conditional create was confirmed. Inside the
+protocol nothing deletes it before `promote`: the orphan sweep does not touch the prefix of a
+build sequence that is still active. A body removed outside the protocol (a bucket lifecycle or
+expiry rule on `cas/manifests/`, a manual delete) is not detected by `promote`, which commits the
+ref anyway. The read path then fails with `FILE_DOESNT_EXIST`, and `fsck` reports the reference
+as dangling. Do not put lifecycle or expiry rules on the pool prefix.
+
 ## Manifest lifecycle and the orphan sweep {#manifest-lifecycle}
 
 ```mermaid

docs/en/antalya/cas/architecture/part-lifecycle.md

@@ -53,7 +53,7 @@ sequenceDiagram
     end
     PW->>PW: merge upload results on the owning thread, one no-throw swap
     TX->>PW: promote
-    PW->>S3: GET and validate the precommit manifest body
+    PW->>PW: check the staged manifest record and blob dependencies
     PW->>S3: append ref-log txn: retire old committed, Precommit to Committed, SetPublishedAt
     Note over PW: commit durable, then retire the build sequence
     end
@@ -93,9 +93,12 @@ explicit `Materialized` proof. A trusted source manifest instead yields `Trusted
 blob I/O. Pool size is the
 server setting `cas_blob_upload_pool_size` (default 16).
 
-**Step 4 — `promote`.** Reads and revalidates the precommit manifest body once; sets the commit
-state to Uncertain before the append — past that point, failure is no longer proof of the negative
-— then checks that the precommit is still the live owner and validates explicit dependency proofs.
+**Step 4 — `promote`.** A manifest staged by the same `PartWriteTxn` uses the in-memory proof produced
+by the successful conditional create: the confirmed object incarnation and ordered blob dependencies.
+There is no manifest `GET` on this normal path. A manifest not staged by this transaction is read and
+validated fail-closed. Promotion then sets the commit state to Uncertain before the append — past that
+point, failure is no longer proof of the negative — checks that the precommit is still the live owner,
+and validates explicit dependency proofs.
 `Materialized` leaves are edge-protected; `TrustedManifest` leaves are trusted through the durable
 source-manifest edge with no per-file `HEAD`; anything else is a `LOGICAL_ERROR`. The
 whole thing lands as one ref-log record: optional retirement of the old committed binding, the pure

docs/en/antalya/cas/architecture/replication.md

@@ -69,7 +69,7 @@ sequenceDiagram
 | 4 | The receiver trusts nothing from the wire but the entry list | The sender's manifest id, namespace and payload digest are ignored; the target namespace and ref come from the receiver's own router, and manifest path hygiene is validated at decode |
 | 5 | The confirm is I/O-free and fail-closed | A cold, evicted, unfenced or terminal mount answers `Unknown`. `No` and `Unknown` both go on the wire as `unproven`, because the fence check is evaluated last, so a `No` cannot be distinguished from "cannot prove it right now" |
 | 6 | Only the literal `yes` authorizes promotion | Everything else — including a timeout — is one outcome: throw and retry later |
-| 7 | Promote outcomes are three-way | `Committed` proceeds; a **proven** not-committed state (body-absent precommit, precommit no longer live owner, ref conflict) falls back to a byte fetch; `Unresolved` **throws**, because returning "fall back" there would publish the part twice |
+| 7 | Promote outcomes are three-way | `Committed` proceeds; a **proven** not-committed state (precommit no longer live owner, ref conflict) falls back to a byte fetch; `Unresolved` **throws**, because returning "fall back" there would publish the part twice. The receiver stages its manifest in the same build, so `promote` does not read the body back |
 
 The byte-fetch fallback is bounded: it re-invokes the fetch with relink disabled, which stops the
 receiver advertising its pool uuid, which stops the sender offering relink — so the relink path
@@ -102,7 +102,7 @@ The part is never quietly placed on another disk instead. This is the behaviour
 content-addressed policy always had, and a mixed policy now shares it.
 
 The byte-fetch fallback after a relink that failed for a mechanism reason (a corrupted manifest, a
-body-absent precommit, a ref conflict) re-requests the bytes on the same pool disk, where they
+ref conflict) re-requests the bytes on the same pool disk, where they
 content-address and deduplicate against the pool — the placement outlives the relink. A manifest of a
 newer format generation is not degraded to bytes today (a tracked gap, `[relink-fallback-unknown-format-version]`
 in the backlog).

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp

@@ -75,6 +75,20 @@ String bannerFor(std::string_view path, uint64_t n)
     return std::move(w).take();
 }
 
+void validateManifestEntryPath(std::string_view path)
+{
+    if (path.empty() || path.front() == '/')
+        throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS part manifest: invalid entry path '{}'", path);
+    for (std::string_view rest = path; !rest.empty();)
+    {
+        const size_t slash = rest.find('/');
+        const std::string_view segment = rest.substr(0, slash);
+        if (segment.empty() || segment == "." || segment == "..")
+            throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS part manifest: invalid entry path '{}'", path);
+        rest = slash == std::string_view::npos ? std::string_view{} : rest.substr(slash + 1);
+    }
+}
+
 }
 
 std::string_view entryPlacementToWireWord(EntryPlacement placement)
@@ -94,7 +108,10 @@ String encodePartManifest(const PartManifest & m)
     std::vector<const ManifestEntry *> sorted;
     sorted.reserve(m.entries.size());
     for (const auto & e : m.entries)
+    {
+        validateManifestEntryPath(e.path);
         sorted.push_back(&e);
+    }
     std::sort(sorted.begin(), sorted.end(),
               [](const ManifestEntry * a, const ManifestEntry * b) { return a->path < b->path; });
     for (size_t i = 1; i < sorted.size(); ++i)
@@ -208,20 +225,7 @@ PartManifest decodePartManifest(std::string_view data)
             throw Exception(ErrorCodes::CORRUPTED_DATA, "PartManifest: record must start with \"path\"");
         ManifestEntry e;
         e.path = r.readString();
-
-        /// Manifest bytes arrive over the interserver relink channel: enforce the same path hygiene
-        /// as CasLayout::checkNamespace so no future consumer can inherit a traversal. Relative,
-        /// no empty/'.'/'..' segments. (Syntactic only — legal projection subdirs pass.)
-        if (e.path.empty() || e.path.front() == '/')
-            throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS part manifest: invalid entry path '{}'", e.path);
-        for (std::string_view rest = e.path; !rest.empty();)
-        {
-            const size_t slash = rest.find('/');
-            const std::string_view seg = rest.substr(0, slash);
-            if (seg.empty() || seg == "." || seg == "..")
-                throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS part manifest: invalid entry path '{}'", e.path);
-            rest = (slash == std::string_view::npos) ? std::string_view{} : rest.substr(slash + 1);
-        }
+        validateManifestEntryPath(e.path);
 
         std::optional<String> pm;
         BlobRefFields blob_ref;

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp

@@ -2989,7 +2989,7 @@ Gc::FoldResult Gc::fold(GcState & state, std::optional<Etag> & /*state_etag*/,
                 /// clamps every round with no terminal resolution. Fail-closed: no durable watermark (the
                 /// build cannot be proven dead) keeps the barrier.
                 if (edge.change > 0 && edge.owner_kind == RefOwnerKind::Precommit
-                    && prefixEligible(*store, ns,
+                    && isBuildPrefixCoveredByFloor(*store, ns,
                            BuildPrefix{.writer_epoch = edge.manifest_id.ref.writer_epoch,
                                        .build_sequence = edge.manifest_id.ref.build_sequence}))
                 {
@@ -4846,7 +4846,7 @@ RebuildReport Gc::rebuildBaseline(bool force)
             if (!parsed)
                 return true;   /// foreign key shape — debris
             const ManifestRef & mref = parsed->ref;
-            if (prefixEligible(*store, ns, BuildPrefix{mref.writer_epoch, mref.build_sequence}))
+            if (isBuildPrefixCoveredByFloor(*store, ns, BuildPrefix{mref.writer_epoch, mref.build_sequence}))
                 return true;   /// provably dead — the orphan sweep's territory, never an edge
             const ManifestId id{ns, mref};
             if (foldManifestEdges(reads, id, +1, deltas, mf_cleanup_unused, /*txn_ordinal=*/0))

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.cpp

@@ -45,8 +45,8 @@ void onGcEnumerationPage()
 /// mount there is no deletion authority, so the caller must leave the prefix untouched. The mount's
 /// `writer_epoch` and `min_active_build_sequence` are the single durable epoch/floor pair used for eligibility, including
 /// across process replacement and the retired sentinel.
-std::optional<MountLease> floorForNamespace(CasOperation & op, const Layout & layout, const RootNamespace & ns,
-                                            uint64_t * reads = nullptr)
+std::optional<MountLease> mountLeaseForNamespace(CasOperation & op, const Layout & layout, const RootNamespace & ns,
+                                                 uint64_t * reads = nullptr)
 {
     const String & value = ns.string();
     size_t pos = value.size();
@@ -529,40 +529,40 @@ namespace
 /// do not replace that authority check with a frozen-sequence or judged-dead guess. Compare
 /// `writer_epoch` first, then `build_sequence`, so old-epoch
 /// debris drains after a process restart even when its build_sequence is above the current min_active_build_sequence.
-bool prefixEligibleOn(CasOperation & op, const Layout & layout, const RootNamespace & ns, const BuildPrefix & prefix)
+bool isBuildPrefixCoveredByFloorOn(
+    CasOperation & op, const Layout & layout, const RootNamespace & ns, const BuildPrefix & prefix)
 {
-    return prefixEligibleUnder(floorForNamespace(op, layout, ns), prefix);
+    return isBuildPrefixCoveredByFloor(mountLeaseForNamespace(op, layout, ns), prefix);
 }
 
 }
 
-bool prefixEligibleUnder(const std::optional<MountLease> & floor, const BuildPrefix & prefix)
+bool isBuildPrefixCoveredByFloor(const std::optional<MountLease> & floor, const BuildPrefix & prefix)
 {
     if (!floor)
         return false;
 
-    const MountLease & w = *floor;
-    if (prefix.writer_epoch < w.writer_epoch)
+    if (floor->writer_epoch > prefix.writer_epoch)
         return true;
-    if (prefix.writer_epoch > w.writer_epoch)
+    if (floor->writer_epoch < prefix.writer_epoch)
         return false;
-    if (w.min_active_build_sequence == std::numeric_limits<uint64_t>::max())
+    if (floor->min_active_build_sequence == std::numeric_limits<uint64_t>::max())
         return true;   /// farewell/retired sentinel: every seq is retired
-    return w.min_active_build_sequence > prefix.build_sequence;
+    return floor->min_active_build_sequence > prefix.build_sequence;
 }
 
-bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix)
+bool isBuildPrefixCoveredByFloor(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix)
 {
     CasOperation op = store.openRequests().admit();
-    return prefixEligibleOn(op, store.layout(), ns, prefix);
+    return isBuildPrefixCoveredByFloorOn(op, store.layout(), ns, prefix);
 }
 
 uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix,
                         std::vector<String> * warnings)
 {
     const Layout & layout = store.layout();
     CasOperation op = store.openRequests().admit();
-    if (!prefixEligibleOn(op, layout, ns, prefix))
+    if (!isBuildPrefixCoveredByFloorOn(op, layout, ns, prefix))
         return 0;   /// not eligible by the durable watermark fact — delete nothing (controls #8/#9)
 
     /// Build the protection view. A missing snapshot body, an invalid transaction, or an incomplete
@@ -722,7 +722,6 @@ ManifestSweepResult planManifestCursorPage(
     /// catalog cut: a manifest key is write-once, so its bytes do not depend on when they are read,
     /// and the only thing a later read can observe differently is absence, which retains.
     std::vector<ListedManifestObject> candidates;
-
     for (const ListedKey & listed : page.keys)
     {
         ++result.listed;
@@ -758,23 +757,21 @@ ManifestSweepResult planManifestCursorPage(
             continue;
         }
 
-        const CatalogEntry * catalog_entry = catalogEntryOf(catalog_cut, parsed->ns);
-        if (catalog_entry)
+        auto [floor_it, floor_inserted] = floor_by_ns.emplace(parsed->ns.string(), std::nullopt);
+        if (floor_inserted)
         {
-            auto [floor_it, floor_inserted] = floor_by_ns.emplace(parsed->ns.string(), std::nullopt);
-            if (floor_inserted)
-            {
-                ++result.floor_lookups;
-                floor_it->second = floorForNamespace(op, layout, parsed->ns, &result.floor_reads);
-            }
-            if (!prefixEligibleUnder(floor_it->second, parsed->prefix))
-            {
-                ++result.skipped;
-                decided_through = listed.key;
-                continue;
-            }
+            ++result.floor_lookups;
+            floor_it->second = mountLeaseForNamespace(op, layout, parsed->ns, &result.floor_reads);
+        }
+        if (!isBuildPrefixCoveredByFloor(floor_it->second, parsed->prefix))
+        {
+            ++result.skipped;
+            decided_through = listed.key;
+            continue;
         }
 
+        const CatalogEntry * catalog_entry = catalogEntryOf(catalog_cut, parsed->ns);
+
         auto [view_it, view_inserted] = view_by_ns.emplace(parsed->ns.string(), NamespaceFoldView{});
         auto [active_it, inserted] = active_by_ns.emplace(parsed->ns.string(), std::set<String>{});
         if (inserted)
@@ -882,10 +879,6 @@ ManifestSweepResult planManifestCursorPage(
             continue;
         }
 
-        /// If the post-observation catalog cut has no row, this candidate was necessarily created by a
-        /// now-dead life: creation publishes its row before any life-owned object. No current life can
-        /// name it, and a concurrent later creation can only replace its token after this observation.
-        /// A catalog-named row instead needs the ordinary coverage/owner proof below.
         if (catalog_entry)
         {
             /// THE §6 SAFETY FLOOR, the same predicate `sweepNamespace` calls, under the same watermark
@@ -919,6 +912,21 @@ ManifestSweepResult planManifestCursorPage(
         decided_through = listed.key;
     }
 
+    if (!candidates.empty())
+    {
+        const CasRefCatalog::Snapshot post_floor_catalog = CasRefCatalog::read(op, layout);
+        post_floor_catalog.life_index.throwIfAmbiguous("CAS orphan manifest sweep");
+        std::erase_if(candidates, [&](const ListedManifestObject & candidate)
+        {
+            const CatalogEntry * before = catalogEntryOf(catalog_cut, candidate.ns);
+            const CatalogEntry * after = catalogEntryOf(post_floor_catalog, candidate.ns);
+            if ((!before && !after) || (before && after && *before == *after))
+                return false;
+            ++result.skipped;
+            return true;
+        });
+    }
+
     size_t next_body_hint = 0;
     for (const ListedManifestObject & candidate : candidates)
     {

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasOrphanManifestSweep.h

@@ -190,17 +190,17 @@ struct ManifestSweepResult
 uint64_t sweepNamespace(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix,
                         std::vector<String> * warnings = nullptr);
 
-/// Whether `prefix` is sweep-eligible by the durable watermark fact alone. The floor is read from the
-/// mount lease identified by the namespace's server-root prefix, not inferred from the manifest key or a
-/// judged-dead heuristic. A missing lease provides no deletion authority, so the prefix is not eligible.
-bool prefixEligible(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix);
-
-/// The pure half of `prefixEligible`: whether `prefix` is retired under one observation of the mount
-/// floor. `nullopt` (no mount body under any prefix of the namespace) admits nothing. Retirement is
-/// permanent -- the epoch and the acknowledgement floor only grow and the farewell is terminal -- so
-/// an admission derived from any observation stays true afterwards, which is what lets a page judge
+/// Whether `prefix` is covered by the durable watermark fact alone. The floor is read from the mount
+/// lease identified by the namespace's server-root prefix, not inferred from the manifest key or a
+/// judged-dead heuristic. A missing lease provides no deletion authority, so the prefix is not covered.
+bool isBuildPrefixCoveredByFloor(Pool & store, const RootNamespace & ns, const BuildPrefix & prefix);
+
+/// The pure overload: whether `prefix` is covered by one observation of the mount floor. `nullopt`
+/// (no mount body under any prefix of the namespace) covers nothing. Coverage is permanent -- the epoch
+/// and the acknowledgement floor only grow and the farewell is terminal -- so a result derived from
+/// any observation stays true afterwards, which is what lets a page judge
 /// every build of a namespace against one read.
-bool prefixEligibleUnder(const std::optional<MountLease> & floor, const BuildPrefix & prefix);
+bool isBuildPrefixCoveredByFloor(const std::optional<MountLease> & floor, const BuildPrefix & prefix);
 
 /// Plan one cursor page without deleting. A key is decided from key-derived facts first; only a
 /// candidate's body is read, after the catalog cut, then decoded and identity-validated; its exact