CAS: ref-log owner Move for part renames (format generation 2)

ci/jobs/scripts/clickhouse_proc.py

@@ -321,9 +321,14 @@ def start_rustfs(self):
         ):
             print("Failed to start rustfs")
             return False
-        # The `test` bucket the storage policy expects.
+        # The `test` bucket the storage policy expects. The stateless-test image ships the client at
+        # /opt/aistor/mc or /mc.
+        mc = next((p for p in ("/opt/aistor/mc", "/mc") if Path(p).is_file()), None)
+        if mc is None:
+            print("minio client not found at /opt/aistor/mc or /mc")
+            return False
         res = Shell.check(
-            "/mc alias set carustfs http://localhost:11121 clickhouse clickhouse && /mc mb --ignore-existing carustfs/test",
+            f"{mc} alias set carustfs http://localhost:11121 clickhouse clickhouse && {mc} mb --ignore-existing carustfs/test",
             verbose=True,
         )
         if not res:

docs/en/antalya/cas/architecture/manifests-and-refs.md

@@ -195,9 +195,10 @@ writer_epoch)` they run `1, 2, 3, …` with no holes, and a new mount epoch rest
 as `CORRUPTED_DATA`.
 
 The op vocabulary is deliberately tiny: `NamespaceBirth`, `OwnerTransition{old?, new?}`,
-`SetPublishedAt`, `RemoveNamespace`. There are exactly four legal `OwnerTransition` shapes — add
-precommit, remove precommit, remove committed, and promote — enumerated identically by the state
-machine and by `GC`'s edge extractor, so the two readers of the format cannot drift.
+`SetPublishedAt`, `RemoveNamespace`. There are exactly five legal `OwnerTransition` shapes — add
+precommit, remove precommit, remove committed, promote, and move (a committed ref changes its name,
+keeping its manifest; format generation 2) — classified once (`classifyOwnerTransitionShape`) for the
+state machine and for `GC`'s edge extractor, so the two readers of the format cannot drift.
 
 Logs are pure conditional creates on write-once keys. There is no append-to-object and no
 `CAS`-swapped mutable pointer anywhere in the ref lane. The writer never deletes ref objects; only

docs/en/antalya/cas/architecture/part-lifecycle.md

@@ -65,7 +65,9 @@ inline candidate. Blob-class files stage to local scratch by default. Explicit
 `cas_staging_backend = s3` requires native same-store copy at writable mount and stages a complete
 `[header][payload]` object. Its first publication after a destination miss may copy that object
 verbatim; a condemned or subsequent publication opens the staged payload, retags it, and streams.
-The `tmp_ → final` rename is a pure overlay re-key; durable publication happens only in `commit`.
+A rename staged before the commit (`tmp_ → final` inside one transaction) is a pure overlay re-key.
+Writers commit the part right after `finalize`, under its temporary name; the rename that follows is
+one ref-log `Move` of the committed ref (see [replication](replication.md)).
 
 **Step 1 — `stageManifest`.** Caps (see the
 [manifests-and-refs page](/antalya/cas/architecture/manifests-and-refs#part-manifests)) are

docs/en/antalya/cas/architecture/replication.md

@@ -142,13 +142,16 @@ check therefore attributes it to exactly that server root, under the same strict
 content, and that root can confirm its exact refs.
 
 `DETACH`, `ATTACH`, and merge-result renames all reduce to the same two
-moves: re-key any *staged* source into the destination, then `republishRef(src → dst)` for any
-*committed* source. `republishRef` resolves the source ref freshly and reads its manifest through
-the manifest cache, publishes an
-equivalent-entry manifest under the destination ref — a **new** manifest id, with blobs untouched
-and adopted by evidence — then drops the source ref. A destination that already exists with
-identical entries just drops the source, an idempotent re-drive; one with different entries
-throws.
+moves: re-key any *staged* source into the destination, then rename any *committed* source. Inside
+one table namespace the rename is one ref-log `Move` record (`moveRefExact`): the committed ref
+changes its name and keeps its manifest, conditional on the manifest the caller resolved — any other
+current binding is `CAS_OWNERSHIP_CONFLICT`, and a retried rename that finds the move already done is
+a no-op. Across namespaces (`MOVE PARTITION TO TABLE`), or when the destination already exists from an
+earlier attempt, `republishRef` publishes an equivalent-entry
+manifest under the destination ref — a **new** manifest id, with blobs untouched and adopted by
+evidence — then drops the source ref while it still names the republished manifest. A destination
+that already exists with identical entries just drops the source, an idempotent re-drive; one with
+different entries throws.
 
 Manifests are therefore per-ref and never moved: a detach creates a new manifest for
 `detached/<part>` and retires the old one, and the blobs' net in-degree is unchanged.

docs/en/antalya/cas/configuration.md

@@ -253,6 +253,14 @@ requiring a fresh pool.
 | `xxh3-128` | Maximum speed | Fastest, 128-bit, no known collision classes |
 | `cityhash128` (default) | ClickHouse-ecosystem compatibility, and a possible future hash-reuse mode that avoids recomputation | Fast, but has a known class of collisions that occurs far more often than an ideal hash function would predict |
 
+## Upgrading a pool from 26.6.x {#upgrading-from-26-6}
+
+A 26.8 server renames a committed part with one ref-log `Move` record (`cas_log` event `ref_move`); a 26.6.x server cannot read such a record. A pool created by a writable 26.8 mount requires reader generation 2 from birth. The pool records the minimum reader generation it requires (`min_reader_generation`). The first writable mount by a 26.8 server raises it from 1 to 2 and logs a WARNING (`min_reader_generation raised 1 -> 2`); the raise is one-way and happens before any part is renamed. A mount with `readonly` set never raises it. That first writable start is the point of no return for 26.6.x mounts of the pool.
+
+Upgrade order: stop every 26.6.x server that mounts the pool, upgrade all of them to 26.8, then start them. There is no per-server switch: a 26.8 server always raises the floor on its first writable mount. With GC enabled on the 26.6.x servers, follow [Upgrading a pool with GC enabled](operations/debugging.md#upgrade-gc) first. A 26.6.x server that stays up after the raise cannot re-mount the pool on its own: after a lost mount lease or fence it stays unmounted until it is upgraded.
+
+What a 26.6.x server does if it meets a generation-2 pool: it refuses to mount after a restart (`UNKNOWN_FORMAT_VERSION` at startup: `CAS pool meta: pool requires reader generation 2 but this build supports at most 1`). An already mounted 26.6.x server keeps serving namespaces that have no `ref_move` record; direct recovery of a namespace that has one cannot cross the record until a later generation-1 snapshot gives it a new base; its GC round holds on the undecodable log and performs no deletions. Fetch-by-relink is protected by namespace ownership and the mount fence, not by the log version. No data is deleted or lost. Downgrading a pool whose floor is 2 is not supported.
+
 ## Server-level settings {#server-settings}
 
 Source: `ServerSettings.cpp`. This setting is process-wide rather than scoped to one disk block.

docs/en/antalya/cas/operations/debugging.md

@@ -44,9 +44,33 @@ ORDER BY event_time_microseconds;
 `skipped`) and `reason` are the two columns to read first; `detail` is a
 `Map(LowCardinality(String), String)` of decision-specific facts (`condemn_round`,
 `superseded_token`, `code`, `site`) worth `arrayJoin(detail)` when the summary columns alone do not
-explain the decision. See [`system.cas_log`](/operations/system-tables/cas_log) for the full column
+explain the decision. A `ref_move` row (a committed part renamed in place) is keyed by the old part name
+and carries the new one in `detail['dst_ref']`, so a search by the final name must also match `dst_ref`. See [`system.cas_log`](/operations/system-tables/cas_log) for the full column
 reference.
 
+### A query failed with CAS_OWNERSHIP_CONFLICT {#ownership-conflict}
+
+Error code 1039. The committed reference a writer had read was moved, dropped or repointed by another
+writer before the writer's own change landed: a rename found the destination name already committed
+with other content, the source renamed away, or a live precommit under the destination; a standalone
+write into a committed part found the part renamed or dropped meanwhile. The source part is untouched;
+a destination reference the failed cross-table rename itself created is taken back.
+
+The writer must re-resolve the part before trying again. Background work (replication queue entries,
+merges, mutations) retries on its own and records the message in `system.replication_queue.last_exception`;
+a user command (`ATTACH PART`, `MOVE PARTITION TO TABLE`, `ALTER ... MATERIALIZE`) reports the error
+and is rerun by the user. A conflict that repeats on every retry names two parts that really collide
+under one name — look at both sides:
+
+```sql
+SELECT event_time_microseconds, event_type, ref_name, detail['dst_ref'] AS dst_ref, reason
+FROM system.cas_log
+WHERE disk_name = 'cas' AND (ref_name IN ('<src_part>', '<dst_part>') OR detail['dst_ref'] IN ('<src_part>', '<dst_part>'))
+ORDER BY event_time_microseconds;
+```
+
+`ProfileEvents.CASOwnershipConflict` counts every refusal.
+
 ### Why GC is not reclaiming {#why-not-reclaiming}
 
 Two questions, in order: is this node's scheduler leading, and did its recent rounds actually fold?
@@ -251,6 +275,12 @@ clickhouse-disks -C config.xml --disk cas cas-inspect '<raw-object-key>'
 clickhouse-disks -C config.xml --disk cas cas-gc-rebuild --force
 ```
 
+Exit code of a non-interactive `--query`: 1 when any `cas-*` command in it threw (`cas-fsck` throws on
+`dangling`, `chain_broken`, `corrupted_runs` and `lifeless_keys`; `unreachable` is a soft count that
+includes the GC pipeline and in-flight builds), and so does a `cas-*` command given bad arguments,
+whatever ran after it; the upstream commands keep `clickhouse-disks`'s "always 0" contract, as does an
+unknown command name.
+
 `cas-drop-member` — the offline twin of `SYSTEM CAS DROP POOL MEMBER` — is covered on the
 [migration page](/antalya/cas/operations/migration#decommission) alongside the SQL form, since
 decommissioning a pool member is a migration/scale-down operation, not an incident-time tool.
@@ -266,7 +296,12 @@ decommissioning a pool member is a migration/scale-down operation, not an incide
 
 A delete request sent before a restart can still reach the object store later and remove a marker GC relies on,
 so a blob can be lost; nothing bounds when such requests stop. Entries the old version already scheduled for
-deletion are deleted by the upgraded GC without a fresh marker read. Downgrades follow the same procedure.
+deletion are deleted by the upgraded GC without a fresh marker read. A downgrade within 26.6.x follows the same
+procedure; a pool whose reader floor was raised to 2 by a writable 26.8 mount cannot be downgraded (see
+[Upgrading a pool from 26.6.x](../configuration.md#upgrading-from-26-6)).
+
+The decommission tool opens the pool with the server's CAS disk configuration and never admits a new
+`cas_blob_hash` into the pool: run it with the algorithm the pool already records.
 
 ## The CLICKHOUSE_USER_FILES gotcha when reproducing a test manually {#user-files-gotcha}
 

docs/en/antalya/metrics_and_events.md

@@ -252,6 +252,8 @@ These counters cover ref-log appends, recovery, snapshots, and reachability edge
 | `CASRefRecoveryStragglerAdopted` | `Number` | Number of straggler ref-log transactions a recovery compare-and-swap walk met at the slot it tried to seal and adopted, re-sealing at the new T+1. Non-zero means writes from a dying epoch were still materializing when recovery ran. |
 | `CASRefRecoveryStreamHole` | `Number` | Number of times CAS ref-table recovery found a 404 BELOW a durable same-epoch witness -- a hole in a stream INV-1 makes dense. Restarted while the restart budget lasts (a racing cleanup is the innocent explanation), then reported as corruption. Any sustained non-zero value is data loss, not noise. |
 | `CASRefRepoint` | `Number` | Number of CAS committed-reference repoints that republish a manifest. Growth indicates standalone part updates or removals. |
+| `CASRefMove` | `Number` | Number of CAS committed-reference renames done as one ref-log owner move, with no manifest republication. |
+| `CASOwnershipConflict` | `Number` | Number of CAS operations refused with `CAS_OWNERSHIP_CONFLICT`: the committed reference the writer had read was moved, dropped or repointed before its change landed. |
 | `CASRefRollbackBestEffortDropFailed` | `Number` | Number of rollback cleanup drops that hit a backend failure. A non-zero value means refs may remain live and GC may be delayed. |
 | `CASRefSnapshotPublishBackoff` | `Number` | Number of CAS ref-table snapshot publications put into backoff after a non-committed result. A non-zero value indicates write failure or uncertainty. |
 | `CASRefSnapshotPublishDispatched` | `Number` | Number of background CAS ref-table snapshot publications started. High values indicate frequent threshold or read-triggered publishing. |
@@ -327,7 +329,6 @@ These counters cover conditional writes made without transparent SDK retries.
 | `CASConditionalWriteAttempts` | `Number` | Number of CAS conditional-write HTTP attempts made without transparent SDK retries. Growth indicates CAS metadata or reference updates. |
 | `CASConditionalWriteCommitted` | `Number` | Number of CAS conditional writes completed successfully. Growth indicates successful conditional-update activity. |
 | `CASConditionalWriteDefiniteFailure` | `Number` | Number of CAS conditional writes rejected with certainty before applying. A non-zero value indicates invalid requests, oversized entities, or access denial. |
-| `CASConditionalWriteFenceLostPostWrite` | `Number` | Number of CAS writes that succeeded but lost the final mount-fence check. A non-zero value indicates late responses after the mount lifecycle changed. |
 | `CASConditionalWriteUnresolved` | `Number` | Number of CAS conditional writes with an unknown outcome after conflict, timeout, connection loss, or server error. A non-zero value indicates backend instability or state requiring resolution. |
 
 #### Hot-key lane {#cas-hot-key}

programs/disks/CommandCaDropMember.cpp

@@ -24,6 +24,7 @@ class CommandCaDropMember final : public ICommand
     CommandCaDropMember() : ICommand("CommandCaDropMember")
     {
         command_name = "cas-drop-member";
+        exits_nonzero_on_error = true;
         description = "Decommission a DEAD pool member: erase its namespaces, debris, staging, roots "
                       "objects and mount slot. Refuses a live member. Open the CA disk read-only "
                       "(the admin claim is made internally).";

programs/disks/CommandCaGcDryRun.cpp

@@ -20,6 +20,7 @@ class CommandCaGcDryRun final : public ICommand
     CommandCaGcDryRun() : ICommand("CommandCaGcDryRun")
     {
         command_name = "cas-gc-dryrun";
+        exits_nonzero_on_error = true;
         description = "Preview the next GC round's deletes for a content-addressed pool (read-only, no deletes).";
     }
 

programs/disks/CommandCaGcRebuild.cpp

@@ -32,6 +32,7 @@ class CommandCaGcRebuild final : public ICommand
     CommandCaGcRebuild() : ICommand("CommandCaGcRebuild")
     {
         command_name = "cas-gc-rebuild";
+        exits_nonzero_on_error = true;
         description = "Disaster recovery: rebuild a content-addressed pool's gc/state baseline from raw owner "
                       "state after the GC guard has refused every round (see CORRUPTED_DATA in the gc log). "
                       "Requires a read-only-opened disk; never run against a disk a live server has mounted.";

programs/disks/CommandCaInspect.cpp

@@ -26,6 +26,7 @@ class CommandCaInspect final : public ICommand
     CommandCaInspect() : ICommand("CommandCaInspect")
     {
         command_name = "cas-inspect";
+        exits_nonzero_on_error = true;
         description = "Decode a content-addressed pool object (by its raw object-storage key) to JSON (read-only).";
         options_description.add_options()("key", po::value<String>(), "the raw object-storage key to decode (mandatory, positional)");
         positional_options_description.add("key", 1);

programs/disks/CommandFsck.cpp

@@ -22,8 +22,10 @@ class CommandFsck final : public ICommand
     CommandFsck() : ICommand("CommandFsck")
     {
         command_name = "cas-fsck";
+        exits_nonzero_on_error = true;
         description = "Independently verify content-addressed pool reachability (read-only). "
-                      "Exits nonzero if any reachable object is missing (dangling).";
+                      "Exits 1 on a hard finding: a reachable object missing (dangling), a hole in a ref-log "
+                      "stream (chain_broken), a corrupted GC run, or a malformed key (lifeless_keys).";
         options_description.add_options()("detail", "list per-object rows (class, key, size, reachable_from)")(
             "timeout", po::value<UInt64>(), "abort the scan after N seconds with a clear error instead of hanging (default 600; 0 = unbounded)")(
             "namespace", po::value<String>(), "scope the scan to namespaces with this prefix (skips the pool-wide "
@@ -177,6 +179,9 @@ class CommandFsck final : public ICommand
                 "cas-fsck: {} key(s) under this pool name are malformed or unresolvable — no current "
                 "writer could have produced them, or their catalog incarnation is ambiguous/unreadable "
                 "(the keys are listed as `lifeless-key` rows under --detail)", report.lifeless_keys);
+        /// `unreachable` stays a soft finding: it counts every present blob no ref names, and that includes
+        /// the GC pipeline's `pending_gc`/`awaiting_gc` and an in-flight build's not-yet-adopted blobs
+        /// (`unaccounted`), so a healthy pool under load is never clean by that measure.
     }
 };
 

programs/disks/DisksApp.cpp

@@ -221,6 +221,7 @@ bool DisksApp::processQueryText(const String & text)
     for (const auto & subquery : subqueries)
     {
         std::optional<String> error_string;
+        command.reset();
         // std::cerr << "SUBQUERY: " << subquery << "\n";
         try
         {
@@ -235,8 +236,8 @@ bool DisksApp::processQueryText(const String & text)
         {
             int code = err.code();
             error_string = getExceptionMessageForLogging(err, true, false);
-            if (command && command->command_name.starts_with("cas-"))
-                last_command_exit_code = code;
+            if (command && command->exits_nonzero_on_error)
+                last_command_exit_code = 1;
             if (code == ErrorCodes::BAD_ARGUMENTS)
             {
                 if (command.get())
@@ -253,14 +254,14 @@ bool DisksApp::processQueryText(const String & text)
         catch (std::exception & err)
         {
             error_string = err.what();
-            if (command && command->command_name.starts_with("cas-"))
-                last_command_exit_code = ErrorCodes::STD_EXCEPTION;
+            if (command && command->exits_nonzero_on_error)
+                last_command_exit_code = 1;
         }
         catch (...) // Ok: report unknown exception
         {
             error_string = "Unknown exception";
-            if (command && command->command_name.starts_with("cas-"))
-                last_command_exit_code = ErrorCodes::STD_EXCEPTION;
+            if (command && command->exits_nonzero_on_error)
+                last_command_exit_code = 1;
         }
         if (error_string.has_value())
         {
@@ -624,8 +625,8 @@ int DisksApp::main(const std::vector<String> & /*args*/)
     if (log_file)
         log_file->close();
 
-    /// Non-interactive runs surface a failing command as a nonzero process exit (CI/cron gating,
-    /// e.g. `cas-fsck` reporting dangling objects). Interactive sessions are unaffected.
+    /// A non-interactive run exits 1 when a `cas-*` command threw (`ICommand::exits_nonzero_on_error`),
+    /// whatever ran after it; every other command keeps the upstream "always 0" contract.
     if (query.has_value() && last_command_exit_code != 0)
         return last_command_exit_code;
     return Application::EXIT_OK;