[filimonov]

SYSTEM CAS DROP POOL MEMBER no longer deletes a member's owner, epoch and mount objects; it deletes only the member's data. Deleting those objects let a restarting victim be issued a writer epoch it already used and let a late tombstone land on a live successor. Keeping them makes epochs strictly increasing per server_root_id across lives and removes the tail's race handling.

Changes

• Writer-epoch counter nonce. Every counter write carries a fresh allocation_nonce. Two processes with the same server_uuid that race on an ambiguous counter write no longer resolve to the same epoch: their bodies differ, so the byte-equality resolve cannot adopt the other's write.

• Owner life. The owner object gains owner_life, a counter every owner write bumps by If-Match, and retirement_verified. A mount records the life it started under. After every slot claim (start and self-remount) it re-reads the owner and requires its own uuid, that life, and no tombstone; otherwise it fences its own slot body and fails. A self-remount compares the life before it allocates an epoch: an owner at another life ends the remount loop in IdentityLost, and a restart picks up the current life.

• Decommission run:

  1. claim the victim's slot;
  2. tombstone the owner (new life) on the admin's fenced mount plane, so no attempt is sent after the admin's lease deadline;
  3. confirm the claim still holds, otherwise roll the tombstone back on the unfenced control plane, unless the owner was already retired;
  4. drain the data on the admin's fenced mount plane, so a fenced admin stops;
  5. write the farewell;
  6. verify the slot and the owner and mark the retirement verified.

  Every exit between steps 2 and 5 runs the confirm. When the tombstone's outcome is unknown and the owner does not read back as the tombstone, the run fails without its clean farewell (Pool::withholdFarewell): the slot keeps the admin's live body, so the victim restarts only after GC fences it (about 1.05 × the lease TTL plus a GC round) instead of reclaiming at once. member_retired = 1 only for a verified mark. An owner that moved on to a later unretired life (a rejoin, or another run's rollback) reports member_retired = 0 with its own warning. A run that cannot tell whether its tombstone or verified mark landed re-reads the owner before advising a rerun.

• S3 staging keys carry the writer epoch (staging/<srid>/<epoch>/<random>.tmp), built and parsed in Layout. The startup sweep deletes only earlier epochs, so a paused older process can no longer delete its successor's staging. The epoch and the admission generation come from one capture.

• Removed: the epoch re-mint policy for decommission (EpochMintPolicy). A member whose counter is missing (left by an older decommission) is refused, not repaired.

• A mount that skips the capability probe (skip_access_check, always the case for decommission) logs it.

• Lease renewer fix, all CAS mounts. The stop flag was read before the renewal's admission. A stop landing in between made the renewal terminal with nothing sent, so a clean shutdown skipped its farewell and the next start paid the unclean-reclaim wait.

Risks / notes

• Result columns change. slot_removed is renamed member_retired at the same position, and namespaces_pending_gc is appended after warnings. Rows still Removing after the drain are reported in namespaces_pending_gc; GC deletes them and no rerun is needed.
• A retired server_root_id refuses to mount, as before. The refusal names the owner object, its retirement and its life; the troubleshooting table gives the owner rewrite a restart accepts. The alternative is a new server_root_id. Rejoining a retired id without a hand edit is a separate change.
• Format: allocation_nonce, owner_life and retirement_verified are optional plain keys; 26.6.4 readers skip them. An owner without owner_life reads as life 0, and a body without the new fields keeps its exact bytes.
• Mixed versions: a 26.6.4 decommission tool still deletes the control objects, and a 26.6.4 server has no owner recheck; both keep their old races. Use this version's tool for decommission. Downgrade: 26.6.4 refuses a retired member as before and treats the kept farewell as terminal.
• Known shutdown race, not fixed here. A stop that lands after a renewal write was sent still ends the renewer and skips the farewell. This is fail-safe: a decommission reports member_retired = 0 with "the slot does not hold this run's farewell" and asks for a rerun, which waits until GC fences the slot (about one mount lease TTL plus a GC round). test_cas_drop_pool_member accepts exactly this outcome; CASDecommission.LongDrainOutlivesOneTtl places the stop between two renewals and requires the farewell. Follow-up: shutdown waits for an in-flight renewal.
• Late tombstone. The admin's fenced plane bounds when the tombstone is sent, not when the store applies it. Withholding the farewell makes the victim wait for a GC fence; a tombstone applied later than that is a timing assumption, like an in-flight drain request.
• A remount over a retired owner still retries (reads only, no epoch, no slot write) until the server restarts and refuses.

Testing

• New gtests fail on the base for the stated reason, among them:
  • twin allocators sharing an epoch, and a resend keeping its nonce;
  • a late tombstone after a same-uuid rejoin, failing on the owner life;
  • the claim recheck: absent, foreign, retired or moved owner; self-fence only of its own body; recheck on remount; a remount over a moved life allocates no epoch and stops;
  • decommission: keeps epoch and farewell; a tombstone retry after the lease deadline is not sent; the owner rewritten before the tombstone aborts; a rollback keeps an earlier retirement, verified or not; an owner that moved on is not reported retired; a member written by 26.6.4 rolls back and restarts (pins existing behaviour); victim claiming after the tombstone; fence before the tombstone; rollback on the control plane; a paused run cannot clear a rerun's tombstone; resume after a crash at each step; drain stops on a fence;
  • staging: an older-epoch sweep keeps a successor's staging; capture vs remount; staging key shapes;
  • a slot deleted under the observation loop is re-claimed on the paced, bounded budget; the observation map drops an unlisted slot;
  • the renewer stop race;
  • CASDecommission.UnresolvedTombstoneLeavesNoFarewell: an unresolved tombstone leaves the admin's live body, and the victim reclaims only after a fence.
• Model: CaEpochAllocCore round 12 (docs/superpowers/models/CaEpochAllocCore_RESULTS.md on the CAS design branch): the fenced tombstone keeps NoTombOverLive; without it, and with the wider rollback rule, the sabotages are red.
• CAS* unit tests: release 2720, ASan 2724, all passed. test_cas_drop_pool_member: 2/2.
• Every commit builds and passes release CAS*.

Commits

• 8c0f14fd7fc4 Add an allocation nonce to the CAS writer-epoch counter
• 86b20da89e7f Record the CAS owner life and recheck the owner after a mount claim
• d1f194489a2c Keep the CAS control objects on decommission and fence the tombstone
• 133ee5786ae3 Put the writer epoch into CAS S3 staging keys
• e25a8282614e Remove the CAS epoch re-mint policy for decommission
• 93adba90c85d Keep a CAS renewal refused by a mid-renewal stop releasable
• 3a2641dc7032 Document the CAS member lifecycle and decommission

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

SYSTEM CAS DROP POOL MEMBER keeps the member's owner, epoch and mount objects, tombstones the owner before draining and verifies the retirement. Its result column slot_removed is renamed member_retired, and namespaces_pending_gc is appended. Every CAS mount re-reads its owner after claiming its slot. Writer-epoch allocation is unique between same-uuid processes, and S3 staging keys carry the writer epoch. A clean CAS shutdown no longer skips its farewell when the stop races the start of a lease renewal. A decommission whose tombstone outcome is unknown no longer writes its clean farewell, so the member restarts only after GC fences its slot.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [b856f34]

[filimonov]

Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) and Stateless tests (amd_binary, cas s3 storage, parallel): Server died with Logical error: GroupingAggregatedTransform pushed bucket A twice. The stack is in aggregation (MergingAggregatedMemoryEfficientTransform), no CAS frames; the same failure hits other PRs on this base. Not related to this PR. Tracked upstream: ClickHouse#115986

[k-morozov]

I am wandering about retries in remount loop.

Model explanation:
Self-remount over a retired owner retries forever instead of going terminal

Scenario: member r1 loses its lease (pause, partition), and GC fences it. Then SYSTEM CAS DROP POOL MEMBER 'r1' runs on another node and tombstones the owner (retired_at_ms, owner_life + 1). When r1 wakes up, its renewal ends as
terminal and remountLoop calls tryRemountOnce.

In tryRemountOnce, claimOwnerOrThrow (CasPool.cpp:1498, default RetiredOwnerPolicy::Refuse) throws CORRUPTED_DATA on the retired owner. This happens before the life check at :1499. The catch (...) at :1640 returns false.
remountLoop (CasMountRuntime.cpp:887-899) waits (backoff up to 30 s) and calls it again. remount_handled_generation is updated only on success, and remountTerminal() stays false, so the loop never ends until restart.

This retry can never succeed. The tombstone already moved owner_life, and life only grows. So after the tombstone the owner can only be in one of two states:

• still retired: the same throw again;
• un-retired (rollback or hand edit): a life mismatch, which leads to IdentityLost.

Effects:

• the pool stays in TransientNotLive (lifecycle = 'not_live' in system.cas_mounts), the same as a normal short lease loss;
• every 30 s it reads _pool_meta, the catalog and the owner;
• every attempt adds CASRemountFailed and a WARNING;
• CASIdentityLost is not incremented, so there is no terminal signal and no alert.

There is no safety problem, because nothing is written under the tombstone. This is an operability and consistency issue. The PR description says "an owner at another life ends the remount loop in IdentityLost", but a tombstone is
also a new life and does not end the loop. The PR notes the retry as known behavior ("still retries ... until the server restarts"), but does not explain why it is kept.

Suggested fix: on remount, compare owner_life before the retired check, or handle the retired owner the same way as a life mismatch, so that both call enterIdentityLost. Also add a troubleshooting row for this case and a test that
pins the lifecycle state.

[k-morozov]

Suggest rename isCleanRelease() .

Model:
There are several reasons, and all of them are visible in the code.

1. The flag is encoded in a field that means something else. isFarewell (CasServerRootFormats.h:82) checks min_active_build_sequence == UINT64_MAX. That field is the build watermark ("the minimal active build"), and UINT64_MAX is
   a sentinel value inside it. The method name does not show that it reads the watermark. A reader of body.isFarewell() is more likely to expect a separate field.

2. One concept has several names. The state "the holder released the slot cleanly by itself" is named differently in different places:

   • isFarewell (the new helper);
   • clean_marker in claimMount;
   • FenceCertificate::CleanFarewell in classifyFenceCertificate;
   • "terminated" in listMounts / system.cas_mounts;
   • release / terminate in MountLeaseRenewer;
   • "farewell/retired sentinel" in prefixEligibleUnder.

   The operator sees terminated, the code says farewell, and the renewer calls it release.

3. "Retired" now means two different things. The field comment says UINT64_MAX = retired (farewell) (CasServerRootFormats.h:75). In the same PR, "retired" also means the owner tombstone (retired_at_ms, retirement_verified,
   member_retired). A farewell on the mount does not mean a retired member: every clean shutdown writes a farewell, and the member is not retired by that. In the decommission code the two concepts appear side by side, for example
   in verifyAndMark: farewell_held && tombstone_held.

4. The name does not say who wrote it. This is a property of the body that the holder wrote. But fencedCopy keeps min_active_build_sequence, so a body can be gc_fenced and isFarewell at the same time. Correctness depends on the
   check order: claimMount, classifyFenceCertificate and listMounts check gc_fenced first. The name hides this detail.

Suggestions

• Rename it to isCleanRelease() or holderReleased(), or at least to isFarewellSentinel(), so that it is clear it is a sentinel in the watermark.
• Use one word for the concept: release in the code and terminated in the system table. This can go into a separate refactoring PR, because the helper itself is a drive-by change from 86b20da (the Ockham note about the
  replacements in GC).
• In the field comment, replace "retired (farewell)" with "clean release (farewell)", so that "retired" is not confused with owner retirement.

[alsugiliazova]

PR #2567 Audit Review

Verdict

One High defect. The epoch counter and the kept control objects do what the PR says. The drain's reaction to a lost claim does not.

A decommission that loses its slot after it has already deleted data rolls the owner back to an unretired life. The member can mount again, immediately, onto the hole. Two narrower findings sit beside that: a retired owner never ends a live remount loop, and a late roots delete can hit a rejoined member's identical bytes.

What the change does

SYSTEM CAS DROP POOL MEMBER used to delete the victim's owner, epoch, and mount. A restart could then be handed a writer epoch it had already used, and a late delete of those objects could land on a successor. The PR keeps the three objects.

The run now:

1. Claims the victim's slot (Pool::openForDecommission, NoWait).
2. Tombstones the owner on the admin's fenced mount plane: the next owner_life, retired_at_ms set, an earlier verified mark kept. If-Match is the etag just read.
3. Reads the slot again. A lost claim rolls that tombstone back, unless the owner was already retired.
4. Drains namespaces, manifest debris, staging, and mountpoint objects on the fenced plane.
5. Writes the admin's farewell when the Pool is destroyed.
6. Marks retirement_verified on the next owner life. Only that mark sets member_retired = 1.

Every owner write goes through nextOwnerLife, so the life only grows. Every epoch allocation writes a fresh allocation_nonce, so two same-uuid processes cannot resolve an ambiguous counter write onto one epoch. Staging keys are staging/<srid>/<writer_epoch>/<leaf>.tmp; a mount sweeps only older epochs. A renewal that a stop refuses before any byte is sent stays Active, so shutdown can still write the farewell.

Model

Actors	The victim server, the admin running SYSTEM CAS DROP POOL MEMBER, the GC leader, a later start of the same server_root_id
Channels	Conditional PUT/DELETE on owner, epoch, and mount; the admin's fenced mount plane for the tombstone and the drain; an unfenced control plane for confirm, rollback, and the verified mark
Authoritative state	The owner body (server_uuid, owner_life, retired_at_ms, retirement_verified); the epoch counter and its nonce; the mount slot's (server_uuid, writer_epoch) body
Failure model assumed by the code	Crash-recover processes, a store that honors If-Match, in-flight requests that can land after the sender has stopped, a lease TTL that bounds how long a fenced admin keeps sending
Safe failure criterion	A writer epoch is never issued twice. A tombstone does not land on a later owner life. A member that is allowed to mount again still has the namespaces and roots objects it had when the run decided not to retire it.

The central argument, and where it holds

Keeping the control objects is the right fix for the two bugs named in the PR.

allocateWriterEpoch decides inside readModifyWrite. The engine pins that decision's bytes for the resolve read, and it clears any_ambiguous at the start of each inner write (CasRequests.cpp, writeLoop). A twin that bumped the same counter writes a different nonce, so the byte-equality resolve cannot adopt it. The loser re-decides against the stored counter and takes the next epoch. A resend repeats the same nonce. A missing counter over a non-empty member is refused; decommission does not mint one (Pool::openForDecommission).

A tombstone is replace against the etag read a moment earlier, and the body carries owner_life + 1. A late apply therefore conflicts once any later owner write has landed. recheckOwnerAfterClaim runs after every slot claim, including a self-remount, and fences the claimer's own body before it throws. A non-retired life change stops tryRemountOnce in IdentityLost before allocateWriterEpoch.

None of that is what the High finding breaks. The break is the policy that undoes a tombstone after the drain has already deleted.

Findings

Recovery and rollback

High: a fence during the drain un-retires the member after its data is already gone.

• Impact — member_retired = 0, the owner has no retired_at_ms, and namespaces or roots objects this run already removed stay removed. A restart claims the gc_fenced slot immediately (claimMount treats gc_fenced as a certificate of death) and serves the hole. decommissionPoolMember's scope exit still calls request_gc_round when any namespace reached Removing, so GC keeps deleting that member's rows after the owner has been restored.
• Anchor — decommissionPoolMember (Tools/CasDecommission.cpp), the confirmClaimOrRollBack on the drain-warnings path and on the exception path; rollBackTombstone; drainVictim, which drops namespaces before it deletes staging and roots.
• Trigger — GC fences the admin's mount body after at least one delete has been applied and before the farewell. A slow store or a missed renewal during a long drain is enough. The local lease deadline alone does not do it: adminClaimStillHeld reads the slot, and an unfenced body still counts as held, so DrainStopsOnTheAdminLeaseFence correctly leaves the tombstone in place.
• Why defect — the confirm before the drain is what proves the tombstone was exclusive. A later fence means this admin stopped being the slot holder, not that the tombstone was wrong. A successor cannot mount while the owner is retired, so rolling it back is what lets the member return. The store fence is the case the protocol is surest about, and it is the case that un-retires. RollbackRunsOnTheControlPlaneAfterAFence and FailedRemountStillRunsTheConfirm both expect owner life 3, not retired, after a roots delete that the fixture applies. Neither test checks that the deleted object is still there. The migration page says a drain failure leaves the unverified tombstone; the troubleshooting row says the same. Both are wrong for this path. The warning text is the accurate one: it says the tombstone was rolled back, and then it says to rerun, which reads as permission to start the member.
• Fix direction — once any namespace drop or object delete has succeeded, keep the tombstone, call withholdFarewell, and report rerun. Roll back only when the claim is already gone before the drain starts, which FenceBeforeTombstoneUntombstonesAndFails already covers and where the seeded file survives.
• Regression test direction — fence the admin after dropNamespace returns, or after the first roots delete, and assert the owner stays retired and the removed namespace does not reappear on a restart. Assert a_first is gone in the existing rollback test so the hole stays visible.

Observability and diagnosability

Medium: a decommissioned member's still-running process retries remount forever and never says the identity is lost.

• Impact — system.cas_mounts.lifecycle stays not_live, the same value as a short lease loss. Every 30 seconds the process reads the catalog and the owner, increments CASRemountFailed, and logs a warning. CASIdentityLost does not move. Nothing is written: the throw happens before allocateWriterEpoch. An operator watching the live process cannot tell a tombstone from a blip, and a restart is the only way to get the was decommissioned refusal.
• Anchor — claimOwnerOrThrow (Pool/CasServerRoot.cpp) calls throwIfOwnerRetired before it returns the life; Pool::tryRemountOnce enters IdentityLost only when that call returns a different life; the catch returns false; CasMountRuntime::remountLoop waits and calls it again until remountTerminal().
• Trigger — the process is up, its lease has been fenced, and SYSTEM CAS DROP POOL MEMBER tombstones its owner. A tombstone is a new life and it is retired, so the life comparison never runs. A rollback or a hand edit that clears retired_at_ms does reach IdentityLost, which is why the architecture page says "not retired".
• Why defect — the PR's own summary says an owner at another life ends the remount loop in IdentityLost. The life a decommission actually writes is the one that does not. The architecture page carves the retired case out, and the troubleshooting table has a row for identity_lost and no row for a process that stays not_live after its id was retired. k-morozov's review describes this path; the code matches it.
• Fix direction — on remount, treat a retired owner like a life mismatch and call enterIdentityLost. Add the troubleshooting row.
• Regression test direction — tombstone the owner under a live Pool, trip the lease, and assert lifecycle() == IdentityLost and that allocateWriterEpoch was not called again.

Partial and gray failure

Low: a delete still in flight removes a rejoined member's roots object when the bytes match.

• Impact — roots/<srid>/clickhouse_access_check, or any other mountpoint object rewritten with the same bytes, disappears after the new member has created it. Namespace, manifest, and epoch-scoped staging keys do not match, so committed table data is not in this class.
• Anchor — deleteListedPrefix (CasDecommission.cpp); CASDecommission.InFlightDeleteAfterRejoinHitsOnlyIdenticalRootsBytes.
• Trigger — S3's ETag for a single PUT is the MD5 of the body. The drain's conditional delete names that ETag. The troubleshooting procedure for bringing a retired id back rewrites the owner and the new process writes the access-check probe with the same bytes. A delete captured before the rejoin then matches.
• Why defect — the test calls this residual out, and the PR groups it with the late-apply assumption. It becomes an operator path on this PR because rejoin-by-rewriting-the-owner is now the documented recovery, and the roots keys are not generation-scoped the way staging keys are.
• Fix direction — put a writer epoch or a random suffix on mountpoint keys the successor rewrites, or refuse the late delete unless the mount slot is still the admin's farewell.
• Regression test direction — the existing test is the pin; extend it to assert the access-check object survives once the key stops being content-addressed.

Checked and clean

• encodeOwner omits owner_life when it is 0 and omits retirement_verified when it is false, so a 26.6.4 body keeps its bytes. Decoders tolerate the new fields missing. A 26.6.4 reader that skips unknown fields does not strip them, because a matching owner claim does not rewrite the object.
• rollBackTombstone refuses to clear an owner that was already retired (previously_retired). A paused run's rollback is If-Match on its own tombstone etag, so it cannot remove a rerun's later life. PausedRunCannotUntombstoneARerunsTombstone pins that.
• An unresolved tombstone whose read-back is not this run's bytes calls withholdFarewell. The slot keeps the admin's live body, and the victim reclaims only after a fence. The known leftover — a stop that lands after a renewal write was sent — still skips the farewell and is fail-safe: the next decommission reports member_retired = 0 and asks for a rerun.
• A stop that lands before the renewal sends anything returns NotAttempted and leaves the renewer Active (MountLeaseRenewer::renewOn). Shutdown can still release the slot.
• stagedEpochOf parses <epoch>/<leaf> and treats a flat leaf as epoch 0. The sweep deletes only epoch < live_writer_epoch. A nested member's key has an extra slash and is skipped. captureMountIncarnation admits, reads the epoch, then rechecks the admission, and a remount publishes the new epoch before it bumps the generation.
• recheckOwnerAfterClaim on a decommission passes AcceptForDecommission, so the admin's own tombstone does not fence the admin. After the tombstone, the admin pool's self-remount uses RetiredOwnerPolicy::Refuse and does not allocate another epoch (FencedAdminFailsInsteadOfRemounting).
• The verified mark is written on the control plane after the farewell, If-Match on this run's tombstone etag. A life that moved on is reported as member_retired = 0 with the "moved on" warning, not as retired.

Coverage summary

• Reviewed — network and transport (tombstone and renewal give-up, fence admission), partition and availability (lost admin claim, GC fence), timing and ordering (owner_life, epoch nonce, staging epoch), partial and gray failure (late apply, late delete), idempotency (rerun over a verified or unverified tombstone), recovery and rollback, configuration and version skew (26.6.4 bodies, absent counter, old decommission tool), observability (remount loop, result columns, warnings).
• N/A — security and trust across nodes: no new RPC path; the admin impersonates the victim only after reading its owner uuid, which is the existing decommission trust boundary.
• Deferred — backpressure of a large drain (renewal versus TTL) beyond the fence reaction above. LongDrainOutlivesOneTtl covers a healthy renewer; the High finding is the unhealthy one.
• Limits — static review of head d45685f0. The gtests were read, not re-run. dropNamespace's internal steps were not re-audited; the finding uses its returned success as the point of no return.

Operational readiness

Scope: SYSTEM CAS DROP POOL MEMBER and the following start of that server_root_id, PR #2567 at d45685f0.
Persona: Operator / DBA.
Environment: Shared CAS pool, one admin disk, the victim stopped or GC-fenced.

Primary flows

1. SYSTEM CAS DROP POOL MEMBER on a member whose system.cas_mounts row is terminated or fenced, then read member_retired.
2. Leave the victim process running, or let an orchestrator restart it, after that command.
3. Bring a retired id back by rewriting the owner body, as the troubleshooting page describes.

Flow 1: Drop a stopped member

1.1 A fenced drain reports failure and has already deleted data

• Symptom: The row comes back member_retired = 0 and a warning that the tombstone was rolled back. namespaces_removed or mountpoint_objects_removed is already non-zero. The troubleshooting page still says the owner keeps its tombstone (docs/en/antalya/cas/operations/troubleshooting.md, the member_retired = 0 row).
• Trigger: GC fences the admin during the drain. See the High finding.
• Signal quality: Confusing. The warning and the doc disagree, and neither says the data is already gone.
• Severity: High.
• Recovery: Do not start the member. Rerun the command and wait for member_retired = 1 before treating the id as either alive or finished.

Flow 2: The victim process is still up

2.1 The process looks like a lease blip after it has been decommissioned

• Symptom: lifecycle = 'not_live', remount warnings every 30 seconds, no identity_lost. See the Medium finding.
• Trigger: Decommission tombstones the owner while this process is still running.
• Signal quality: Confusing.
• Severity: Medium.
• Recovery: Restart the server. The start refuses with was decommissioned.

Flow 3: Hand-edit the owner to rejoin

3.1 The access-check object can vanish after the new server creates it

• Symptom: roots/<srid>/clickhouse_access_check disappears shortly after the restart. See the Low finding.
• Trigger: Rewrite the owner and restart while a drain delete of that key is still in flight.
• Signal quality: Silent.
• Severity: Low.
• Recovery: Restart once more after the decommission run has finished, so the probe is rewritten with no delete outstanding.

Readiness verdict

• Flow 1 in isolation: ready with caveats, because of 1.1. A drain that keeps its lease finishes and sets member_retired = 1. A drain that is fenced does not leave a member you can safely start.
• Flow 2 in isolation: ready with caveats, because of 2.1. The process does not write, and it also does not stop.
• Flow 3 in isolation: ready with caveats, because of 3.1.
• Overall in production: ready with caveats. Guardrails below, until 1.1 is fixed. Do not treat member_retired = 0 as "nothing changed".

Operator guidance

• After member_retired = 0, read the warning. If it says the tombstone was rolled back, do not start that server_root_id. Rerun the drop and wait for member_retired = 1 (1.1).
• Ignore the troubleshooting sentence that says the owner keeps an unverified tombstone whenever the claim was lost. That is true only when the slot was not fenced (1.1).
• A process that stays not_live and keeps logging remount failures after its id was dropped is retired. Restart it so the refusal becomes visible (2.1).
• Rewrite the owner to rejoin only after the drop command has returned and no second drop is running (3.1).

Recommended fixes

• Keep the tombstone after the first successful delete, and withhold the farewell (1.1).
• Make a retired owner terminal in the remount loop, and add the troubleshooting row (2.1).
• Correct the migration and troubleshooting sentences that say a lost claim always leaves the tombstone in place (1.1).
• Stop roots deletes from matching a successor's identical bytes (3.1).

[alsugiliazova]

PR #2567 CI Verification

Verdict

CI blocks approval. One job is unknown.

The head is decommission, owner life, and the epoch nonce. The diff has no Iceberg file, no IDataType change, and no image or package change. The red regression jobs fail the same scenarios on other commits. The stateless failure is a refused connection to object storage, and the job's own rerun passed. The asan unit test is the exception: the sanitizer stack is outside this diff, and the test still failed on all five commits of this pull request, well above its rate on other pull requests.

Category	Count	Jobs
regression	0	—
pre-existing-flaky	5	cas_alter_attach_3, cas_s3_cache_alter_attach_3, cas_lightweight_delete_4, Grype on the keeper image, Grype on the Alpine server image
infrastructure	1	Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)
cascade	parent scenarios inside the three regression jobs	shorter feature paths above each leaf below
unknown	1	Unit tests (asan_ubsan)

The PR rollup is red because those jobs are red.

How the list was reduced

gh-data.checks has two FAIL rows on the head: 03755_circular_dictionary and IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop.

The regression database has fail rows in three jobs. Each recorded leaf is a scenario path. The shorter prefixes of that path (/alter, /alter/attach partition, /lightweight delete, and so on) are the same failure reported again at the parent feature.

1. Cascades. Parent feature rows in cas_alter_attach_3, cas_s3_cache_alter_attach_3, and cas_lightweight_delete_4 repeat the leaf below them. They are not separate failures.
2. Environment, from the log text. 03755_circular_dictionary died while publishing a blob: e.code() = 111, Connection refused. The in-job diagnosis reran with the same settings and passed 6 of 6. Grype reports the same Alpine High advisories as on other images built from this base.
3. Rates, the regression leaves. Every leaf below already fails on the same job on other commits. cas_alter_attach_3 has a fail row on 19 of 19 other commits in this window. cas_s3_cache_alter_attach_3 has one on 17 of 19. cas_lightweight_delete_4 has one on 18 of 19.
4. The unit test does not fit that rate. Five commits of this pull request, five failures. Other pull requests fail 34 of 174 commits (20%). The branch fails 1 of 12. The stack does not enter this diff, so the category is unknown.

Findings

Unit tests (asan_ubsan) — unknown

IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop aborts in UBSan:

runtime error: member call on null pointer of type 'DB::IDataType'
DB::WhichDataType::WhichDataType(shared_ptr<IDataType const> const&)
  src/DataTypes/IDataType.h:380
DB::isNothing
  src/DataTypes/IDataType.cpp:520
DB::Iceberg::stripNothing
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /ClickHouse/src/DataTypes/IDataType.h:380:67

Job 118316963854. The same summary line is the failure text on other pull requests. This diff does not change IDataType.h, IDataType.cpp, or Iceberg.

On this job, over 60 days, counted by commit:

Who	Commits	Commits that fail this test
Other PRs	174	34 (20%)
antalya-26.6 branch	12	1 (8%)
PR 2567	5	5

The five failing commits are 81e5e081 (2026-10-14), d6ffc7b7, 9fc9db0f, 3a2641dc, and the head. A 20% test can fail twice in a row. Five commits in a row is a different rate, and there is no path from owner life, the epoch nonce, or decommission into stripNothing.

What would close this: a run of the merge base that fails in stripNothing at a similar rate, or a frame that shows this diff passing a null type into that function. Until one of those exists, this job blocks approval.

Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) — infrastructure

03755_circular_dictionary failed once on the head. The server log in the test context is a refused connection while publishing a blob:

CasBlobPublishAttempt: Publication of CAS blob cas_s3/blobs/ch128/cf/cf7ba47141a3b0b20d1d56789c00082e (streaming) failed, attempt 1 of 8:
Poco::Exception. Code: 1000, e.code() = 111, Connection refused
bucket test, key cas_s3/blobs/ch128/cf/cf7ba47141a3b0b20d1d56789c00082e

The job then reran with the same randomized settings: 6 runs, 0 failed. Job 118354995975. Check summary: Failed 1, Passed 5554, Skipped 142.

Across every check, 60 days, this test has 0 fails in 572 branch runs and 0 fails in 3,365 other-PR runs. On this pull request the earlier four commits passed it. The failure is the storage endpoint refusing the connection. Rerun the job.

RegressionTestsRelease / CASAlter (attach, 3) — pre-existing-flaky

Job 118344581525. Seven leaves failed. Each one is NETWORK_ERROR (210) while a CAS write was being committed. Two wordings show up:

• read of 'data/cas/ref_catalog': gave up at the policy deadline after N attempt(s) during CREATE TABLE
• CAS _ckpt for namespace '...@cas@': persistent CAS contention, the checkpoint contribution was not published

/alter/attach partition/part 3/operations on attached partitions/update in partition is code 341: mutation mutation_62.txt failed because the inner error is that same code 210.

Parent rows (/alter, /alter/attach partition, /alter/attach partition/part 3, and the feature nodes directly above each leaf) are cascade.

Same job, 60 days, excluding this commit:

Scenario	Runs	Fails
.../partition key datetime/attach partition from	183	66
.../partition key/attach partition from without id	183	63
.../check simple attach partition 2/simple attach partition	183	61
.../partition key/attach partition from with id	183	56
.../operations on attached partitions/update in partition	183	54
.../rbac/check attach partition from with privileges	183	38
.../conditions/primary key	183	18

The job itself has a fail row on 19 of 19 other commits in the window. One red run is the baseline for this lane. The deadline and contention sentences are the existing CAS commit retry, not the decommission steps in this diff.

RegressionTestsRelease / CASS3CacheAlter (attach, 3) — pre-existing-flaky

Job 118344581389. One leaf:

/alter/attach partition/part 3/partition key/attach partition from with id

Code: 210. DB::Exception: CAS write could not be committed
(CAS _ckpt for namespace 'alter-cas-clickhouse1/store/c78/c7847f38-...@cas@':
persistent CAS contention, the checkpoint contribution was not published)

The four shorter prefixes of that path are cascade. On this job, excluding this commit, the leaf failed 34 times in 180 runs. The job has a fail row on 17 of 19 other commits.

RegressionTestsRelease / CASLightweightDelete (4) — pre-existing-flaky

Job 118344582054. One leaf:

/lightweight delete/concurrent delete/MergeTree/random delete half of the table without overlap

concurrent_delete.py:350
assert r.output == f"{block_size * partitions * (100 - percent_to_delete) // 100}"
AssertionError: Oops! Assertion failed

/lightweight delete, /lightweight delete/concurrent delete, and .../MergeTree are cascade. On this job, excluding this commit, the leaf failed 16 times in 190 runs. The job has a fail row on 18 of 19 other commits. The concurrent-delete row count is the usual miss for this scenario.

Grype — pre-existing-flaky

Both scans fail on Alpine High advisories. The keeper image reports CVE-2026-84782, CVE-2026-84784, CVE-2026-85091, and CVE-2026-72897. The Alpine server image reports CVE-2026-85091. Jobs 118352883660 and 118353778287. The diff does not change a Dockerfile, a package list, or an image base. A rerun scans the same base and fails the same way.

Recommendation

Do not approve on CI grounds while Unit tests (asan_ubsan) is unknown.

Rerun Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2) if you want that box green. A rerun will not clear Grype, and it will not clear cas_alter_attach_3: that job is red on every other commit recorded in the last 60 days. The cache-alter and lightweight-delete jobs are red on 17 and 18 of 19. The unit test has failed on every commit of this pull request; a single green rerun would still leave the five earlier commits to explain.