Repository navigation
Conversation
Server start creates each table's `disk = disk(...)` custom disk inside `Context::getOrCreateDisk`, which holds `storage_policies_mutex` while the disk starts. A CAS disk whose previous process left no farewell waits out the stale mount lease during that start, so N stale mounts cost N observation windows and every disk or storage-policy lookup in the server blocks meanwhile. Both tests fail on the current code: three stale mounts take 10820 ms against a 3450 ms window, and `getDisksMap` blocks for 3603 ms. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2516&sha=b7e424247ab997ae2b0d3c06853e10a2c445f2a5&name_0=PR&name_1=Stress%20test%20%28amd_debug%29 PR: #2516 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Context::getOrCreateDisk` ran the disk creator under the server-wide `storage_policies_mutex`. A CAS disk waiting out a stale mount lease (about 36.5 s) inside the creator blocked every other disk creation and every disk and storage policy lookup, so stale mounts were waited out one after another at server start. The mutex now guards only the map of in-flight creations and the final insert into `DiskSelector`. Concurrent requests for one name share one creation and get the same disk or a copy of the same exception. A failed creation is still not cached. CI report: #2516 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A configuration reload could create a configured disk whose name was being created as a custom disk, opening the same storage twice. The reload now skips such names and logs a warning. Callers of a shared disk creation rethrow a copy of the stored exception through `Poco::Exception::rethrow`, so `ErrnoException` and other derived types keep their type and fields. Other exceptions become a `DB::Exception` with the same code and message. Tests: shared failure with independent copies, config reload during a creation, unique disk names so the suite survives `--gtest_repeat`, and the CAS logger channel is restored only after the disks stop. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the existing `copyMutableException` helper: `Poco` exceptions are copied through their virtual `rethrow`, other exceptions are rethrown unchanged. State the `disks_to_skip` contract in `DiskSelector.h`, wrap a long log line, and bound the waits in the shared-failure test. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #2573 Audit ReviewVerdictThe core concurrency change is correct. The hand-off between the creating thread and the Two Medium findings and three Low. The most consequential is not a bug in the new code but the What the change does
The function now runs in three phases.
Model
Why the core hand-off holdsThe admission and completion phases are the same critical section, so the window the old code A caller arriving between The promise lives on the creator's stack but its shared state is reference-counted by the One pre-existing hazard is removed rather than added. Under the old code a creator that called FindingsPartition / availabilityMedium: the stall remains for CAS disks defined in the configuration.
Recovery & rollbackMedium: a configuration reload inside a custom-disk creation drops every storage-policy change in
Observability & diagnosabilityLow: every failed disk creation is now rethrown through
Low: a disk that is being created is reported as
Low:
The timing bound in Coverage
Assumptions: static reading of head Operational readinessScope. Server start, configuration reload, and table loads on a server with CAS disks, at head Issues1. Configured CAS disks still freeze the server during start
2. A reload during a table load can silently discard policy changes
3. A disk in the middle of being created looks like a typo
Readiness verdictMergeable for what it targets. The new code is the correct shape and its tests pin the behaviors Recommended fixes
|
PR #2573 CI VerificationVerdictNo failure is caused by this PR. Approvable on CI grounds. Seven failing checks resolve to five distinct causes, all of which reproduce on sibling PRs that Summary
Funnel: 7 failing checks, 0 removed as cascade, 4 placed by sibling-PR comparison, 2 placed by The PR's own tests passAll six tests in the new
The first test is the one that would have failed before the change: three stale mounts opened in FailuresUnit tests (asan_ubsan) and Unit tests (tsan) —
|
Run the creator of a
disk(...)custom disk outside the server-widestorage_policies_mutex.Context::getOrCreateDiskran the disk creator under that mutex. A CAS disk whose previous process exited without a farewell waits out the stale mount lease inside the creator (about 36.5 s with the default 30 s lease). While it waited, every other custom-disk creation and everygetDisk,getDisksMapandgetStoragePolicycall queued behind it. With three stale mounts, server start would take about 110 s; the stress test harness gave up at its limit with "Cannot start clickhouse-server":#2516 (Stress test (amd_debug), https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2516&sha=b7e424247ab997ae2b0d3c06853e10a2c445f2a5&name_0=PR&name_1=Stress%20test%20%28amd_debug%29)
Changes
getOrCreateDiskreleases the mutex while the creator runs. The mutex still guards selector initialization, lookup, the disks-map snapshot passed to the creator, the in-flight creation map, the final insert intoDiskSelector, and configuration reload.Poco::Exception; the dynamic type is kept for exception types that overriderethrow(ErrnoException,NetException). Other exceptions are rethrown unchanged.Risks / notes
DiskSelector::initialize.Testing
src/Disks/tests/gtest_cas_startup_stale_mounts.cpp, all also with--gtest_repeat=3:getDisksMapcall during a stale-mount wait returns under the 1.15 s bound instead of after 3.6 s.ErrnoExceptioncopy.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Creating a
disk(...)custom disk no longer blocks other disk creations and disk or storage policy lookups. Server start with several CASdisk(...)custom disks that wait out a stale mount lease now waits for them in parallel instead of one after another.Documentation entry for user-facing changes
No documentation change: no setting, syntax or documented behavior changes.
CI/CD Options
Exclude tests:
Regression jobs to run: