Repository navigation
Conversation
The expected result matches initiator-side join semantics; the test currently fails because shards emit extra unmatched right-side rows. Related: #1664 Co-authored-by: Cursor <cursoragent@cursor.com>
Broadcast the original left table and keep the preserved right side sharded, so unmatched right rows are not duplicated on every shard. Related: #1664 Co-authored-by: Cursor <cursoragent@cursor.com>
After ClickHouse#94748 the planner always wrapped the left cluster source, so object_storage_cluster_join_mode='global' never saw the JOIN and ran on the initiator. Keep wrapping for allow/local and for RIGHT/FULL to avoid duplicated unmatched rows. Co-authored-by: Cursor <cursoragent@cursor.com>
Port of the review fixes made for the `master` pull request ClickHouse#118611 `tryRewriteGlobalRightJoinAsLeftJoin` only recognised a narrow shape of the query it was meant to fix, so a `GLOBAL RIGHT JOIN` whose left table is a sharded `Distributed` table still emitted the rows preserved from the broadcast side once per shard in these cases: * `USING` and `NATURAL` joins were skipped, because only `ON` was accepted. A `USING` key stores its sides positionally, so the key pairs are now swapped together with the table expressions, and a key that does not hold a plain column per side leaves the query alone. * The rewrite required the right side to be a sharded `Distributed` table too. Only the left table fans the query out across shards, so the gate is now on the left table alone, and a local table, a one shard `Distributed` table, a subquery or a table function on the right is handled as well. `Any`, `Semi` and `Anti` join over to the swapped sides the same way `All` does and are accepted; `Asof` and `RightAny` do not mirror and stay out. * Only the topmost join was examined. Join trees are left deep, so the walk now descends to the join that reads the leftmost table. A `RIGHT` or `FULL` join that survives the rewrite, such as the one in `(A INNER C) RIGHT B`, cannot be swapped without moving a join into the right table expression. Those now read the leftmost table through a subquery, which keeps the join on the initiator, the only remaining way to emit the preserved rows once. `getDistributedStorageFromTableExpression` also missed `remote` and `clusterAllReplicas` on the left, because it only looked at `TableNode`. Co-authored-by: Cursor <cursoragent@cursor.com>
…istributed tables. An `Alias`, `Buffer`, `MaterializedView` or `StorageProxy` in front of a sharded `Distributed` table still fans the query out across shards, so unmatched right rows would otherwise be duplicated once per shard. Co-authored-by: Cursor <cursoragent@cursor.com>
…al-right-join-test
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
…al-right-join-test
After the join is rewritten as a LEFT JOIN, a `PREWHERE` bound to the original left table followed it into a `Memory` table and shards threw `ILLEGAL_PREWHERE`. Move the predicate into the broadcast subquery when its columns belong to that table, and skip the swap when `PREWHERE` has no column so the initiator wrapper keeps the filter. When wrapping a remote table that owns `PREWHERE`, move the full predicate onto the wrap and drop the initiator actions so nested `GLOBAL IN` sets are not materialized twice. Port of ClickHouse#119780 Co-authored-by: Cursor <cursoragent@cursor.com>
The namespace janitor deleted a dropped table's dead `_log`/`_snap` keys one page per round, one `HEAD` + `DELETE` per key. `Gc::runNamespaceJanitor` now repeats `NamespaceJanitor::runOnePage` while a page deleted something, until a 20 s soft budget. A page deletes its dead `_log`/`_snap` keys as write-once batches split evenly across the GC I/O pool; `_ckpt` and `_files` keep the exact-token delete. After the last page of the stream the pass continues from its beginning, so debris on both sides of the cursor drains in one round. A smaller implementation of #2468. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A pool that refused a delete job threw out of `runOnePage` and lost that page's counters and anomalies. The unscheduled keys now count as leaked, like a failed request, and the page is published. Tests: a quiet pool larger than one page takes one `LIST`; a refusal after the first job; the split across the pool on a store without a batch delete; a dead and a reborn life on one page. Docs: the phase 16 cost table and `namespaces.md` described one page per round and a `HEAD` + `DELETE` per object. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The live namespace had no `_ckpt`, so the fold suppressed the round and the janitor page was never decided: the test passed with the `deleted > 0` clause of `more` removed. The namespace now has a checkpoint and 1200 `_files` keys, and the test asserts the published cursor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…tch-size setting `cas_gc_bulk_delete_chunk_keys` now also caps the namespace janitor's batch deletes; its description said only phases 15 and 17. The test pins that a listed key which is not the canonical key of its parsed identity gets no write-once form. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
- A failed `LIST` reset the persisted cursor on any page. It now resets only on the first page of a pass; later the cursor the previous page published stays, so one transient failure mid-drain does not send the next round back to the stream start. - `leaked` is zero for a page that did not publish its cursor: the page is listed again, so authority lost mid-page no longer inflates `CASGCNamespaceCleanupLeaks`. - New phase metric `delete_requests`: the delete jobs run on the GC I/O pool, so the phase row's `ProfileEvents` miss their requests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A batch store drains in under a tenth as many delete calls as keys; a store without a batch delete makes at least one call per key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The old analyzer drops constant GROUP BY keys when distributed_depth is 0. Cluster table function reads never increased that depth, so workers folded hostName() and returned only the aggregate. https: //github.com//issues/2139 Co-authored-by: Cursor <cursoragent@cursor.com>
Raising distributed_depth on cluster reads also changes parallel replicas, max_distributed_depth, and block marshalling. The old analyzer now keeps a key such as hostName() and still drops ordinary constants. https: //github.com//issues/2139 Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
__getScalar is marked as a server constant, but the initiator evaluates it and sends the same value to every shard. Keeping it made GROUP BY of a scalar subquery with Dynamic throw ILLEGAL_COLUMN. https: //github.com//pull/2490 Co-authored-by: Cursor <cursoragent@cursor.com>
…-26.6/cas-gc-janitor-time-budget Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> # Conflicts: # docs/en/antalya/cas/configuration.md # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp
…eaks per key With `Gc::bulkDeleteChunkKeys` in the base, a page's dead keys go as one job per that many keys instead of an even split across the pool: 1000 keys are one `DeleteObjects` on S3, and a storage without a batch delete sends one request per key with no refused batch first. A job whose batch the storage refuses deletes its keys one by one and accounts each on its own, so a failure late in the chunk leaks only the keys after it. The unscheduled chunks after a refused enqueue are counted from an explicit `scheduled` index. Integration test: 1100 parts instead of 2500 (the INSERT timed out under MSan), an 1800 s INSERT timeout, and a disk with `objects_chunk_size_to_delete = 100`. Rename `remove_write_once` to `remove_write_once_keys`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Antalya 26.6 - Backport of 11568 - Parquet: size the initial footer read to 1% of the file within [128 KiB...2 MiB]
…al-right-join-test
A `GLOBAL` `INNER` or `LEFT` join sends the right table and reads the left table on the shards. A `GLOBAL` `RIGHT` join of a multi-shard `Distributed` table sends the left table instead, so unmatched rows of the preserved side are returned once. `FULL` joins and `object_storage_cluster_join_mode` `allow` and `local` run the join on the initiator. Co-authored-by: Cursor <cursoragent@cursor.com>
…on ARM sanitizer builds ARM sanitizer builds have been built without SimSIMD since #2341, so USearch converts float32 to bf16 with its own fallback. The fallback truncates while SimSIMD rounds, and the test gets 0.1875 instead of 0.203125 for id 7. It fails deterministically on Stateless tests (arm_asan_ubsan, azure, parallel). Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
…e_cluster. Alias, Buffer, Merge, materialized views, loop, and Distributed all resolve a fresh catalog storage, so each one is checked the same way for local, query, and database cluster settings. Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: CarlosFelipeOR <carlosfelipeor@gmail.com>
The customer filter, limit, and count checks now run against the REST catalog and cluster_simple, including parallel replicas, so the stateless REST stub is no longer needed. Co-authored-by: Cursor <cursoragent@cursor.com>
Related: #2550 Co-authored-by: Cursor <cursoragent@cursor.com>
Related: #2550 Co-authored-by: Cursor <cursoragent@cursor.com>
…rtial-column-stats Do not treat missing Iceberg column metrics as NULL
`StorageSnapshot` only referenced the target, and `DataLakeCatalog` drops that storage when `getStorageSnapshot` returns. The analyzer then read freed memory under ASan. https: //altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2550&sha=2a09ee317a90d65f72165bbb9f1c4d407ec41c48&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%208%2F8%29 Related: #2550 Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit abc516e.
`StorageSnapshot` keeps a bare reference to its storage, and `Alias` returned that snapshot after the temporary from `getTargetTable` was destroyed. This brings `withStorageHolder` from upstream, cloning snapshot `Data` because it is still a `unique_ptr` on this branch. https: //github.com/ClickHouse/pull/120098 Co-authored-by: Cursor <cursoragent@cursor.com>
`getStorageSnapshotWithoutData` had no pin test, and the data clone was only exercised with an empty snapshot. The stateless join now goes through a `View`, because an `Alias` of a `Distributed` table does not itself request filter collection. Related: #2557 Co-authored-by: Cursor <cursoragent@cursor.com>
A null snapshot made the pointer comparison pass even when `clone` dropped the field. Related: #2557 Co-authored-by: Cursor <cursoragent@cursor.com>
…shot-storage-holder Antalya 26.6, backport of 120098: Keep the target storage alive for a snapshot handed out by an Alias table
Antalya-26.6 - Round distances in 02354_vector_search_rescoring_distance_in_select_list
`getTargetTable` returns a `StoragePtr` that can be the only owner of a catalog storage. Destroying it before `isMergeTree` was a heap-use-after-free under AddressSanitizer. Related: #2550 https: //altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2550&sha=bb444f10dd1de59c9ae3adcf33540371cedf6c5b&name_0=PR&name_1=Integration%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20old%20analyzer%2C%208%2F8%29 Co-authored-by: Cursor <cursoragent@cursor.com>
Add CAS disk to existing backup tests
…ame-object-storage-cluster Fix GROUP BY server constants over s3Cluster and object_storage_cluster
…erg-cluster-metadata Fix metadata initialize for complex engines over cluster request.
…igned-to-signed Accept unsigned to signed promotion when possible
Brings 16 merges from antalya-26.6, among them #2521 (null type in Iceberg `MODIFY COLUMN`, the `IcebergMetadataGenerator` UBSan red on every PR of this base), #2529 (Alpine upgrade), #2483 (CAS GC janitor time budget) and the version bump to 26.6.8.20002. `docs/en/antalya/cas/configuration.md`: keep both edits, the janitor batch deletes in `cas_gc_io_concurrency` and the blob-publication note in `cas_attempt_timeout_ms`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge
antalya-26.6intoantalya-26.6-20261008: 16 merges, 74 commits. Merge with a merge commit; open CAS branches are rebased onto18cb4952478b.MODIFY COLUMN. It is theIcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoopUBSan red on every PR into this base.One conflict,
docs/en/antalya/cas/configuration.md: both edits kept (janitor batch deletes incas_gc_io_concurrency, blob-publication note incas_attempt_timeout_ms).Local gates on
18cb4952478b: release and ASanunit_tests_dbms --gtest_filter='CAS*:IcebergMetadataGenerator*'passed (2767 and 2771 tests).Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: