Antalya 26.8: Implement additional Iceberg drop partition features

[xieandrew]

Based on backport PR #2533 for basic DROP PARTITION for Iceberg. Rework of #2361 for 26.8. Adds support for catalog tables, purging data files, mixed manifests, equality deletes, and partition spec evolution for Iceberg DROP PARTITION.

Changelog category (leave one):

• Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for catalog tables, purging data files, mixed manifests, equality deletes, and partition spec evolution for Iceberg DROP PARTITION.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [d73e882]

[zvonand]

please, do not forget to put port-antalya labels to the PRs that will later be ported to future releases

[zvonand]

@blau-ai

[blau-ai]

CI triage for d73e8821

Verdict: 0 PR-caused failures. Every red check is pre-existing, flaky, or infra. The failures land in subsystems this PR doesn't touch (Hybrid table engine, swarms node-failure stress, Azurite config, container CVE scans), while the suites that do exercise your code — iceberg_1, iceberg_2, all s3_*/parquet_*/alter_*partition regression modules, and every stateless/integration/unit test — are green. This PR only changes src/Storages/ObjectStorage/DataLakes/Iceberg/*, the partition docs, and two Iceberg integration tests.

Evidence baseline: the previous merge into antalya-26.8 (#2586, e1e35b8) shows the same Grype, Parser-memory, and swarms failures, and passed the stress test — i.e. these are branch-level/flaky, not introduced here.

Check	Class	PR-caused?	Evidence
GrypeScanKeeper (4 high/crit)	infra / deps	❌ No	Identical 4 vulns on base #2586; no Dockerfile/deps in this PR
GrypeScanServer (-alpine) (1 high/crit)	infra / deps	❌ No	Identical on base #2586
Parser memory check	infra	❌ No	ERROR: No master clickhouse-examples artifact was found (missing baseline binary); same failure on base #2586
Stress test (amd_asan_ubsan)	pre-existing flaky	❌ No	See below — Hybrid-engine test, not Iceberg; passed on base #2586
Regression swarms	flaky / infra	❌ No	Same module fails on base #2586; node-failure/cpu-overload socket timeouts
Regression s3_azure_1	flaky / infra	❌ No	Azurite/config env issue; s3_azure_2 + all aws_s3/gcs/minio modules pass
PR (aggregate gate)	rollup	❌ No	Red only because of the above

Details

Stress test (amd_asan_ubsan) — server fails to restart:

Code: 60. Table default.test_tiered_watermark_before does not exist.
  Cannot attach table test_5.test_tiered_watermark ...
  ENGINE = Hybrid(remote(..., 'test_5', 'test_tiered_watermark_after'), date >= '2025-09-01',
                  remote(..., currentDatabase(), 'test_tiered_watermark_before'), date < '2025-09-01')

test_tiered_watermark comes from tests/queries/0_stateless/03643_hybrid.sql, the Hybrid table engine feature (merged via #2144), not from this PR. The restart fails because currentDatabase() embedded in the stored remote(...) resolves to default instead of test_5 on ATTACH. It's restart-sample-dependent (hence it passed on #2586) and lives entirely in the Hybrid feature. Not related to Iceberg drop-partition.

Regression swarms (3 scenarios, also failing on base #2586):

• cluster discovery/check cluster discovery with wrong cluster name → Code: 279 ALL_CONNECTION_TRIES_FAILED
• node failure/cpu overload → Code: 209 SOCKET_TIMEOUT (300s) under induced node failure + CPU overload
• node failure/cpu overload all swarm nodes → Code: 209 SOCKET_TIMEOUT

All are server-side connection/timeout errors under node-failure stress conditions. No failing scenario name mentions iceberg/drop-partition; the two node-failure cases issue SELECT count() reads against datalakecatalog_db_* tables (not a write/drop path) and time out under load.

Regression s3_azure_1 (2 scenarios):

• disk/remote host filter → server unhealthy: URL "http://azurite:10000" is not allowed ... remote_url_allow_hosts (UNACCEPTABLE_URL) + SSL tlsv1 alert unknown ca
• disk/syntax → check_preprocessed_config_is_updated assertion (config reload)

Both are Azurite endpoint / config-reload environment issues. Note s3_azure_2 and every other object-storage backend (aws_s3, gcs, minio) passed, which rules out a systemic storage-code regression.

Recommended next steps

• Re-run Stress (amd_asan_ubsan), Regression swarms, and Regression s3_azure_1 — they're flaky/infra and unrelated to this change.
• Grype and Parser memory check are infra/branch-level (missing master baseline binary; base-image CVEs); they are red on the base branch too and aren't fixable from this PR.
• Optional, out of scope for this PR: the Hybrid-engine restart bug (currentDatabase() captured literally in a remote(...) and mis-resolving on ATTACH) is a genuine latent issue worth a separate ticket against the Hybrid feature (Antalya 26.6: Hybrid tables #2144).

PR health

The change is scoped and self-consistent: Iceberg drop-partition code + docs + targeted integration tests, with the Iceberg and object-storage regression suites and the full functional/integration/unit matrix passing on d73e8821. No action is required on this PR for the current red checks.

_{🤖 Automated triage by @blau-ai. Report data is from the praktika S3 reports; I cannot build or run ClickHouse, so final correctness is confirmed by CI.}