Repository navigation
Antalya 26.8: Forward auth tokens to data lake catalogs - #2641
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #2329 from Altinity/feature/antalya-26.6/aouth-forward-to-datalake Antalya 26.6: Forward auth tokens to data lake catalogs # Conflicts: # docs/reference/engines/database-engines/datalake.mdx # src/Access/AccessControl.h # src/Common/ErrorCodes.cpp # src/Core/ServerSettings.cpp # src/Databases/DataLake/DatabaseDataLake.cpp # src/Databases/DataLake/DatabaseDataLakeSettings.cpp # src/Databases/DataLake/GlueCatalog.cpp # src/Databases/DataLake/GlueCatalog.h # src/Databases/DataLake/HiveCatalog.cpp # src/Databases/DataLake/HiveCatalog.h # src/Databases/DataLake/ICatalog.cpp # src/Databases/DataLake/ICatalog.h # src/Databases/DataLake/PaimonRestCatalog.cpp # src/Databases/DataLake/PaimonRestCatalog.h # src/Databases/DataLake/RestCatalog.cpp # src/Databases/DataLake/RestCatalog.h # src/Databases/DataLake/S3TablesCatalog.cpp # src/Databases/DataLake/S3TablesCatalog.h # src/Databases/DataLake/UnityCatalog.cpp # src/Databases/DataLake/UnityCatalog.h # src/Databases/DataLake/tests/gtest_rest_catalog.cpp # src/IO/S3/Credentials.cpp # src/IO/S3/tests/TestPocoHTTPServer.h # src/Interpreters/Session.cpp # src/Server/TCPHandler.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp
Kept antalya-26.8 behaviour where the cherry-pick only carried pre-existing antalya-26.6 code, and translated the PR's changes to antalya-26.8's shape: - Catalog listing API: antalya-26.8 returns `CatalogTables` from `getTables` and `listTablesInNamespaceDirect`; the PR's `auth_token` parameter is added to those signatures in every catalog. - `RestCatalog`: merged the `flat_namespaces` constructor argument with the PR's `TokenForwardingConfig`; `createReadBuffer` and `sendRequest` keep antalya-26.8's virtual, default-free signatures with the PR's `generation` and `auth_token` added, so call sites pass all arguments explicitly. `getAuthHeaders` takes the PR's `AuthContext`; the OneLake refresh-token branch is kept on top of it. `sendRequest` passes empty `extra_headers` because antalya-26.8 has no such local there. - Vended credentials cache: kept antalya-26.8's mutex-guarded TTL and the `state_snapshot` staleness check in `cacheCredentials`, keyed by the PR's `CredentialsCacheKey` and bounded by `credentials_cache_max_entries`. - `requestToken` / `retrieveAccessToken`: took the PR's refactor and kept antalya-26.8's Horizon rule of omitting an empty `client_id`; the antalya-26.8-only OneLake refresh-token function is unchanged. `HorizonCatalog` (antalya-26.8 only) uses the new `loadConfig` signature. - `S3TablesCatalog`: antalya-26.8 signs requests by overriding `createReadBuffer` / `sendRequest` and has no `getAuthHeaders` or `getCredentialsConfigurationCallback` override, so the PR's versions of those overrides are not applied; the existing overrides gain the new parameters. - `UnityCatalog`, `GlueCatalog`, `DatabaseDataLake`: token arguments added to antalya-26.8 code paths (`getWriteCredentialsConfigurationCallback` kept, credentials-refresh lambda, `CatalogTables` listings, `createNamespaceIfNotExists` with an empty location). - `AlterDropPartitionExecutor` (antalya-26.8 only) passes the forwarded token to the new `updateMetadata` signature. - `CATALOG_USER_TOKEN_NOT_AVAILABLE` renumbered to 784 after the codes antalya-26.8 already uses. - `gtest_rest_catalog.cpp`: kept antalya-26.8's in-file test server (it backs tests that exist only on antalya-26.8) and applied the PR's `auth_token` arguments to the catalog calls. - `TCPHandler::processTablesStatusRequest`: the PR's `try`/`catch` around the loop, with antalya-26.8's `castStorage(..., DeferredTable::Load)`. Source-PR: #2329 (#2329)
zvonand
added
releasy
CI triage for #2641 (head
|
| Check | Status | Classification |
|---|---|---|
| Parser memory check | ❌ | Infra (missing master artifact) |
| Regression release swarms | ❌ | Not PR-caused (env/timing) |
| Regression aarch64 swarms | ❌ | Not PR-caused (env/timing) |
1. Parser memory check — infra, not PR-caused
The failing sub-step is "Resolve master binary → No master clickhouse-examples artifact was found". This check diffs parser memory usage against a master/base build's clickhouse-examples binary, and that reference artifact wasn't available for this run, so the job errored before doing any actual measurement. Nothing in this PR touches the parser.
Next step: safe to re-run; it clears once the base-branch build artifact is present.
2 & 3. Regression swarms (release x86_64 + aarch64) — not PR-caused
Both arches fail the identical 3 of 1535 scenarios (1518 ok, 14 skipped), all in the Altinity swarms (object_storage_cluster) regression suite:
node failure/cpu overload—Code: 209 … Timeout exceeded while reading from socket … receive timeout 300000 ms (SOCKET_TIMEOUT)on aSELECT … sleepEachRow(0.5) … SETTINGS object_storage_cluster='static_swarm_cluster'query, i.e. a deliberately slow query run while the test is injecting CPU overload onto the node.node failure/cpu overload all swarm nodes— sameSOCKET_TIMEOUTpattern.cluster discovery/check cluster discovery with wrong cluster name—Code: 279 … Cannot connect to any replica for query execution (ALL_CONNECTION_TRIES_FAILED); the test expected client exit code 33 but got 23.
Why this is not caused by the PR:
- This PR's only object-storage code change threads a
catalog_auth_tokenintoStorageObjectStorage::dropImpl → catalog->dropTable(...)— i.e. DROP TABLE on catalog-backed tables only. The failing scenarios areSELECT/s3Clusterreads and cluster discovery, which are not on the changed code path. - The failure modes are connectivity/timeouts (socket timeout under injected CPU stress; all-connection-tries-failed), not auth/credential errors. There is no
auth/token/access denied/403anywhere in either swarm report. - The suite's thousands of iceberg/
s3Clusterswarm join (816480) and union (500) scenarios — which are exactly what exercises object_storage_cluster worker credential forwarding — all passed. If this PR's forwarding had broken worker reads, those are the scenarios that would have gone red with auth errors. - The two
cpu overloadcases are explicit resource-stress tests; a 300 s socket timeout under deliberately starved CPU is a textbook timing/infra flake.
Caveat (honest): no other recent PR against antalya-26.8 ran the swarms suite, so I couldn't produce a base-branch run showing these exact 3 scenarios already red — the "not PR-caused" call rests on the code-path + error-mode analysis above, which is strong but not a direct A/B.
Next step: re-run the swarms regression. The two cpu overload timeouts should clear on re-run (stress-flaky). If check cluster discovery with wrong cluster name persists across re-runs, it's worth raising with the clickhouse-regression owners as a suite/environment issue rather than a change in this PR — the PR doesn't touch cluster discovery or connection logic.
PR health
The functional matrix is clean: all Builds (amd/arm debug/binary/release + wasm_parser), Fast test, Quick functional, all Stateless shards, all Integration shards, and Unit tests passed. The only red is 1 CI-infra miss and 3 environment-sensitive scenarios in an unrelated stress suite. No code change on this branch is indicated by the failures.
🤖 Generated with Claude Code
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Forward auth tokens to data lake catalogs (#2329 by @zvonand).
CI/CD Options
Exclude tests:
Regression jobs to run:
Cherry-picked from #2329.