branch-4.1: [improvement](fe) Reduce cloud version sync config (#66296) - #67929
branch-4.1: [improvement](fe) Reduce cloud version sync config (#66296)#67929mymeiyi wants to merge 1 commit into
Conversation
1. Cloud table and partition version synchronization ran every 20 seconds, batched up to 2000 version reads, and reused the global Meta Service retry limit of 200, which could create high concurrent FDB read pressure and amplify failed requests. Increase the sync interval to 60 seconds, reduce the batch size to 200, and limit background get-version tasks to 3 attempts without changing other callers. 2. Skip the daemon when both global/default cache TTLs are finite while retaining proactive refresh whenever either cache never expires.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
There was a problem hiding this comment.
🔵 Needs a closer look
Moderate findings remain in compatibility-mode retry handling and partition-test mocking.
Pull request overview
This PR reduces FE cloud version-sync load while preserving cache refresh behavior and configurable retries.
Changes:
- Increases sync interval to 60 seconds and reduces batch size to 200.
- Adds configurable retry limits for background synchronization.
- Uses default cache TTLs without a
ConnectContextand updates related tests.
Review findings:
- Moderate (1 vote): Both daemon retry paths can repeat guaranteed failures in compatibility mode; use the effective retry count or skip the daemon.
- Moderate (1 vote): The partition test must mock both
VersionHelperoverloads to prevent a real RPC. - Moderate (1 vote): The same compatibility-mode retry issue affects partition synchronization.
File summaries
| File | Summary |
|---|---|
fe/fe-core/src/test/java/org/apache/doris/cloud/rpc/VersionHelperTest.java |
Tests retry limits and terminal handling. |
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudPartitionTest.java |
Tests default TTL behavior and cache handling. |
fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java |
Updates mocks for retry overloads. |
fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/VersionHelper.java |
Supports configurable retry attempts. |
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java |
Applies reduced sync frequency, batching, TTL checks, and retries. |
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java |
Propagates retries and uses default TTLs. |
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java |
Propagates configurable table-version retries. |
fe/fe-common/src/main/java/org/apache/doris/common/Config.java |
Updates sync defaults and adds retry configuration. |
Review details
Suppressed comments (3)
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java:133
- In compatibility mode,
Config.metaServiceRpcRetryTimes()intentionally resolves to 1, whileMetaServiceProxy.getProxy()rejects every RPC. Passing the new raw value here therefore makes each table-sync task repeat a guaranteed failure up to 3 times instead of preserving the effective compatibility-mode limit, adding avoidable delay and logs. Cap this value by the effective retry count (or skip the daemon in compatibility mode).
dbIds, tableIds, Config.cloud_version_syncer_get_version_retry_times);
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java:203
- In compatibility mode,
Config.metaServiceRpcRetryTimes()intentionally resolves to 1, whileMetaServiceProxy.getProxy()rejects every RPC. Passing the new raw value here therefore makes each partition-sync task repeat a guaranteed failure up to 3 times instead of preserving the effective compatibility-mode limit, adding avoidable delay and logs. Cap this value by the effective retry count (or skip the daemon in compatibility mode).
partitions, false, Config.cloud_version_syncer_get_version_retry_times);
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudPartitionTest.java:127
- This mock now covers only the new two-argument overload, but
CloudPartition.getVisibleVersionFromMsstill calls the one-argumentVersionHelper.getVersionFromMeta(req)forpart.getVisibleVersion(). That call will escape the mock and execute the real Meta Service RPC, so the test no longer follows its mockedcallCountpath. Keep mocks for both overloads (the one-argument form for single-part reads and the two-argument form for batch reads).
public Cloud.GetVersionResponse getVersionFromMeta(Cloud.GetVersionRequest req, int maxAttempts) {
- Files reviewed: 8/8 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
FE Regression Coverage ReportIncrement line coverage |
pick #66296