Skip to content

branch-4.1: [improvement](fe) Reduce cloud version sync config (#66296) - #67929

Open
mymeiyi wants to merge 1 commit into
apache:branch-4.1from
mymeiyi:branch-4.1-pick-66296
Open

branch-4.1: [improvement](fe) Reduce cloud version sync config (#66296)#67929
mymeiyi wants to merge 1 commit into
apache:branch-4.1from
mymeiyi:branch-4.1-pick-66296

Conversation

@mymeiyi

@mymeiyi mymeiyi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

pick #66296

1. Cloud table and partition version synchronization ran every 20
seconds, batched up to 2000 version reads, and reused the global Meta
Service retry limit of 200, which could create high concurrent FDB read
pressure and amplify failed requests. Increase the sync interval to 60
seconds, reduce the batch size to 200, and limit background get-version
tasks to 3 attempts without changing other callers.
2. Skip the daemon when both global/default cache TTLs are finite while
retaining proactive refresh whenever either cache never expires.
@mymeiyi
mymeiyi requested a review from yiguolei as a code owner September 14, 2026 03:27
Copilot AI lite review requested due to automatic review settings September 14, 2026 03:27
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi

mymeiyi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Moderate findings remain in compatibility-mode retry handling and partition-test mocking.

Pull request overview

This PR reduces FE cloud version-sync load while preserving cache refresh behavior and configurable retries.

Changes:

  • Increases sync interval to 60 seconds and reduces batch size to 200.
  • Adds configurable retry limits for background synchronization.
  • Uses default cache TTLs without a ConnectContext and updates related tests.

Review findings:

  • Moderate (1 vote): Both daemon retry paths can repeat guaranteed failures in compatibility mode; use the effective retry count or skip the daemon.
  • Moderate (1 vote): The partition test must mock both VersionHelper overloads to prevent a real RPC.
  • Moderate (1 vote): The same compatibility-mode retry issue affects partition synchronization.
File summaries
File Summary
fe/fe-core/src/test/java/org/apache/doris/cloud/rpc/VersionHelperTest.java Tests retry limits and terminal handling.
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudPartitionTest.java Tests default TTL behavior and cache handling.
fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java Updates mocks for retry overloads.
fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/VersionHelper.java Supports configurable retry attempts.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java Applies reduced sync frequency, batching, TTL checks, and retries.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java Propagates retries and uses default TTLs.
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java Propagates configurable table-version retries.
fe/fe-common/src/main/java/org/apache/doris/common/Config.java Updates sync defaults and adds retry configuration.
Review details

Suppressed comments (3)

fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java:133

  • In compatibility mode, Config.metaServiceRpcRetryTimes() intentionally resolves to 1, while MetaServiceProxy.getProxy() rejects every RPC. Passing the new raw value here therefore makes each table-sync task repeat a guaranteed failure up to 3 times instead of preserving the effective compatibility-mode limit, adding avoidable delay and logs. Cap this value by the effective retry count (or skip the daemon in compatibility mode).
                        dbIds, tableIds, Config.cloud_version_syncer_get_version_retry_times);

fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java:203

  • In compatibility mode, Config.metaServiceRpcRetryTimes() intentionally resolves to 1, while MetaServiceProxy.getProxy() rejects every RPC. Passing the new raw value here therefore makes each partition-sync task repeat a guaranteed failure up to 3 times instead of preserving the effective compatibility-mode limit, adding avoidable delay and logs. Cap this value by the effective retry count (or skip the daemon in compatibility mode).
                        partitions, false, Config.cloud_version_syncer_get_version_retry_times);

fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudPartitionTest.java:127

  • This mock now covers only the new two-argument overload, but CloudPartition.getVisibleVersionFromMs still calls the one-argument VersionHelper.getVersionFromMeta(req) for part.getVisibleVersion(). That call will escape the mock and execute the real Meta Service RPC, so the test no longer follows its mocked callCount path. Keep mocks for both overloads (the one-argument form for single-part reads and the two-argument form for batch reads).
            public Cloud.GetVersionResponse getVersionFromMeta(Cloud.GetVersionRequest req, int maxAttempts) {
  • Files reviewed: 8/8 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 47.83% (11/23) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants