[Bug] [Config] Inconsistent defaults for storage flood stage thresholds between FE and BE

[simges]

Search before asking

• I had searched in the issues and found no similar issues.

Version

v3.1.4

What's Wrong?

[Config] Inconsistent defaults for storage flood stage thresholds between FE and BE

Description

Apache Doris defines storage_flood_stage_usage_percent and storage_flood_stage_left_capacity_bytes in both FE and BE configuration. The two components use the same parameter names, but the default value of storage_flood_stage_usage_percent is inconsistent.

According to the official documentation:

Parameter | FE default | BE default -- | -- | -- storage_flood_stage_usage_percent | 95 | 90 storage_flood_stage_left_capacity_bytes | 1 GiB | 1 GiB

References:

• FE Config.java

• FE configuration documentation

• BE configuration documentation

Problem

### Conflicting Configuration Defaults Between FE and BE

The same configuration parameters have conflicting defaults in FE and BE, which can mislead users when configuring a Doris cluster.

For example, at 92% disk usage:

• FE: Stays normal (default threshold is 95%).
• BE: Triggers flood-stage protection and returns errors (default threshold is 90%).

Impact

This mismatch causes unexpected query failures. It makes disk protection behavior unpredictable and complicates both cluster configuration and troubleshooting.

What You Expected?

The default values and descriptions of the corresponding FE and BE parameters should be consistent, or the documentation should clearly explain why they intentionally differ.

How to Reproduce?

• set config::storage_flood_stage_usage_percent 92% as hard-coded.
• set storage_flood_stage_left_capacity_bytes to a high value enough for satisy this condition in data_dir.cpp:

bool DataDir::reach_capacity_limit(int64_t incoming_data_size) {
    double used_pct = get_usage(incoming_data_size);
    int64_t left_bytes = _available_bytes - incoming_data_size;
    if (used_pct >= config::storage_flood_stage_usage_percent / 100.0 &&
        left_bytes <= config::storage_flood_stage_left_capacity_bytes) {
        LOG(WARNING) << "reach capacity limit. used pct: " << used_pct
                     << ", left bytes: " << left_bytes << ", path: " << _path;
        return true;
    }
    return false;
}

• then the backend will return an error. It is still treated as a query execution candidate, though, since exceedLimit continues to return false:

    public boolean exceedLimit(boolean floodStage) {
        if (LOG.isDebugEnabled()) {
            LOG.debug("flood stage: {}, diskAvailableCapacityB: {}, totalCapacityB: {}",
                    floodStage, diskAvailableCapacityB, totalCapacityB);
        }
        if (floodStage) {
            return diskAvailableCapacityB < Config.storage_flood_stage_left_capacity_bytes
                && this.getUsedPct() > (Config.storage_flood_stage_usage_percent / 100.0);
        } else {
            return diskAvailableCapacityB < Config.storage_min_left_capacity_bytes
                || this.getUsedPct() > (Config.storage_high_watermark_usage_percent / 100.0);
        }
    }

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_1dc63241fb95

Initial assessment

Disposition: confirmed configuration inconsistency; the load/write scheduling gap is supported by the code, but the broader claim that this causes general query failures is not yet demonstrated. No labels are currently applied to the issue.

Verified facts

• The local mirror has 3.1.4-rc01 and 3.1.4-rc02 rather than a plain v3.1.4 tag. Both release-candidate tags, current branch-3.1, and current master define the FE default as 95 and the BE default as 90.
• In 3.1.4-rc02, FE defines storage_flood_stage_usage_percent = 95. BE defines the corresponding default as 90. Both byte thresholds default to 1 GiB.
• FE's DiskInfo.exceedLimit(true) requires both available < 1 GiB and used > 95%. BE's DataDir::reach_capacity_limit requires both available-after-write <= 1 GiB and used-after-write >= 90%.
• FE uses its flood-stage check when preparing OLAP load destinations in OlapTableSink, while BE independently enforces its local threshold on disk-consuming operations. Therefore, with effective defaults and a path at 92% usage and at most about 1 GiB free, FE can admit a load that BE later rejects. Usage of 92% alone is not sufficient because both components use a two-part capacity condition.
• The equality boundaries also differ (>/< in FE versus >=/<= in BE), so merely making the numeric defaults equal would still leave exact-threshold behavior inconsistent.
• Backend.isQueryAvailable() checks liveness and administrative query-disable/shutdown flags, not disk capacity. Thus the observation that the BE can remain a query candidate is correct. However, reach_capacity_limit() is a per-operation guard, not a persistent global flood-state transition. Ordinary read queries do not inherently require new local data files; failures are expected only on disk-consuming paths such as load/segment finalization, migration/clone/restore, or a query that actually spills to that path.

History and likely scope

The original disk-flood-stage implementation introduced FE and BE defaults together at 95 in commit 7e981b2. BE alone changed from 95 to 90 in commit 206a711; that same commit adjusted trash-sweep guard space, but neither the configuration comment nor tests explain an intended FE/BE policy split. This supports treating the mismatch as a valid maintenance issue, while the reason for the 2021 BE change remains unverified.

Information still needed

To establish the claimed user-visible failure rather than only the code-level mismatch, please provide:

1. The exact FE and BE build strings or Git commit hashes for the reported v3.1.4 deployment.
2. The effective runtime values of both flood-stage settings on FE and on every relevant BE; they are mutable and may differ from compiled defaults.
3. A black-box reproduction using unchanged binaries, including the exact SQL/load/restore operation. The current reproduction proposes hard-coding a value and artificially enlarging the byte threshold, so it does not establish what happens with shipped defaults.
4. The affected data-path total/available capacity and path hash at the failure time, plus the corresponding BE DISK_REACH_CAPACITY_LIMIT message and FE/coordinator error. If the failed operation was a read query, include whether spilling was enabled/observed and its query profile.

Recommended next steps

1. Confirm whether BE's 90% value is an intentional safety floor. If it is, make FE's load/restore preflight at least as strict (or propagate the BE's effective threshold to FE) rather than relaxing BE to 95 without a separate disk-safety assessment.
2. If independent component policies are intentional, document their distinct roles explicitly and avoid presenting the same names as equivalent cluster-wide settings. FE should still avoid scheduling writes that BE will predictably reject from the same reported disk state.
3. Align the comparison boundaries as part of any fix and add FE/BE boundary tests covering just below, exactly at, and just above both percentage and remaining-byte thresholds. Add an integration test showing that FE preflight and BE enforcement agree for a load.
4. Do not make general query eligibility depend on disk flood stage as part of this issue without a separate design decision; that could unnecessarily remove readable replicas. Limit the fix to admission of disk-consuming operations unless a reproducible non-spill read-query failure shows otherwise.

[Asthenia0412]

Hi, I've submitted a fix in #67944 that aligns the FE default of storage_flood_stage_usage_percent to 90, consistent with the BE config. This prevents FE from scheduling load/restore jobs onto a BE whose disk is already in flood stage. Please take a look when you have time, thanks!