[Bug] (storage policy) Cooled rowset reads fail with [E-206]get fs failed after BE restart in non-cloud mode: storage resource map not re-synced, self-heal gated by is_cloud_mode

[Olrookie]

Search before asking

• I had searched in the issues and found no similar issues.

Version

• Apache Doris 4.0.3 (community release, tag 4.0.3-rc03), non-cloud mode (1 FE + 1 BE on one node)
• Remote storage: Alibaba Cloud OSS (S3-compatible), registered via CREATE S3 RESOURCE + CREATE STORAGE POLICY with cooldown_ttl = 600s
• Affected table: AUTO PARTITION duplicate-key table; ETL runs INSERT OVERWRITE on its partitions every minute
• The relevant code is unchanged in 4.0.4–4.0.8 / branch-4.0 (verified at source level)

What's Wrong?

Summary

In non-cloud mode, after a BE restart, queries that need to open segments of rowsets already cooled down to S3/OSS fail and keep failing with [E-206] get fs failed: the BE-side in-memory storage-resource map is empty after restart and is never re-populated, and the read-path self-heal is gated by config::is_cloud_mode().

Root cause chain (verified in 4.0.3-rc03 source)

1. BE keeps storage resources only in an in-memory global map (s_storage_resource_mgr in be/src/olap/storage_policy.cpp). It is not persisted.
2. The map is populated only by the PUSH_STORAGE_POLICY agent task (push_storage_policy_callback in be/src/agent/task_worker_pool.cpp → update_s3_resource() → put_storage_resource()).
3. After a BE restart the map is empty. FE does not re-push storage policies/resources to a restarting BE (the tablet report already carries the BE's known resource ids/versions, but no re-push is triggered).
4. When a query opens a cooled segment, RowsetMeta::remote_storage_resource() (be/src/olap/rowset/rowset_meta.cpp:167) misses the map; its self-heal (sync_storage_vault()) is gated by config::is_cloud_mode() (line 178), so in non-cloud mode it returns InternalError("cannot find storage resource. resource_id={}") (line 188) without any re-sync attempt.
5. RowsetMeta::physical_fs() (line 113) logs that error and returns nullptr; BetaRowset::load_segment (be/src/olap/rowset/beta_rowset.cpp:190) then fails the query with the bare [E-206] get fs failed.
6. Nothing heals it: the query keeps failing on every attempt until something else re-populates the map (a new cooldown triggering a policy push, or a partition drop/rebuild).

Production evidence (observational — we did not run a controlled restart experiment)

Single-node 4.0.3 cluster, one S3-compatible (OSS) resource; minute-level INSERT OVERWRITE ETL on local (non-cooled) partitions kept working throughout everything below.

• 2026-09-04 15:37 BE restart (upgrade; our be.WARNING window opens 09-07 12:12, already inside the resulting error storm): queries against partitions cooled to OSS failing with [E-206] get fs failed (~92K warnings), which only stopped when those partitions were dropped/recreated (09-07 15:55).
• 09-07 15:50 the policy was attached to two other partitions and they cooled down successfully — a successful cooldown needs the resource on the BE, so a push must have re-populated the resource map between the restart and that cooldown.
• 2026-09-10 10:25 BE restart. The first warning after BE came back (10:25:46.927) is cannot find storage resource. resource_id=... (twice), then >150K [E-206] warnings within ~5 hours from 6,720 distinct queries, until the BE process was stopped again at 15:22. Between the two storms (multi-day window) there is not a single such warning.

Observability traps

• SELECT count(*) on the affected table succeeds (it only reads rowset meta), so health checks look fine.
• Point / LIMIT queries that stop early can succeed via the segment cache, so the failure looks intermittent.
• Deterministic verification needs a real full scan of the cooled tablet (e.g. ORDER BY on a non-key column).

What You Expected?

Cooled (remote) rowsets should remain readable after a BE restart in non-cloud mode. Concretely, any of:

1. FE re-pushes storage policies/resources to a BE when it registers/reports after a restart (the report already carries resource ids/versions);
2. or the read path self-heals on a map miss in non-cloud mode too — remove the config::is_cloud_mode() gate in RowsetMeta::remote_storage_resource() and re-fetch the resource from FE;
3. or BE persists / rebuilds the resource map at startup.

Independently, the failure should be actionable instead of [E-206] get fs failed: BetaRowset::load_segment should include resource_id in the message (nearby code already does this, e.g. beta_rowset.cpp:100/157), and the underlying cannot find storage resource root cause should not stay hidden.

How to Reproduce?

We have NOT run a controlled reproduction. The steps below are derived from the 4.0.3 source analysis above and are consistent with what we observed in production, but they are untested — treat them as a suggested path; corrections welcome.

1. Non-cloud cluster, 1 FE + 1 BE.
2. CREATE S3 RESOURCE (any S3-compatible storage), then CREATE STORAGE POLICY ... PROPERTIES("cooldown_ttl"="60s").
3. Create a table with the policy attached, load data once, wait for rowsets to cool down to S3; verify a full scan works (e.g. SELECT ... ORDER BY <non-key column>).
4. Restart the BE, and do NOT load new data before step 5 — anything that makes FE push the policy again re-populates the BE map and closes the failure window.
5. Run the same full-scan query; per the code it should fail with [E-206] get fs failed, and be.WARNING should log cannot find storage resource. resource_id=... on every attempt.

Why reproduction may look non-deterministic (matches our production experience):

• Only queries that actually open cooled segments fail: SELECT count(*) reads meta only and always succeeds; LIMIT-style queries can succeed via early termination + segment cache.
• The failure window is "from BE restart until the next policy re-push"; after a re-push every query succeeds again.

Anything Else?

• Related open issue: [Bug] [E-206]get fs failed, resource_id= #63669 hits the same get fs failed family on a different path (clone). Our case is the plain read path after restart in non-cloud mode.
• Environment details: single node, S3-compatible object storage; storage policy attached to AUTO PARTITION table partitions; heavy minute-level INSERT OVERWRITE workload (writes unaffected).
• We also observed a separate [E-3115] version not continuous for mow problem on a mow table the same day; I can file it separately once I have a minimal repro.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial maintainer triage (source-checked against 4.0.3-rc03 and branch-4.0; no controlled reproduction was run):

Assessment

There is a real restart-time availability risk in this area, and the loss of the underlying error in BetaRowset::load_segment() is confirmed. However, the multi-hour failure is not yet explained by the proposed root-cause chain, because one important claim is contradicted by the same-version FE/BE code.

Verified facts

• In non-cloud mode, s_storage_resource_mgr is process-local and is not persisted. Among production non-cloud paths, push_storage_policy_callback() is what constructs the S3/HDFS filesystem and calls put_storage_resource().
• A remote-rowset read can therefore miss this map after restart. RowsetMeta::remote_storage_resource() has no on-demand local-mode lookup; physical_fs() logs the detailed status and returns null; BetaRowset::load_segment() then replaces it with the generic get fs failed. Adding resource_id would improve the symptom, although propagating the original status would be better.
• The cloud-mode guard itself is not evidence of a missing local recovery path. Its body calls storage_engine().to_cloud().sync_storage_vault(), while to_cloud() checks that the engine type is CLOUD. Simply removing the guard would make the local path hit that check; local mode needs a distinct FE-backed mechanism.

Contradiction that must be resolved

report_tablet_callback() explicitly reports the BE's in-memory storage-policy and resource ID/version lists, including empty lists after a restart. ReportHandler.storagePolicyReport() compares those lists with FE metadata, and diffPolicy() / diffResource() add missing entries to a PushStoragePolicyTask. The default tablet-report interval is 60 seconds (plus up to 5 seconds of random delay), and a changed/initial master heartbeat can also wake report workers immediately.

Consequently, the code already intends to repopulate the map after a BE restart. A brief failure before the first successful report/push is plausible from source inspection, but failures persisting for hours require another failure in that pipeline (or a different condition) and cannot currently be attributed only to is_cloud_mode().

The production timeline also needs reconciliation: _cooldown_data() obtains the filesystem through get_resource_by_storage_policy_id(). If the two partitions successfully cooled through the same resource while older-rowset reads still failed, the global map existed at that time and subsequent reads should have found it. Exact completion timestamps and resource IDs could show whether the old rowsets referenced a different/stale resource ID or whether the successful cooldown occurred later than the policy attachment.

Information needed

1. A controlled run with a full scan immediately before restart, immediately after restart, and again after at least two configured tablet-report intervals, with no intervening DDL/load/policy change. Please include the effective report_tablet_interval_seconds and report_random_wait values.
2. FE logs from the first post-restart heartbeat through those report intervals, especially lines containing backend[...] reports policies, after diff policy, and any agent-task submission/RPC error.
3. BE INFO/WARNING logs for the same window, including failed to report tablet, successfully submit task for PUSH_STORAGE_POLICY, and successfully update s3 resource or update s3 resource failed.
4. The exact resource_id from the read failure and the current FE resource ID/version, plus confirmation that the resource was not dropped/recreated. Please redact credentials from resource properties.
5. For the September 7 sequence, timestamps for the actual cooldown upload completion and the last failing read, not only the policy-attachment time.

Recommended next steps

• First locate the break across BE report -> FE diff -> task submission -> BE resource construction using the logs above, then fix that specific stage and add a restart regression/integration test.
• Independently harden startup so remote reads do not race the initial local-mode resource synchronization; an eager post-heartbeat push/rebuild is consistent with the existing architecture. Do not reuse the cloud vault sync by only removing the mode check.
• Independently preserve the remote_storage_resource() failure through the read stack (or at minimum include resource_id, tablet ID, and rowset ID in load_segment()'s error).

Current triage: credible bug / startup race and observability defect confirmed; claimed permanent non-cloud re-sync absence and the multi-hour root cause remain unverified pending the report/push evidence above. The issue currently has no labels; a bug plus storage-policy/cooldown area label would be appropriate under the repository's label taxonomy.

Breakwater-GitHub-Analysis-Slot: slot_680b283fba8a