[Bug] BE `rs_normal` scanner threads leak on every external catalog query; `doris_max_remote_scanner_thread_pool_thread_num` has no effect

[wangmingzhou1986]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris-4.1.3-rc02(AVX2) RELEASE
features: -TDE,-VARIANT_NESTED_GROUP,-HDFS_STORAGE_VAULT,+UI,+AZURE_BLOB,+AZURE_STORAGE_VAULT,-HIVE_UDF,+BE_JAVA_EXTENSIONS
build: git://vm-122@7126cf65d96ebc43fce0906f51e92c1a2ccf24a6

What's Wrong?

Every query against an external catalog (Paimon on S3/MinIO, via CREATE CATALOG ... 'type'='paimon')
leaks roughly 10 threads per BE, named rs_normal. The threads are never reclaimed.
Thread count therefore grows linearly with the number of external-table queries,
and because jemalloc's per-thread tcache scales with thread count,
BE rss grows with it until it hits mem_limit and every query fails with
MEM_LIMIT_EXCEEDED — while SHOW BACKENDS still reports Alive: true.

Setting doris_max_remote_scanner_thread_pool_thread_num to a finite value
(we tried 256, down from the default -1) and restarting BE does not bound this pool:
the count climbed past 3,192 and kept going.

Evidence

1. Only rs_normal grows

Thread-name histogram from /proc/<be_pid>/task/*/comm:

thread name	be-0 (uptime 17.7 h)	be2-0 (uptime 22 h)
rs_normal	24,377	35,332
brpc_arrow_flig	512	512
doris_be	352	344
brpc_light / brpc_heavy / EvHttpServer	128 / 128 / 128	128 / 128 / 128
SendBatchThreadP / DownloadThreadP	64 / 64	64 / 64
ls_normal	48	48
p_normal_blocki / TabletPublishTx / SegmentPrefetch	32 / 32 / 32	32 / 32 / 32
everything else, summed	~1,779	~1,771

Every other pool is identical between the two BEs and stays flat. Only rs_normal diverges,
and it diverges in proportion to uptime (i.e. to accumulated query count).

2. Controlled experiment — growth is caused by external-catalog queries, and is not reclaimed

Both BEs observed simultaneously; rs_normal counted before and after each phase:

phase	be-0 Δ	be2-0 Δ
idle 60 s (zero queries)	-1	0
20 × SELECT COUNT(*) FROM <paimon_catalog>.<db>.<tbl>	+203	+201
idle 120 s	-4	-1

≈ 10 threads per query per BE. Queries fan out to all BEs, so both grow together.
Idle neither grows nor reclaims.

3. The config knob does not bound it

From /api/show_config (all three are mutable=false):

doris_remote_scanner_thread_pool_thread_num       = 48
doris_max_remote_scanner_thread_pool_thread_num   = -1     <- default in this build
doris_remote_scanner_thread_pool_queue_size       = 102400

Note the docs state the default for doris_max_remote_scanner_thread_pool_thread_num
is 512, but this build reports -1.

We set it explicitly to 256 in be.conf and restarted the BE. Verified it took effect:

doris_max_remote_scanner_thread_pool_thread_num = 256

rs_normal nevertheless climbed past 3,192 and kept growing at the same rate.
So whatever creates rs_normal threads is not governed by this parameter.

Ruled out: workload groups are not the cause. We have exactly one workload group,
and enable_workload_group_for_scan = false.

4. Memory consequence

jemalloc stats at rss 6.81 GB (be2-0, 35 k threads):

metric	be-0	be2-0
jemalloc_allocated_bytes	2.49 GB	4.02 GB
jemalloc_tcache_bytes	1.09 GB	1.51 GB
jemalloc_metadata_bytes	0.77 GB	1.08 GB
jemalloc_retained_bytes	1.12 GB	1.52 GB

35 k threads × ~40 KB tcache ≈ 1.5 GB, matching the measured tcache almost exactly.
tcache + metadata = 2.59 GB = 38 % of rss.

Eventually:

[MEM_LIMIT_EXCEEDED] ... process memory used 6.62 GB(= 6.62 GB[vm/rss]),
limit 7.00 GB, soft limit 6.30 GB, sys available memory 398 MB

At that point every query through the BE fails, including SELECT ... LIMIT 1,
yet SHOW BACKENDS still shows Alive: true — so health checks based on liveness
do not detect it.

Restarting BE clears it completely, confirming the threads (not data) hold the memory:

	before restart	after restart
be2-0 rss / threads	6,805 MB / 37,103	971 MB / 1,778
be-0 rss / threads	4,514 MB / 26,579	1,361 MB / 1,933

5. Environment note (may or may not be relevant)

BE runs in a container. nproc inside the container reports 16 (the host's core count),
while the cgroup CPU limit is 2 (be2-0) / 6 (be-0).

JEMALLOC_CONF=percpu_arena:percpu,background_thread:true,metadata_thp:auto,
muzzy_decay_ms:5000,dirty_decay_ms:5000,oversize_threshold:0,prof:true,
prof_active:false,lg_prof_interval:-1,lg_extent_max_active_fit:8

dirty_decay_ms/muzzy_decay_ms are already aggressive (5 s), so this is not jemalloc
withholding freed pages — the memory is genuinely attached to live threads.

What You Expected?

rs_normal threads are returned to (or bounded by) a pool after the scan finishes,
so that BE thread count and rss stay flat under a steady stream of external-catalog
queries, and doris_max_remote_scanner_thread_pool_thread_num actually caps the pool.

How to Reproduce?

1. Create an external catalog (we used Paimon on S3; a Hive/Iceberg catalog on object
   storage will likely do as well).
2. Record the baseline: for t in /proc/<be_pid>/task/*/comm; do cat $t; done | grep -c '^rs_normal'
3. Run N (e.g. 20) trivial queries against a table in that catalog,
   e.g. SELECT COUNT(*) FROM <catalog>.<db>.<tbl>;
4. Re-count rs_normal. Expect ≈ 10 × N more threads per BE.
5. Wait a few minutes with no queries and count again — the threads are not reclaimed.

Anything Else?

• Happy to provide the full thread-name histograms, /api/show_config dumps,
  /metrics jemalloc series, or a time series of rss / thread count
  (we sample both every 5 minutes).
• If rs_normal is expected to be bounded by a different config than
  doris_max_remote_scanner_thread_pool_thread_num, please point us at it and
  we'll re-test and report back.
• The discrepancy between the documented default (512) and this build's default (-1)
  may itself be worth a look.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Initial triage: the evidence is internally consistent and this should be treated as a high-severity availability/resource-exhaustion problem. It is also a near-exact duplicate of #65416, which reports the same unbounded rs_normal growth for Paimon-on-S3 on 4.1.1-rc01, including ineffective BE and workload-group caps. This report adds a much better controlled reproducer, so the two issues should be linked and investigated together.

What is verified from the reported revision

I checked the exact build commit, 7126cf65d96ebc43fce0906f51e92c1a2ccf24a6:

• rs_normal is the remote scan scheduler owned by the normal workload group (construction path). Having only one workload group narrows the problem, but does not bypass this workload-group-owned scheduler.
• The raw default of doris_max_remote_scanner_thread_pool_thread_num is indeed -1 (config), but -1 is an automatic-sizing sentinel, not an unlimited value: it becomes max(512, detected_cores * 10) (calculation). Thus /api/show_config = -1 and the documented effective floor of 512 are compatible, although the documentation should explain the sentinel.
• With the shown local size of 48, an explicit BE value of 256 should yield a 256 maximum unless the normal workload group supplies a positive max_remote_scan_thread_num, which overrides the BE fallback (override path).
• The task executor only creates a worker when num_threads + pending_start < max_threads (cap check), and idle workers above the minimum are supposed to retire after 500 ms (retirement path). A single correctly accounted pool therefore cannot intentionally contain 3,192 or 35,000 live workers.

This confirms that the observed behavior violates the intended cap/lifecycle. It does not yet identify which invariant is failing. The remaining possibilities include scheduler accounting/teardown, multiple same-named scheduler owners, or non-Doris/JVM-side threads retaining or inheriting the native name. The thread-name histogram alone cannot distinguish them, so I would not assign a Paimon or scheduler root cause yet.

Highest-value missing evidence

Please collect the following on a freshly restarted BE, immediately before and after the 20-query reproducer:

1. SHOW WORKLOAD GROUPS output for normal, especially max_remote_scan_thread_num and min_remote_scan_thread_num.
2. The BE INFO lines containing [topic_publish_wg] ... thread info=. This revision already logs remote-pool accounting as r_scan num:<accounted>, real_num:<thread-set-size>, min_num:<...>, max_num:<...> (format). The before/after values are the most direct test of whether the executor itself sees the leaked threads.
3. /threadz and /threadz?group=thread%20pool before/after. Comparing Doris-registered thread counts with /proc/<pid>/task will show whether all rs_normal TIDs were created by Doris's thread wrapper.
4. For 5 newly created and 5 old rs_normal TIDs: /proc/<pid>/task/<tid>/status, wchan, and stack (if permitted), plus native backtraces. Also provide jcmd <pid> Thread.print -l or jstack <pid> and map those decimal TIDs to Java nid=0x.... This will establish whether they are idle in the scheduler, blocked in JNI/Paimon/Hadoop/S3 code, or absent from the JVM.
5. The Paimon table type (append-only vs primary-key), file format (ORC/Parquet), bucket mode, approximate split count for the test query, and whether every query completed normally or any were cancelled/timed out.

Suggested next steps

• Link/consolidate with BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes #65416 and route jointly to BE execution/workload scheduling and Lakehouse/Paimon maintainers. No labels are currently set; kind/bug and area/catalog are appropriate, plus the repository's high-severity/availability label if applicable.
• Use r_scan num/real_num/max_num plus representative stacks to choose the owner before changing code. If real_num stays at or below 256 while /proc grows, investigate JNI/plugin-created threads or native-name inheritance. If real_num itself exceeds max_num, the task-executor accounting invariant is directly broken. If several pool instances appear, trace workload-group scheduler creation/destruction.
• Run one staging A/B on the same table with SET enable_paimon_cpp_reader=true if that reader supports this table. Growth only on the default JNI path would isolate Paimon/JNI lifecycle from the common remote scheduler; growth on both paths would point back toward scheduler ownership/accounting.
• The reported build predates Paimon observability and cleanup work in [improvement](paimon) Add Paimon JNI scanner observability #65354, [opt](paimon) control paimon jni reader async file reader #65365, and [fix](be) Harden JNI reader split lifecycle #65503. A repeat of the exact 20-query test on 4.1.4-rc04 is useful because that source contains the relevant async-reader metrics/control and iterator/JNI cleanup changes. This is a diagnostic upgrade test, not a claim that the issue is already fixed.
• Until ownership is confirmed, rolling BE restarts before thread/RSS exhaustion are the only demonstrated mitigation here. The 256 cap and allocator tuning should not be presented as fixes for the observed behavior.

Breakwater-GitHub-Analysis-Slot: slot_52e24c6dcbe1

[wangmingzhou1986]

Thanks for the very precise triage — it pointed straight at the right discriminator. I collected the requested evidence, and it resolves the ambiguity: the leaking threads are not Doris scheduler threads at all.

TL;DR

The rs_normal name in /proc/<pid>/task/*/comm is inherited comm, not a Doris thread. In jstack those same TIDs are AWS SDK v2 sdk-ScheduledExecutor-<n>-<m> threads. Doris's remote scan pool is correctly bounded the whole time (r_scan real_num:8, cap 256). What actually leaks is AWS SDK async clients created per scan and never closed — roughly 2 SDK ScheduledExecutor instances (10 threads) per query per BE.

So your first branch is the right one:

If real_num stays at or below 256 while /proc grows, investigate JNI/plugin-created threads or native-name inheritance.

Both conditions hold simultaneously.

Build: doris-4.1.3-rc02-7126cf65d96 (same as reported). JVM: OpenJDK 17.0.2+8-86.

---

1. SHOW WORKLOAD GROUPS for normal

Name   scan_thread_num  max_remote_scan_thread_num  min_remote_scan_thread_num
normal      -1                    -1                          -1

All three are -1 at the workload-group level, i.e. the group does not override the BE fallback. Consistent with your reading of the override path.

2. [topic_publish_wg] ... thread info= — the decisive measurement

Identical before and after the 20-query reproducer, on both BEs:

thread info=scheduler: p_normal_simple_scheduler, info:
  [exec num:16, real_num:16, min_num:16, max_num:16],
  [l_scan num:48, real_num:48, min_num:48, max_num:48],
  [r_scan num:8, real_num:8, min_num:8, max_num:256],
  [mem_tab_flush num:8, real_num:8, min_num:8, max_num:64]

r_scan real_num never moves off 8, while /proc grows by hundreds. The executor's accounting is not broken, and the explicit BE value of 256 is in effect as max_num — it simply does not govern the threads that are actually leaking.

This also corrects the framing of my original report: it is not that doris_max_remote_scanner_thread_pool_thread_num "has no effect". It bounds the remote scan scheduler correctly; it has no effect on this leak because the leaked threads were never in that pool. The issue title is misleading and should probably be amended.

3. Controlled before/after (freshly restarted BE)

Reproducer: 20 × SELECT COUNT(*) FROM paimon_lake.store.sys_user. All 20 completed normally (0 cancelled, 0 timed out).

doris-be-0 — restarted immediately before BEFORE:

metric	BEFORE	AFTER	Δ
/proc/<pid>/task total	1,783	1,993	+210
comm == rs_normal	8	217	+209
JVM threads (jstack)	27	244	+217
sdk-ScheduledExecutor threads	0	200	+200
distinct sdk-ScheduledExecutor-<n> instances	0	40	+40
VmRSS	1,008 MB	1,236 MB	+228 MB
r_scan real_num (BE self-report)	8	8	0
/threadz rows	16	16	0

doris-be2-0 — not restarted, ~15 h uptime, same workload:

metric	BEFORE	AFTER	Δ
/proc/<pid>/task total	39,148	39,350	+202
distinct SDK instances	7,473	7,513	+40
r_scan real_num	8	8	0

Rate: +2 SDK executor instances = +10 threads, per query, per BE. Identical on a clean BE and on one that already holds 37 k threads — the leak rate does not decay.

/threadz and /threadz?group=thread pool are unchanged across the run, i.e. Doris-registered pools are stable while /proc grows.

4. Thread identity — this is where it becomes unambiguous

On the 15 h BE (37,537 threads total, 35,757 with comm == rs_normal):

jstack thread-name histogram (top):
  35745  sdk-ScheduledExecutor-<n>-<m>
     10  Thread-<n>
      6  GC Thread#
      6  G1 Refine#
      1  org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner
      1  Timer for 's3a-file-system' metrics system
      1  idle-connection-reaper
      ... (VM/Service/Sweeper/Finalizer, 1 each)

jstack total JVM threads : 35,789
distinct sdk-ScheduledExecutor instances : 7,149  (ids 0..7148, ~5 threads each)
occurrences of "rs_normal" / "scan" / "Scanner" in jstack : 0

Mapping the decimal TIDs to nid=0x… as you asked — 5 lowest and 5 highest rs_normal TIDs, all ten are JVM threads:

TID 10000 (0x2710) -> "sdk-ScheduledExecutor-1096-3" #6135 daemon
TID 10001 (0x2711) -> "sdk-ScheduledExecutor-1100-3" #6136 daemon
TID 10002 (0x2712) -> "sdk-ScheduledExecutor-1091-3" #6137 daemon
TID 10003 (0x2713) -> "sdk-ScheduledExecutor-1088-3" #6138 daemon
TID 10004 (0x2714) -> "sdk-ScheduledExecutor-1093-2" #6139 daemon
TID  9995 (0x270b) -> "sdk-ScheduledExecutor-1089-2" #6109 daemon
TID  9996 (0x270c) -> "sdk-ScheduledExecutor-1099-2" #6131 daemon
TID  9997 (0x270d) -> "sdk-ScheduledExecutor-1097-1" #6094 daemon
TID  9998 (0x270e) -> "sdk-ScheduledExecutor-1092-2" #6133 daemon
TID  9999 (0x270f) -> "sdk-ScheduledExecutor-1077-3" #6134 daemon

/proc/<pid>/task/<tid>/status for those TIDs:

Name:   rs_normal [work        <- truncated at TASK_COMM_LEN (16)
State:  S (sleeping)
wchan:  futex_wait_queue_me

(/proc/<pid>/task/<tid>/stack is not readable in our container — no CAP_SYS_ADMIN. The Java-side stack below makes it unnecessary.)

Every single one of the 35,745 is WAITING (parking), with an identical stack:

"sdk-ScheduledExecutor-1096-3" #6135 daemon prio=5 os_prio=0 cpu=0.29ms elapsed=46067.04s nid=0x2710 waiting on condition
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.2/LockSupport.java:341)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(...)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(...:1177)
	at java.util.concurrent.ThreadPoolExecutor.getTask(...:1062)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(...:1122)
	at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)

cpu=0.29ms after elapsed=46,067s — these threads have done essentially nothing since creation. They are parked scheduler threads of abandoned AWS SDK async clients.

Why they are named rs_normal

Linux threads inherit comm from the creating thread unless explicitly renamed. These executors are constructed on a Doris rs_normal [worker] thread (inside the Paimon JNI call), and the JDK does not rename ScheduledThreadPoolExecutor worker threads at the OS level. So /proc shows rs_normal [work for threads that have nothing to do with the Doris scheduler. A /proc-based thread-name histogram is actively misleading here — exactly the caveat you raised.

Scope of the leak

Note Timer for 's3a-file-system' metrics system and idle-connection-reaper appear exactly once. So S3AFileSystem itself is effectively a singleton, but the SDK clients underneath are not — 7,149 of them accumulated in ~15 h. The lifecycle problem is on the client, not on the filesystem object.

5. Table / query characteristics

Table	Paimon primary-key table (primaryKeys: [user_id])
Partition keys	none
File format	parquet, zstd
Bucket	4
Columns	24
Storage	S3 (MinIO), s3a:// via Hadoop
Query	SELECT COUNT(*), small table, few splits
Outcome	20/20 completed normally, none cancelled or timed out

The leak therefore does not require large scans or failures — 20 trivial counts on a tiny table are enough.

6. enable_paimon_cpp_reader A/B — attempted, but not a clean result

I ran the same 20 queries with SET enable_paimon_cpp_reader=true. All 20 failed at:

doris::PaimonCppReader::init_reader()

presumably because the C++ reader does not support this primary-key table in this build. Thread and SDK-instance deltas were exactly 0.

I want to be explicit that this is not evidence that the C++ path is leak-free — the queries never reached the read path. It is only consistent with the leak living behind the JNI reader. A proper A/B needs an append-only table that the C++ reader actually supports; I can run that if it would help.

---

Where this leaves the investigation

• The task-executor accounting invariant is not broken (num == real_num == 8, max_num == 256).
• There are not multiple same-named scheduler owners.
• It is native-name inheritance plus a genuine JNI-side client lifecycle leak: each Paimon scan appears to build an AWS SDK async client (with its own 5-thread ScheduledExecutor) that is never close()d, so both the threads and their per-thread allocator caches accumulate until rss hits mem_limit.

That suggests ownership sits with the Lakehouse/Paimon JNI reader (S3 client construction/close per scan or per split), not with BE execution/workload scheduling. Linking with #65416 still makes sense — the /proc symptom is identical and would have the same explanation.

Two small notes on the mitigation guidance in the original report, for anyone who finds this later:

• Rolling BE restarts remain the only demonstrated mitigation. Confirmed again here: a restart took this BE from 37,537 threads / 5,449 MB to 1,783 threads / 1,008 MB.
• SHOW BACKENDS reports Alive: true right up to the point where every query fails on MEM_LIMIT_EXCEEDED, so it must not be used as the health signal. Tracking thread count gives hours of warning; tracking memory percentage alone does not.

Happy to collect anything else — an append-only-table C++ reader A/B, a repeat on 4.1.4-rc04, or jcmd VM.native_memory / allocator breakdowns.

[wangmingzhou1986]

Resolved in 4.1.4. Upgraded the cluster today and the leak is gone.

Measurement

Same reproducer as before (20 × SELECT COUNT(*) on a Paimon primary-key table via external catalog), both BEs freshly restarted:

metric, per BE per 20 queries	4.1.3 (doris-4.1.3-rc02-7126cf65d96)	4.1.4 (doris-4.1.4-rc04-ad35a140c7f)
distinct sdk-ScheduledExecutor-<n> instances	+40	+0
threads in /proc/<pid>/task	+200	+1 (noise)

BE              threads before  after   Δ      SDK instances before  after   Δ
doris-be-0             1860     1861    +1                       1      1    +0
doris-be2-0            1851     1852    +1                       1      1    +0

The SDK executor instance count stays at 1 on both BEs instead of growing by 2 per query. That is exactly the mechanism identified earlier (AWS SDK async clients created per Paimon scan and never close()d), so this issue can be closed as fixed in 4.1.4.

One small note for anyone searching: the published image apache/doris:be-4.1.4 self-reports as doris-4.1.4-rc04-ad35a140c7f. There is no separate -rc04 tag on Docker Hub — the rc04 build is the 4.1.4 release.

A compatibility caveat worth documenting for Paimon-catalog users

4.1.4 bumps the bundled Paimon from 1.3.1 to 1.4.2, and the java_extensions directory is renamed paimon-scanner → paimon-connector. Two things followed from that in our environment:

1. Lake-format compatibility is fine. Our tables are written by Paimon 1.3.1 (Flink CDC) and now read by 1.4.2. Before upgrading I verified this in isolation — a throwaway pod running the 4.1.4 image with sleep (no Doris process), using its bundled Paimon 1.4.2 through the Java API to read production tables directly:

• 3 static tables matched field-by-field (11×24, 1×16, 2159×10)
• a 57.8M-row primary-key table, both engines pinned to the same snapshot, matched exactly: 57,783,725 from FOR VERSION AS OF 2333 on 4.1.3 and 57,783,725 from the 1.4.2 direct read. After the upgrade the same query still returns 57,783,725.

(Comparing row counts across engines on a live table is meaningless — the table commits every ~30 s, so each engine pins a different snapshot. Pinning both to one snapshot is what makes the comparison valid.)

2. FE and BE must be on the same Paimon version. The documented upgrade order is BE first, then FE. During that window — BE on 4.1.4, FE still on 4.1.3 — every external-catalog query fails:

[INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException:
java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment;
local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2

The FE serializes Paimon split/table objects and the BE deserializes them; CatalogEnvironment's serialVersionUID changed 1 → 2 between Paimon 1.3.1 and 1.4.2. So for deployments using Paimon external catalogs, the BE-then-FE window is a planned outage, not a transparent rolling upgrade. Ours lasted about 6 minutes. It might be worth a line in the upgrade docs.

Happy to run anything else if it would help confirm.

[Asthenia0412]

Hi @wangmingzhou1986, I've submitted the master-side fix in #67963: Paimon is bumped 1.3.1 -> 1.4.2 (same as the 4.1.4 release you verified), with the paimon-scanner code adapted to the 1.4.2 API. This should fix the AWS SDK async client thread leak on master. Thanks for the detailed triage!

[wangmingzhou1986]

@Asthenia0412 thanks — that matches what we measured. Confirming the fix from the release side, in case it helps the PR review.

We upgraded the whole cluster (2 FE + 2 BE) from doris-4.1.3-rc02-7126cf65d96 to doris-4.1.4-rc04-ad35a140c7f and re-ran the identical 20-query reproducer on freshly restarted BEs:

per BE, per 20 external-catalog queries	4.1.3	4.1.4
distinct sdk-ScheduledExecutor-<n> instances	+40	+0
threads in /proc/<pid>/task	+200	+1 (noise)

So the Paimon 1.3.1 → 1.4.2 bump is indeed what fixes it, and the three JNI-cleanup PRs suggested earlier in triage were not the relevant change. Since #67963 is the same bump for master, I'd expect it to close this.

Two things we verified before upgrading that may save someone else the trouble:

1. Lake-format compatibility (1.4.2 reader over 1.3.1-written tables) holds. Our tables are written by Flink CDC with Paimon 1.3.1. Before touching production we ran the 4.1.4 image as a throwaway sleep pod (no Doris process) and used its bundled Paimon 1.4.2 through the Java API to read production tables directly. Three static tables matched field-by-field, and a 57.8M-row primary-key table matched exactly when both engines were pinned to the same snapshot: 57,783,725 from FOR VERSION AS OF 2333 on 4.1.3, and 57,783,725 from the 1.4.2 direct read. After the upgrade the same query still returns 57,783,725.

Worth noting for anyone repeating this: comparing row counts across engines on a live table is meaningless — ours commits every ~30 s, so each engine pins a different snapshot and the numbers differ in both directions. Pinning both sides to one snapshot is what makes the comparison valid.

2. FE and BE must be on the same Paimon version — the documented BE-then-FE order is an outage window for Paimon catalogs. With BE on 4.1.4 and FE still on 4.1.3, every external-catalog query fails:

[INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException:
java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment;
local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2

The FE serializes Paimon split/table objects for the BE to deserialize, and CatalogEnvironment's serialVersionUID changed 1 → 2 between Paimon 1.3.1 and 1.4.2. Ours was down about 6 minutes. For deployments using Paimon external catalogs this is a planned outage rather than a transparent rolling upgrade — might be worth a line in the upgrade docs alongside this change.

Happy to test anything else on 4.1.4 if useful.