[chore](thirdparty) Upgrade Apache Arrow to 24.0.0

[hubgeter]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Doris used Apache Arrow 17.0.0, which blocked consuming newer Arrow fixes and APIs. Upgrade the C++ third-party dependency to 24.0.0, align xsimd with Arrow 24, port the Paimon row-group, INT96, and Parquet LZO compatibility patches, add Arrow Compute to the static dependency graph, and migrate removed Arrow and Parquet APIs while preserving Decimal128 semantics.

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[hubgeter]

run buildall

[hubgeter]

/review

[github-actions]

Request changes: the Arrow 24 upgrade has two build-delivery blockers. The clean Linux third-party job succeeds, but both supported macOS third-party jobs fail at Arrow configuration, and an incremental Paimon source tree skips the new compatibility patch set.

Checkpoint conclusions:

• Goal and test sufficiency: not achieved on all supported paths; no upgrade-specific behavioral test was added, and the failed paths must be rerun after the fixes.
• Scope: all 13 authoritative changed paths and their Arrow/Paimon/build consumers were reviewed; the source changes are otherwise focused.
• Concurrency: no new shared-state or locking defect was introduced; Arrow's parallel column-reader ownership remains valid.
• Lifecycle and cleanup: runtime ownership/error cleanup is preserved, but extracted Paimon patch state is stale across an incremental upgrade (inline finding 2).
• Configuration and dynamic behavior: Arrow 24's CMake 3.25 floor conflicts with Doris's pinned 3.22.1 supported toolchains (inline finding 1).
• Compatibility and rolling upgrade: no persisted-data, journal, RPC, or storage-format rolling-upgrade change was found; source/API and prebuilt-library compatibility are blocked by the two findings.
• Parallel and equivalent paths: Linux fresh-source third-party CI passes; macOS and macOS-arm64 fail, macOS BE lacks the matching Compute archive, and an existing Paimon source directory follows the stale-marker path.
• Special conditions: malformed LZO, INT96 unit variants, projected/selected row groups, and decimal precision boundaries were traced. The empty-row-group truncation is real but behaviorally identical in the base Arrow 17 patch, so it is not attributed to this PR.
• Tests and results: review-only, so no local build/test was run. Live CI currently has Linux third-party passing; both macOS third-party jobs and macOS BE failing; Doris compile and BE UT pending.
• Observability: the build failures are explicit; no additional runtime observability regression was found.
• Persistence, transactions, data writes, and new variables: no new transaction/persistence or FE/BE variable path applies; the ported writer accounting and data semantics showed no PR-introduced defect.
• Performance: no substantiated CPU, memory, I/O, or link-time performance regression was found.
• Other issues: no additional nonduplicate issue remained after two convergence rounds.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[github-actions]

[P1] Raise the supported CMake floor before selecting Arrow 24

Arrow 24's cpp/CMakeLists.txt now requires CMake 3.25, but both macOS third-party workflows explicitly install 3.22.1 (and the gcc10 compilation image pins the same version). The current macOS and macOS-arm64 jobs both fail at this configure step with CMake 3.25 or higher is required; CMAKE_POLICY_VERSION_MINIMUM cannot override cmake_minimum_required. Please update every supported Arrow build path to provide CMake >=3.25 and regenerate the matching prebuilt archives before adopting this version, otherwise the standard macOS/toolchain builds remain unusable.

[github-actions]

[P1] Invalidate the old Paimon patch marker

PAIMON_CPP_SOURCE is unchanged, so an incremental tree already patched on the base commit still has patched_mark. This guard then skips this whole new loop, including paimon-cpp-arrow-24-compatibility.patch, leaving the removed three-argument unique_ptr out-parameter GetRecordBatchReader calls and bare RETURN_NOT_OK uses in place when building against Arrow 24. Please invalidate/re-extract the source when the patch set changes (for example with a patch-set-specific marker or a bumped source identity), so the documented rerunnable download flow cannot silently retain incompatible sources.

[hubgeter]

run buildall

[hubgeter]

/review

[github-actions]

Request changes: the Arrow 24 upgrade still has one build-delivery blocker and one CI integration-coverage defect. The targeted macOS path cannot obtain xsimd 14, and the generic third-party trigger rebuilds only Arrow/Paimon.

Checkpoint conclusions:

• Goal and test sufficiency: not achieved on the targeted supported macOS path. The changed BE tests adapt Arrow APIs but do not exercise third-party source acquisition or generic third-party-to-BE integration.
• Scope: all 17 authoritative changed paths and their Arrow/Paimon/build consumers were reviewed. The upgrade is mostly focused, but the new macOS trigger is broader than its implementation.
• Concurrency: no runtime shared-state, locking, or static-initialization defect applies to these build and API changes.
• Lifecycle and cleanup: runtime reader/writer ownership and error propagation remain intact; workflow source/prebuilt artifact lifecycle is broken by inline finding 1, and dependency selection is stale in inline finding 2.
• Configuration: no dynamic Doris configuration item was added. Current-head CMake/toolchain selection is compatible with Arrow 24, and no separate configuration defect survived validation.
• Compatibility: the inspected Arrow 24 API migrations, INT96 property propagation, row-group behavior, and Doris external-Arrow Paimon path are consistent. No persisted-data, journal, RPC, or rolling-upgrade format defect was found.
• Parallel paths: Linux, Intel/arm64 macOS, fresh/incremental source, Doris external-Arrow Paimon, and final static-link paths were traced. The unsupported standalone Paimon bundled-Arrow branch was not treated as a Doris defect.
• Special conditions: the broad thirdparty/** condition does not match the hard-coded dependency set rebuilt (inline finding 2); no other incorrect conditional path survived validation.
• Test coverage: no upgrade-specific end-to-end test covers source-bundle closure or generic third-party-to-BE integration.
• Test results: review-only, so no local build/test was run. Live BE UT (macOS) currently fails exactly because 14.0.0.tar.gz is missing; style, license, secret, and dependency checks pass, while the full third-party and Doris compile checks were still pending at freeze time.
• Observability: the CI failure is explicit and diagnostic; no additional runtime logs or metrics are needed for this change.
• Persistence, transactions, and data writes: no FE journal, transaction, storage-version, or atomic data-write path is modified.
• FE/BE propagation: no new FE/BE variable or protocol field is introduced.
• Performance: no substantiated runtime CPU, memory, or I/O regression was found. The broad macOS trigger adds avoidable CI work while still providing stale integration coverage, as noted inline.
• Other issues: the remaining CMake-floor and incremental patch-marker concerns are already covered by existing P1 threads, so they were not duplicated. The apparent LZO include failure was disproved because thrift::thrift propagates the installed include root to Parquet's object target.

No additional user-provided review focus was supplied; the whole PR was reviewed.

[github-actions]

[P1] Fetch xsimd 14 before this targeted Arrow build

Passing only arrow paimon_cpp restricts download-thirdparty.sh to those two archives, but Arrow is configured with xsimd_SOURCE=BUNDLED and a local ${TP_SOURCE_DIR}/14.0.0.tar.gz URL. The shared source bundle does not contain that newly selected archive, so the live macOS job fails in xsimd_ep with File not found: .../thirdparty/src/14.0.0.tar.gz before Arrow is compiled. Please download Arrow's complete source-only dependency closure (at least xsimd) before this targeted build, or provide a source bundle guaranteed to match the checked-out vars.sh.

[github-actions]

[P2] Align this broad trigger with the dependency set actually rebuilt

thirdparty_changes matches every file under thirdparty/**, but this branch always rebuilds only Arrow and Paimon. For example, a protobuf version or patch change triggers this BE job while both Arrow and BE still consume protobuf from the shared prebuilt, so a green result would not validate the PR-selected dependency as the new comments imply. Please either narrow the filter to the Arrow/Paimon inputs this step covers or derive and rebuild the changed dependency set (including the required transitive closure).