[branch-4.1] (topn lazy materialization) Disable topn lazy materialization on non-light-schema-change tables (#65415) - #67927
Closed
liutang123 wants to merge 2804 commits into
Closed
liutang123 wants to merge 2804 commits into
liutang123 wants to merge 2804 commits into
Conversation
…pache#66658 (apache#66718) Cherry-picked from apache#66658 Co-authored-by: wudi <wudi@selectdb.com>
… load commit- apache#66552 (apache#66724) cherry pick: apache#65476 and apache#66552 --------- Co-authored-by: zclllyybb <zhaochangle@selectdb.com>
…grade apache#66604 (apache#66640) Cherry-picked from apache#66604 --------- Co-authored-by: 924060929 <lanhuajian@selectdb.com>
…ed through projects apache#66531 (apache#66741) picked from apache#66531
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65730, apache#66581 Problem Summary: Ordinary Lance scans currently read **every row of a fragment** even when the query only needs the first N rows (e.g. `SELECT ... LIMIT 10`). Lance applies its own LIMIT *after* the scanner's filter, so the query LIMIT can be forwarded to each fragment scanner and let it stop early, cutting IO and decode cost. **How it is fixed** - `thrift`: add an optional `TLanceFileDesc.limit`. - `FE` (`LanceScanNode`): push the query limit into each fragment split via `canPushDownLimit()`, and surface `lanceLimit` in the explain output. - `BE` (`lance_reader`): forward it to the scanner through `lance_scanner_set_limit` for ordinary scans; vector search keeps its own `top_k` limit. **Correctness** The limit is pushed **only when all predicates are already pushed into Lance** (no residual Doris conjunct). Otherwise Doris still re-filters the returned rows, and truncating a fragment early could drop valid results. `OFFSET` needs no special handling: Nereids' `SplitLimit` rewrites `Limit(limit, offset)` into a global `Limit(limit, offset)` over a local `Limit(limit + offset, 0)`, and that local bound is what reaches the scan node. So `getLimit()` already includes the offset; each fragment fetches up to `limit + offset` rows and the upper global LIMIT still applies the offset and the final bound. Per-fragment truncation is therefore always safe. **Behavior change** Query results are unchanged. Only the number of rows scanned per fragment is reduced for LIMIT queries; the explain output shows an extra `lanceLimit=N` line when the limit is pushed. ### Release note Push down LIMIT into Lance fragment scanners to reduce the rows scanned for `LIMIT` / `LIMIT ... OFFSET` queries over Lance tables. ### Check List (For Author) - Test - [x] Unit Test (`LanceThriftContractTest` covers the limit round-trip and the no-limit case) - [ ] Manual test — `SELECT * FROM <lance_tbl> LIMIT 10` returns 10 rows and `EXPLAIN` shows `lanceLimit=10`; a query with a non-pushable predicate keeps the limit out of the scan - Behavior changed: - [x] No. - Does this need documentation? - [x] No.
… values (apache#66714) ### What problem does this PR solve? Problem Summary: Import validated unshredded metadata/value bytes directly into ColumnVariantV2 instead of recursively decoding and re-encoding them. Add Parquet profile counters for direct-import time, rows, and bytes, and cover lazy materialization and shredded-path isolation. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…e#66736) (apache#66738) ### What problem does this PR solve? Issue Number: N/A Related PR: apache/doris-thirdparty#402 Problem Summary: Advance `contrib/clucene` from `c51b5cc9adc` to `abe2b71a1c1`, which contains the fix for block-WAND reads through composite readers (`MultiTermDocs`). The prior submodule revision did not expose or correctly maintain block-postings state across segment-reader boundaries. The updated revision globalizes physical block document IDs, invalidates cached block metadata when a cross-reader skip occurs, preserves short tail blocks, and corrects the first-block posting count after a skip-list lookup. The upstream change includes composite-reader regression coverage for 511/512/tail block layouts, cross-reader `skipToBlock()` transitions, and short child readers. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test: upstream CLucene PR apache#402 adds `TestReadRange.cpp` coverage for composite-reader block reads and skips. - [x] Unit Test: upstream CLucene PR apache#402 CI passed `CLucene UT (Linux)`; parent-repository `git diff --check origin/master...HEAD` and `git fsck --no-progress --connectivity-only HEAD --no-dangling` pass locally. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. Doris inherits the corrected composite-reader block-WAND behavior from the updated CLucene revision. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…pache#66726 (apache#66755) Cherry-picked from apache#66726 Co-authored-by: Yixuan Wang <wangyixuan@selectdb.com>
### What problem does this PR solve? Issue Number: DORIS-27899 Problem Summary: The native Parquet schema validator rejected every group deeper than 100. That bound is below the physical depth produced by standard automatic Variant shredding: - Paimon 1.4.2 defaults `variant.shredding.maxSchemaDepth` to 50 ([source](https://github.com/apache/paimon/blob/release-1.4.2/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java#L418-L424)). - Iceberg also caps automatic shredding at 50 logical levels ([source](https://github.com/apache/iceberg/blob/35889387c8c6ff0a3f57d17a304709dc1b7d9340/parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java#L79-L80)). - An object level adds a field wrapper and a `typed_value` wrapper, so 50 levels can reach group depth 101. - A nested array level adds `typed_value (LIST)`, repeated `list`, and `element` groups, so 50 levels can reach group depth 151. This PR raises the bound to 192. The value covers the 151-level nested-array case plus Doris's maximum nine enclosing logical type levels ([source](https://github.com/apache/doris/blob/branch-4.1/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java#L52)), where list/map encodings may add up to three physical groups per level, and leaves a small compatibility margin. The limit remains a fixed reader-side safety bound. It is intentionally derived from the physical Parquet schema instead of a Paimon table option: the option controls schema inference, is not guaranteed to be present in the file footer, and explicit or third-party writers can produce the same layout. Validation still happens before recursive parsing or schema-driven allocation, and schemas deeper than 192 remain rejected. Two unit tests construct Paimon-style object and nested-array schemas at the default 50-level shredding depth. The existing boundary test also verifies that depth 192 is accepted and depth 193 is rejected. ### Release note Fix native Parquet reads of deeply shredded Variant schemas generated by standard writers. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Native Parquet readers now accept valid group nesting up to 192 instead of 100. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…apache#66786) ### What problem does this PR solve? cherry-pick from master apache#66698 ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
### What problem does this PR solve? Issue Number: Part of apache#66495 Problem Summary: test_lance_vector_search documents its doris.vector_search fixture as carrying an IVF_PQ index, but the fixture SQL (run07_create_vector_types.sql) delegated index creation to a companion create_vector_search_index.py that was never committed, and lance-spark-bundle 0.4.0 cannot create vector indexes through SQL. The table therefore had no index at all, so every use_index / nprobes / refine_factor query in the suite silently executed a flat KNN scan while the goldens still looked correct. Nothing in the repository could observe the difference. Reproduction: build the table the old fixture built and probe it with nprobes=1. Lance ignores nprobes on an unindexed dataset and returns exactly the flat top-10 (rows 256,255,257,254,258,253,259,252,260,251 for the boundary query) - identical to the flat baseline, which is why the defect was invisible. Fix: replace the Spark-created table with an offline-generated Directory Namespace V2 catalog that carries a real IVF_PQ index, and add the evidence that the index is actually used. - lance_build_preinstalled_catalog.py rebuilds the committed fixture and self-checks it: exactly one IVF_PQ index named embedding_ivf_pq_f32 covering every fragment, ANNSubIndex and ANNIvfPartition present in the indexed plan, KNNVectorDistance and no ANN node in the flat plan, and the exact 16 * (n - r)^2 distance ladder that every golden and comment encodes, so a change to the data shape fails here instead of surfacing as an opaque golden diff. Index creation goes through the physical dataset because DirectoryNamespace.create_table_index raises UnsupportedOperationError. - doris.vs_ivf_pq_f32 replaces doris.vector_search: 1024 rows in two fragments, 16-dimensional Float32 embedding[j] = (row_id - 1) + j, so the exact squared L2 distance between rows r and n is 16 * (n - r)^2 and the head/tail queries have no distance ties. Columns are declared NOT NULL to match the fixture being replaced, keeping the only non-nullable Lance column mapping recorded by any Lance suite's DESC golden. The vs_<algorithm>_<element type> name encodes one cell of the algorithm x element type matrix, so a missing combination is visible from the table list alone. - The suite gains a silent-fallback discriminator. Row 256 sits on the first IVF partition boundary, so a genuine single-partition probe must miss true neighbours from the next partition. The suite asserts that the nprobes=1 distance sequence differs from flat search; on the previous unindexed fixture the two are identical and the assertion fails. Distances are compared rather than row ids because the boundary query is symmetric and rows r-d and r+d tie. top_k is 9 there, the last cut that lands on a complete tie pair: at 10 the pair at distance 400 is split, so the golden would pin an arbitrary winner that any change to Lance's top-k selection could flip. Which partition edge row 256 lands next to changes on every retrain, so no measured range is hardcoded; --check prints it instead. - IVF_PQ is lossy, so every indexed query uses refine_factor and the suite documents indexed/flat agreement as an observed property of this frozen fixture and pinned Lance version, not an algorithm guarantee. The fixture is generated with the pins in lance_fixture_requirements.txt. Its readers do not all run the same Lance version - a BE built from source uses lance-c v0.1.2 (lance-rs 4.0.1) per thirdparty/vars.sh, the BE in CI comes from the prebuilt doris-thirdparty package and is already on lance-c v0.1.6 (lance-rs 7.0.0-beta), and Spark writes into the same __manifest through lance-java 4.0.0. The writer is therefore pinned to the oldest Lance in that set, which every reader can read. Verified that this does not make the goldens version-dependent: pylance 7.0.0 reads the committed fixture with results identical to pylance 4.0.1 - same index, same refined top-5, same nprobes=1 boundary rows, same IVF partition ranges. Index training is not bit-reproducible, so regenerating the fixture changes the binary output; the reproducible properties are asserted by the generator self-check instead. IVF_FLAT, IVF_SQ, IVF_HNSW_* and the other vector element types are follow-up work for apache#66495. ### Release note None ### Check List (For Author) - Test: Regression test - Fixture generator self-check with the pinned dependencies - test_lance_vector_search regenerated with -forceGenOut, then passed the normal golden comparison - The whole external_table_p0/lance directory passed (6 suites, 0 failed), covering the pre-existing suites that share the regenerated __manifest - Cross-checked that the nprobes=1 golden row order matches what pylance records probing the same physical index directly - Behavior changed: No, test fixture and regression coverage only - Does this need documentation: No
before
```
case TYPE_DATE:
case TYPE_DATETIME:
case TYPE_DATEV2:
case TYPE_DATETIMEV2:
case TYPE_BOOLEAN:
case TYPE_TINYINT:
```
now
```
dispatch_switch_scalar(left_element_type->get_primitive_type(), call));
```
(cherry picked from commit af9fb9d)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…pache#65982 (apache#66775) Cherry-picked from apache#65982 Co-authored-by: minghong <zhouminghong@selectdb.com>
…6799) ### What problem does this PR solve? Issue Number: DORIS-27887 Problem Summary: On branch-4.1, a Parquet predicate scan can compare a reader type such as a struct with nullable descendants against a block type with nullability represented at a different nesting level. The existing debug check removes only the outer nullable wrapper and aborts the BE even though the recursive type and shape are compatible. This change compares Array, Map, and Struct types recursively while ignoring only nullability at each nesting level. Primitive type and complex-type shape mismatches remain rejected. This is a narrow backport of the relevant type-compatibility fix already present on master; unrelated changes are intentionally excluded. ### Release note Fix a BE crash when Parquet nested predicate columns use equivalent types with different nested nullability representations. ### Check List (For Author) - Test: Unit Test - Added a focused test for equivalent nested Struct nullability and an incompatible nested primitive type. - Ran the focused Parquet scan BE unit test. - Behavior changed: No. The change prevents a debug assertion for semantically compatible nested types. - Does this need documentation: No.
…oncurrent writes (apache#66685) ### What This change introduces a connector-independent write-distribution framework for external table sinks, with Hive, Iceberg, and Paimon as its first consumers. The framework keeps the normal Doris sink Exchange path and composes two independent pieces in BE: - a `PartitionFunction`, which computes a connector ownership key from the input Block; - a `WriterAssigner`, which maps that key either to one stable writer (`IDENTITY`) or through adaptive ScaleWriter behavior (`SKEWED`). The FE selects the connector-specific distribution specification and sends explicit external-sink routing metadata through `TExternalTableSinkHashPartitionInfo`. Connector transforms are evaluated transiently by the Exchange partitioner and are not appended to sink rows as hidden columns. For Hive: - preserve adaptive ScaleWriter behavior for partitioned tables; - preserve adaptive concurrent writing for unpartitioned tables; - route partitioned rows by Hive partition columns through the common external-sink Exchange framework. For Iceberg: - evaluate the active Iceberg partition spec in Doris, including identity, year, month, day, hour, bucket, and truncate transforms; - hash the transformed partition values instead of the raw source columns; - retain `SKEWED` writer assignment so hot partitions may scale to multiple writers; - preserve adaptive concurrent writing for unpartitioned tables; - fall back to a safe single writer when a transform is unsupported or the required BE execution version is unavailable. For Paimon: - support concurrent writes for `HASH_FIXED` tables using Paimon-compatible native routing; - reproduce Paimon `BinaryRow`, default bucket, and `ChannelComputer` semantics directly from Doris Blocks; - map every `(partition, bucket)` to exactly one writer within a write job; - avoid Block-to-Arrow/JNI conversion merely to calculate fixed-bucket routing; - retain the safe single-writer behavior for `HASH_DYNAMIC`, `KEY_DYNAMIC`, custom bucket functions, and unsupported partition or bucket key types; - retain random concurrent writing for bucket-unaware tables. The change also bounds Paimon JNI writer memory: - derive one process-wide limit from `JVM -Xmx * paimon_jni_memory_limit_ratio`; - use `0.5` as the default ratio; - share the limit across all Paimon writers on the BE; - account for Doris-managed Paimon native pages and Java Arrow direct memory; - fail the current Paimon write when the hard limit is exceeded instead of allowing an uncontrolled process OOM; - expose current, peak, limit, and rejected-allocation counters in the writer profile. ### Why External table writers previously reused Hive-specific Exchange types and partitioning behavior. That made connector ownership semantics implicit and prevented Iceberg transforms and Paimon fixed buckets from being represented accurately. For Iceberg, routing by raw source columns does not reproduce transforms such as bucket, truncate, or time transforms. Rows belonging to the same physical Iceberg partition could therefore be distributed inconsistently and produce unnecessary small files. For Paimon fixed-bucket tables, concurrent writing must preserve `(partition, bucket)` ownership. Sending the same bucket to multiple writers can create conflicting writer state and extra files. The previous safe fallback serialized more writes than necessary. The common framework makes the ownership calculation explicit while reusing Doris Exchange scheduling and ScaleWriter assignment: 1. FE selects a connector distribution specification; 2. the Exchange `PartitionFunction` calculates a logical ownership key; 3. `WriterAssigner` applies stable or adaptive writer assignment; 4. the connector writer receives the original row unchanged. ### Compatibility The new external-sink hash metadata is gated by BE execution version 12. Unsupported or incomplete payloads fail before row processing instead of silently choosing a routing algorithm with different ownership semantics. This PR intentionally does not duplicate the legacy value-7 ScaleWriter implementation inside the new partitioner. ### Scope This change manages writer distribution and Paimon JNI memory accounting. It does not change connector file-rolling policies or target file-size settings. Stateful Paimon `HASH_DYNAMIC` assignment and `KEY_DYNAMIC` global-index assignment are intentionally not implemented concurrently in this change. They continue to use the existing single-writer fallback.
…ERGE (apache#66498) ## What changed This PR adds row-level DML support for Apache Paimon tables through the Doris Paimon catalog: - `UPDATE ... SET ... WHERE ...` - Supports expressions in assignments and predicates. - Supports primary-key tables using the `deduplicate` and `partial-update` merge engines. - Rejects updates to primary-key columns. - `DELETE FROM ... WHERE ...` - Supports primary-key tables using `deduplicate`. - Supports `partial-update` and `aggregate` tables when their Paimon delete/removal options are enabled. - `MERGE INTO ... USING ... ON ...` - Supports conditional `WHEN MATCHED THEN UPDATE`. - Supports conditional `WHEN MATCHED THEN DELETE`. - Supports conditional `WHEN NOT MATCHED THEN INSERT`. - Supports combining update, delete, and insert branches in one statement. Append-only tables are rejected for row-level DML. Unsupported merge engines and table options fail during analysis with explicit error messages. ## Implementation - Adds dedicated Nereids commands for Paimon UPDATE, DELETE, and MERGE. - Carries a per-row operation value through the logical and physical Paimon sink. - Converts the operation value into Paimon row kinds in the JNI writer. - Binds changelog sink outputs at the logical sink level, preserving the operation column while coercing data columns to the target schema. - Adds target-table collection and schema-change retry integration for bound Paimon row-level sinks. - Adds regression coverage for successful UPDATE, DELETE, and mixed MERGE operations, plus unsupported append-only and `first-row` cases. ## User impact Users can modify existing Paimon primary-key tables directly with standard Doris SQL instead of rewriting the table through INSERT or an external compute engine. ## Validation - `BUILD_TYPE=Debug DISABLE_BUILD_UI=ON ./build.sh --fe -j12` - Debug BE build using the Doris toolchain - `test_paimon_write_row_level_dml`: UPDATE, DELETE, MERGE, and negative cases all passed - FE and BE deployment health checks passed Related to apache#65086
…G casts (apache#66744) cherry-pick apache#66709
…he#66777 (apache#66828) Cherry-picked from apache#66777 Co-authored-by: Mryange <yanxuecheng@selectdb.com>
…BLETS (apache#65871) apache#66116 (apache#66757) Cherry-picked from apache#66116 Co-authored-by: SudharsanK2308 <62323624+SudharsanK2308@users.noreply.github.com>
…pache#66852) Cherry-picked from apache#66219 Co-authored-by: meiyi <meiyi@selectdb.com>
…ent (apache#66473) cherry pick from apache#66530
…rsion (apache#66907) ### What problem does this PR solve? Related PRs: apache#66809, apache#66685 `branch-4.1` uses Paimon 1.4.2, but `hive-catalog-shade` 3.1.1 embeds Paimon 1.3 classes. The existing packaging workaround therefore removes `org/apache/paimon/**` and related service descriptors from the shade artifact and relies on separately managed Paimon dependencies. `hive-catalog-shade` 3.1.2 now embeds Paimon 1.4.2, so the workaround is no longer needed. In addition, external table sink hash routing on `branch-4.1` currently uses BE execution version 12. Master reserves version 12 for Iceberg Variant compatibility and reserves version 13 for external sink hash routing in apache#66809. Release branches must use the same compatibility boundary. ### What changed? - Upgrade `hive-catalog-shade` from 3.1.1 to 3.1.2. - Remove the dependency-order workaround and the direct `paimon-hive-connector-3.1` dependency. - Remove the Maven Ant repackaging step that strips `org/apache/paimon/**` and Paimon service descriptors. - Move external table sink hash routing from BE execution version 12 to 13. - Advance the FE and BE maximum execution version to 13 and document master's version 12 reservation. ### Validation - Verified XML syntax with `xmllint`. - Verified Maven dependency resolution includes `hive-catalog-shade:3.1.2` and Paimon 1.4.2 modules. - Verified the Paimon core and Hive storage-handler classes embedded in `hive-catalog-shade:3.1.2` match Paimon 1.4.2. - Formatted the changed BE files with clang-format 16.0.6. - Ran `git diff --check`. - BE compilation was intentionally skipped because this change only updates compatibility constants and packaging metadata. ### Release note None
…ailed apache#56207 (apache#66946) Cherry-picked from apache#56207 Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
…6722 (apache#66945) Cherry-picked from apache#66722 Co-authored-by: Mryange <yanxuecheng@selectdb.com>
…iguration apache#66836 (apache#66933) Cherry-picked from apache#66836 Co-authored-by: dzr171712 <166372158+dzr171712@users.noreply.github.com>
…schema table apache#66834 (apache#66931) Cherry-picked from apache#66834 Co-authored-by: dzr171712 <166372158+dzr171712@users.noreply.github.com>
…apache#66629) (apache#66924) Pick apache#66629 to branch-4.1. Original PR: apache#66629 Cherry-picked from: bd12e5a Backport note: branch-4.1 does not have `HttpRequest::mark_send_reply()`, so the master-only call was dropped while resolving `stream_load.cpp`. Validation: - `sh format_code.sh` on the changed BE C++ files - `ninja -C be/ut_build_ASAN -j 39 doris_be_test` - `doris_be_test --gtest_filter=StreamLoadTest.*:HttpAuthTest.*` (9 tests passed) Co-authored-by: gavinchou <gavinchou@apache.org>
…che#61762 apache#62335 (apache#66915) ## What changed - Backport apache#61762 to add the TPC-H MOR unique-key regression suite, including MOR-versus-DUP comparison and MOR value-predicate-pushdown coverage. - Backport apache#62335 to rename the suite from SF10 to SF100, switch its S3 paths and expected row counts to SF100, and add the 22 auto-discovered SQL expected-output files. ## Why apache#62335 depends on the suite introduced by apache#61762. `branch-4.1` does not contain apache#61762, so cherry-picking apache#62335 alone reports `rename/delete` conflicts for the entire suite because the rename source files are absent. Backporting apache#61762 first makes the dependency explicit and allows apache#62335 to apply cleanly as the intended directory rename. ## Impact This adds branch-4.1 regression coverage for reading Merge-on-Read unique-key tables as duplicate-key tables and for MOR value predicate pushdown at TPC-H SF100 scale. It changes regression tests only and does not modify product code. ## Validation - Cherry-picked apache#61762 and then apache#62335 with `-x`; both applied without conflicts in this order. - Verified the final `tpch_sf100_unique_mor_p2` suite and expected-output subtree are byte-for-byte identical to upstream apache#62335. - Verified all 22 expected-output files are identical to the existing `tpch_sf100_p2` outputs from which they were copied. - Verified `read_mor_as_dup_tables` and `enable_mor_value_predicate_pushdown_tables` already exist on `branch-4.1`. - Not run: the full SF100 regression suite, which requires the external SF100 dataset and performs two full loads into both MOR and DUP tables. The upstream change itself has end-of-file blank-line warnings in `q06.sql` and `q21.sql`; this backport preserves those files unchanged. --------- Co-authored-by: Yongqiang YANG <yangyongqiang@selectdb.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…during cleanup (apache#67539) (apache#67833) pick: apache#67539
…arisons apache#67730 (apache#67843) Cherry-picked from apache#67730 Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
…ession extraction apache#67717 (apache#67863) Cherry-picked from apache#67717 Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
…up by apache#67638 (apache#67832) Cherry-picked from apache#67638 Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
…che#67769 (apache#67829) Cherry-picked from apache#67769 Co-authored-by: Gabriel <liwenqiang@selectdb.com>
… the previous task apache#67710 (apache#67821) cherry-pick: apache#67710
…e#67847) Cherry-picked from apache#67065 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
… keys when recycling rowsets (apache#67827) (apache#67854) pick: apache#67827
(apache#67838) Cherry-picked from apache#67013 Co-authored-by: meiyi <meiyi@selectdb.com>
…o eliminate redundant mark joins (apache#66482) (apache#67280) pick apache#66482 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…ification apache#66706 (apache#66938) Cherry-picked from apache#66706 Co-authored-by: dzr171712 <166372158+dzr171712@users.noreply.github.com>
…ific datatypes (apache#67826)【 … ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
### What problem does this PR solve?
Problem Summary:
Upgrade Lance Java to 11.0.0 and automatically replace its Linux x86_64
JNI library with the glibc 2.17 build from apache/doris-thirdparty
during FE packaging.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…n the async cache write UT apache#67901 (apache#67905) Cherry-picked from apache#67901 Co-authored-by: Mingyu Chen (Rayner) <morningman.cmy@gmail.com>
…on_insert/json_replace at analysis time apache#67804 (apache#67914) Cherry-picked from apache#67804 Co-authored-by: Jerry Hu <hushenggang@selectdb.com>
…e#67325) ### What problem does this PR solve? Issue Number: close apache#66496 Problem Summary: The Lance reader previously reported several Arrow and Lance-specific types as `UNSUPPORTED`, preventing Doris from correctly discovering schemas or reading these columns. This PR adds Doris-side support for: - Arrow Null as Doris `NULL` - Arrow Duration as Doris `BIGINT` - Arrow and Lance JSON extensions as Doris `JSON` - Lance BFloat16 as Doris `FLOAT` Implementation details: - FE recognizes the supported Arrow and Lance extension metadata and validates each extension's physical storage type. - BE maps the same logical types during schema discovery and scan execution. - BFloat16 values are widened to Float32 without precision loss, including values nested in arrays and other complex types. - Ordinary nested Arrow columns bypass reconstruction when no BFloat16 or registered extension array requires normalization. - Only top-level Arrow Null fields are supported. Nested Null fields remain unsupported because the current complex-type deserialization path cannot handle them safely. ### Release note Add Doris support for Arrow Null and Duration types, and Lance JSON and BFloat16 extensions. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test Coverage includes: - FE type-conversion tests for Null, Duration, JSON, and BFloat16. - BE schema and value tests, including nested BFloat16 and the nested-column no-op normalization path. - `test_lance_catalog_all_types` - `test_lance_s3_tvf` Validation: - Production-source diff checks passed. - A complete local FE test run is blocked because the standalone `fe-core` build cannot resolve Doris internal SNAPSHOT dependencies. - A complete local BE test run is blocked by the stale macOS build directory and third-party headers; CI will run the full suite. - Behavior changed: - [ ] No. - [x] Yes. Supported Lance schemas are mapped to Doris types instead of `UNSUPPORTED`. - Does this need documentation? - [ ] No. - [x] Yes. A follow-up documentation PR will be submitted separately. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --------- Co-authored-by: wangzhaobo <wangzhaobo@bytedance.com> Co-authored-by: wangzhaobo957-cloud <wangzhaobo@bytedance>
…d do not reuse callback (apache#67755) (apache#67912) Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…on non-light-schema-change tables (apache#65415) A top-N query that emits a non-order-by column fails on a **non-light-schema-change** OLAP table (a table created/upgraded with `light_schema_change = false`, where every column's `uniqueId` is `-1`): ```sql select id, name from tbl order by createdate desc limit 10; ``` ``` ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]field name is invalid. field=__DORIS_GLOBAL_ROWID_COL__tbl, field_name_to_index=[...], col_unique_id=2147483647 ``` Fix: disable topn lazy materialization for non-light-schema-change OLAP tables in MaterializeProbeVisitor and fall back to normal topn (the working two-phase read path). A new helper supportOlapTopnLazyMaterialize() consolidates the existing AGG_KEYS exclusion with the new light_schema_change requirement, applied at visitPhysicalOlapScan, visitPhysicalCatalogRelation and visitPhysicalFilter. Add regression test topn_lazy_light_schema_change verifying: - light_schema_change=false: no lazy materialization in the plan, correct results. - light_schema_change=true: lazy materialization still applies, correct results. ## Behavior after the fix | Table | Plan | Result | |-------|------|--------| | `light_schema_change = false` | plain `PhysicalOlapScan` (no lazy) → safe two-phase read | correct values | | `light_schema_change = true` | `PhysicalLazyMaterialize` / `PhysicalLazyMaterializeOlapScan` (unchanged) | correct values |
liutang123
requested review from
Gabriel39,
airborne12,
csun5285,
eldenmoon,
gavinchou,
hello-stephen,
liaoxin01,
luwei16,
morningman and
yiguolei
as code owners
September 14, 2026 03:17
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bp #65415