Skip to content

Optimize leading ranges on composite sort keys - #29513

Merged
XuPeng-SH merged 3 commits into
matrixorigin:mainfrom
XuPeng-SH:issue-29507-leading-cluster-range
Sep 30, 2026
Merged

XuPeng-SH merged 3 commits into
matrixorigin:mainfrom
XuPeng-SH:issue-29507-leading-cluster-range

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • BUG
  • Improvement

Which issue(s) this PR fixes:

Fixes #29507

What this PR does / why we need it:

A range on the first component of a composite primary or cluster key did not produce a hidden sort-key predicate for object pruning. The scan could load metadata for objects whose sort-key zone map already ruled them out.

The planner adds a supplemental hidden-key block filter for compatible bounded ranges, BETWEEN, and one-sided comparisons, including reversed operands. The original SQL row predicate remains authoritative. The change uses the existing composite-key serializer, BlockFilterList, and readutil's sort-key zone-map fast path. First-component paired bounds stay separate, avoiding a premerge cast that could change bound semantics.

Block filters now retain metadata-only column references without adding those columns to the scan reader. Row-needed columns keep their compact positions; omitted metadata columns receive distinct positions for the combined runtime/block zonemap path. This also removes unnecessary row reads for existing composite-part block filters. The relation's full schema still supplies physical sequence numbers and zonemaps; no new reader, storage format, or persistent state is introduced.

Partition pruning also consumes block filters. It now matches named predicates to the partition column instead of assuming scan-local and stored partition positions are equal. Ambiguous dotted alias/column names conservatively skip partition pruning. The readutil debug diagnostic likewise resolves by column name rather than indexing the full schema with a scan-local position.

The leading-range optimization remains limited to supported integer, temporal, and same-scale decimal bounds. Float and byte-string leading ranges retain the original scan path because signed zero or byte-prefix ordering could otherwise cause false-negative pruning. Some first-component BETWEEN plans on those types therefore lose their former hidden-key rewrite; exact row filtering remains in place.

Validation

  • Full CGo package tests passed for pkg/sql/plan, pkg/sql/compile, pkg/partitionprune, and pkg/vm/engine/readutil.
  • Public SQL-to-plan tests verify that hidden compound and old constituent metadata filters remain effective while block-only columns are absent from reader attributes. A combined runtime/block column map test checks distinct physical sequence numbers.
  • FilterObjects tests verify object rejection before metadata loading. Partition tests cover same-column compaction, different-column position collision, and dotted-name ambiguity.
  • Incremental go vet and golangci-lint passed; git diff --check passed.

Against base a36da7f72d, the final diff has production net +210 lines, tests net +389, and no document changes. The planner mapping and partition identity checks account for the production increase; the tests exercise public planning and pruning behavior rather than only helper internals.

The historical 329-object local reproduction establishes the missed early-pruning opportunity but did not reproduce the reported production latency. Deterministic tests establish object rejection and removal of hidden-key reader attributes; service-level latency and byte savings have not been measured. CI is pending.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working kind/enhancement size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] Use object sort-key zonemap for first-column range on composite CLUSTER BY

2 participants