TiFlash: pack-level max_len skip for MAX(CHAR_LENGTH) / MAX(LENGTH)

[JaySon-Huang]

Feature Request

Is your feature request related to a problem? Please describe:

Production workloads include full-table scans that compute MAX(CHAR_LENGTH(col)) on many string columns of a large table. With no WHERE / LIMIT, TiFlash must read every selected string column.

SELECT
  MAX(CHAR_LENGTH(c1)) AS c1,
  MAX(CHAR_LENGTH(c2)) AS c2,
  -- ... a dozen or more string columns
  MAX(CHAR_LENGTH(c16)) AS c16
FROM t;

One observed run was about 1.8 billion rows, 16 columns, 373 GB, TableScan ~7 s. Per node, decompressed scan rate was ~7.5 GB/s on 7 nodes, saturating cloud-disk read bandwidth (example baseline 1250 MB/s) and stalling other IO on the same nodes.

The plan is:

TableFullScan → Projection(char_length × N) → HashAgg(max × N)

The bottleneck is reading string payload, not aggregation. Late materialization has no filter. Existing MinMaxIndex stores lexicographic min/max of string values, which is unrelated to length, so packs that cannot refresh the global max still cannot be skipped. MAX only cares whether a longer value exists, but the storage layer has no length-dimension pack stats, so it reads everything.

This request is a follow-up of #11049.

Describe the feature you'd like:

Record length upper bounds for string columns in DMFile pack metadata, and prune with a running max while scanning MAX(CHAR_LENGTH(col)) / MAX(LENGTH(col)).

On pack write (cost similar to existing MinMax), record:

• max_byte_len: max byte length of strings in the pack (for LENGTH)
• max_char_len: max character length in the pack (for CHAR_LENGTH / utf8mb4 code points)

While scanning, keep a running max per relevant column:

• If pack_max_len <= running_max: skip the whole pack (including compressed chars and StringSizes)
• Otherwise read the pack, compute its actual max, and update the running max

The same pack stats can also serve filters such as WHERE CHAR_LENGTH(c1) > k: packs with max_char_len <= k can be skipped. That is not the first goal; please leave room for it in the format.

Expected effect: after a few packs that contain longer strings, the rest can be skipped. The more concentrated the length distribution, the more is skipped. This is the main path from “read hundreds of GB” to “read pack metadata + a few packs”.

Implementation notes:

• Keep CHAR_LENGTH and LENGTH stats separate (UTF-8 character count ≠ byte length), or store both.
• Apply this prune to stable DMFile packs; Delta / memtable still need a full scan or equivalent stats.
• Needs a new pack-stat format and a compatible reader when old files lack the stats (treat as unknown, do not skip).
• For multi-column MAX(CHAR_LENGTH(c_i)), each column keeps its own running max and skip decision; skipping one column does not imply skipping others.

Describe alternatives you've considered:

1. Read StringSizes only, skip chars
   Helps LENGTH or ASCII data by not reading payload, but still scans every row’s sizes. Complementary: this request drops packs that cannot refresh max; remaining packs can use sizes-only to cut IO. It does not replace pack skip.

2. Fold CHAR_LENGTH into HashAgg
   Avoids materializing Int64 intermediates for every row; mainly saves CPU / memory. Wall time barely changes when already IO-bound.

3. Persist per-row length in a hidden or STORED generated column
   MAX(length_col) becomes an integer-column scan, with broader coverage, but more write amplification and schema change. TiDB currently cannot add STORED generated columns via ALTER TABLE. For this query, a pack-level upper bound is enough for MAX prune; per-row materialization is not required first.

4. Application-side sampling / splitting columns / Resource Control
   Can ease saturation, but does not reduce engine IO when a true global max is required.

Teachability, Documentation, Adoption, Migration Strategy:

• Users: no SQL change. MAX(CHAR_LENGTH(col)) / MAX(LENGTH(col)), and optionally range filters on CHAR_LENGTH / LENGTH, should benefit automatically on TiFlash.
• When it applies: DMFiles written after the change (or after compaction) that carry length stats. Old files without stats fall back to a full read; results stay correct.
• Observability: expose packs / rows / bytes skipped by max_len skip in TableScan / tiflash_scan, so prune effectiveness is visible.
• Compatibility: readers must tolerate packs without max_byte_len / max_char_len; writers emit the new stats after the file-format upgrade.
• Docs: note in TiFlash scan / predicate-pushdown docs that string-length aggregation and filters can use pack-level length stats without DDL.