TooManyInvalidVersion GC rewrites entire stable when reclaimable data is tiny

[JaySon-Huang]

Enhancement

Background GC (DeltaMergeStore::gcTrySegmentMergeDelta) decides whether to rewrite a whole segment via shouldCompactStableWithTooManyInvalidVersion. That heuristic compares total MVCC version amplification against dt_bg_gc_ratio_threhold_to_trigger_gc (default 1.2), not how much data the current GC safepoint can actually drop. On update-heavy tables this fires full-stable MergeDelta that rewrite hundreds of MB to reclaim well under 1% of versions.

The other two compact reasons in the same function (TooManyDeleteRange, TooMuchOutOfRange) are not the issue here. Production logs below are 100% compact_reason=TooManyInvalidVersion.

Current strategy

gcTrySegmentMergeDelta tries three reasons, in order:

1. shouldCompactDeltaWithStable — delta delete-range vs stable (dt_bg_gc_delta_delete_ratio_to_trigger_gc, default 0.3) → TooManyDeleteRange
2. shouldCompactStableWithTooMuchDataOutOfSegmentRange — packs outside the segment range → TooMuchOutOfRange
3. shouldCompactStableWithTooManyInvalidVersion — once per new gc_safe_point → TooManyInvalidVersion

The third check (code comment: "for optimization purpose, does not mean to be accurate") is:

1. Skip if gc_hint_version > gc_safepoint (nothing older than safepoint).
2. Trigger if num_versions > num_rows * ratio or num_versions > num_puts * ratio.
3. On trigger, segmentMergeDelta rewrites the entire stable DMFile.

gc_hint_version is the minimum second-oldest version across keys/packs. One key with an expired extra version makes the whole segment eligible. num_versions / num_rows (or / num_puts) counts extra versions that may still be above safepoint (recent updates). The decision never estimates reclaimable rows/bytes at the current safepoint.

After a check, the segment records lastCheckGCSafePoint and will not be re-checked until safepoint advances — whether or not MergeDelta ran. Each GC round still scans up to dt_bg_gc_max_segments_to_check_every_round (default 100) segments.

These GC-triggered MergeDeltas are also not the normal delta-flush path. Observed delta_rows / delta_bytes were far below dt_segment_delta_limit_rows (80,000) / dt_segment_delta_limit_size (41 MB), and delta_deletes was almost always 0. The rewrite happens only because of the invalid-version heuristic.

What went wrong in production

Two TiFlash log extracts from the same node (10.107.2.188:3930, 2026-08-17). Write cost = new stable_bytes. Space saved = (old stable + old delta) − (new stable + new delta). Version drop = (old stable_rows + old delta_rows) − (new stable_rows + new delta_rows).

Log tiflash_10.107.2.188_3930.log (10:18:15–10:18:46)

Metric	Value
TooManyInvalidVersion / all GC MergeDelta	100 / 100
Unique (table_id, segment_id)	100 (one full GC round)
New stable written	14.0 GB
Space reclaimed	92 MB
Aggregate write amp (written / reclaimed)	152×
Version rows dropped	0.65% (median 0.58%, max 4.17%)
Median write amp per segment	171× (p90 291×, worst 510×)
Delta vs stable	median 0.61%; max 3.3 MB / 24k rows

96/100 events had write amp > 50×. None of this work would have been scheduled by the regular delta-compact thresholds.

Log tiflash_10.107.2.188_3930.0943.log (09:43:00–09:56:34)

192 paired TooManyInvalidVersion MergeDeltas in two rounds (safepoint advanced in between). Aggregate numbers look acceptable (55.4 GB written, 8.94 GB reclaimed, 6.2×) only because two tables dominate the reclaim. The same heuristic is bimodal:

Slice	N	Written	Reclaimed	Write amp	Median version drop
Round 1 (09:43, table_id 152xxx)	92	40.5 GB	8.83 GB	4.6×	18%
Round 2 (09:56, table_id 1606xx)	100	14.9 GB	108 MB	138×	0.75%
High-reclaim tables 152950 + 152515	84	35.6 GB	8.83 GB	4.0×	21%
The other 19 tables	108	19.8 GB	111 MB	179×	0.77%

Round 2 is the same failure mode as the 10:18 log (same ~30s burst, 100-segment cap, ~0.7% reclaim, ~140× write amp).

Even the “good” round is mixed:

• 152515 had segments that dropped 99.6% of versions (write ~0.4 MB, reclaim ~105 MB) — the heuristic is doing the right thing.
• 152950 internally: 52 segments with GC% ≥ 10% at 2.9× write amp, vs 25 segments with GC% < 10% at 29×.
• Worst case in the same round: table_id=152950 segment 1131229 rewrote 650 MB and dropped 2 rows.

Raising dt_bg_gc_ratio_threhold_to_trigger_gc (e.g. 1.2 → 1.5) only helps if low-reclaim segments sit in a lower version-amplification band than high-reclaim ones. INFO logs do not print StableProperty (num_versions / num_rows / num_puts are TRACE-only), so operators cannot tell whether the two populations are separable on that axis. If both sit just above 1.2 and differ only in how much is below safepoint, turning the knob reduces useful and useless GC together.

Why this is a strategy bug, not a tuning miss

1. Wrong quantity. The threshold is advertised as “ratio of invalid version”, but the implementation compares total versions to unique keys / clean puts. Extra versions that are still live count toward the trigger.
2. Wrong granularity. gc_hint_version is a segment-wide minimum. One GC-able key + many live extra versions ⇒ rewrite the whole DMFile.
3. No ROI gate. There is no minimum reclaimable-bytes or reclaimable-rows check before paying the full stable rewrite. Observed low-reclaim events had write amp of 100×–500000×.
4. Full-round blast radius. Default 100 segments/round × one rewrite each means a safepoint bump can immediately write ~14 GB of useless compact (as in both 1606xx rounds) and occupy the GC thread for ~30s.
5. Observability. Trigger logs only compact_reason and segment id. Without TRACE StableProperty, it is impossible to see why the 1.2 line was crossed or how much was expected to be dropped.

What a better policy should do

Keep MergeDelta when the current safepoint can drop a meaningful fraction of the stable (the 152515 / high-GC 152950 cases, write amp ~1–5×). Do not rewrite when version amplification is high but reclaimable data at this safepoint is tiny (the 1606xx pattern).

Concretely, TooManyInvalidVersion should be gated on an estimate of reclaimable rows/bytes at gc_safe_point, not only num_versions / num_rows (or / num_puts). The existing ratio can stay as a cheap prefilter; a rewrite should also require something like “reclaimable stable bytes ≥ X% of current stable” (or a minimum absolute size).

Tuning-only workarounds (dt_bg_gc_ratio_threhold_to_trigger_gc, dt_bg_gc_max_segments_to_check_every_round) can rate-limit the damage; they cannot express “skip this rewrite because it would drop 0.6% of the file”.

[JaySon-Huang]

Proposal: pack-level version CDF to gate TooManyInvalidVersion

Pack-level gc_hint_version is not enough to fix this. It only answers “does this pack contain some obsolete data?”, which is the same existence check we already do at segment granularity. If expired extra versions are sprinkled across packs (the 1606xx pattern: ~0.6% reclaim, 140× write amp), every pack still looks eligible and we still rewrite the whole DMFile.

The missing input is an estimate of how much the current safepoint can actually drop, cheap enough to compute from metadata.

What the GC decision needs

TooManyInvalidVersion always rewrites the entire stable. The gate should be:

estimated_reclaim_bytes(S) / stable_bytes >= θ

S is the current gc_safe_point. θ is a new threshold (suggested default 0.1), separate from dt_bg_gc_ratio_threhold_to_trigger_gc.

S is not known at DMFile write time, so we must not persist “reclaimable rows at the safepoint used during last compact”. That number is stale on the next safepoint bump. We need a small distribution that can answer prefix queries for an arbitrary S.

Under DMVersionFilterMode::COMPACT, rows with version < S are candidates to drop, but each key keeps at most one row visible at S. Pack property already has num_rows (distinct keys). The estimate is:

estimated_reclaim_rows(S) ≈ max(0, count(version < S) - num_rows)
estimated_reclaim_bytes(S) ≈ estimated_reclaim_rows(S) / pack.rows * pack.bytes

This matches the two populations in the logs:

• Low-reclaim segments (~0.65% dropped): most keys still have one row < S, plus very few truly expired extras → count(v < S) - num_rows is tiny → skip.
• High-reclaim segments (18%–99% dropped): extras are also < S → estimate is large → compact.

num_versions / num_rows > 1.2 cannot tell these apart. It only measures leftover amplification in the file, not how much of it is below S.

Metadata to add

1. Primary (required): per-pack CDF of the version column

Write path already scans version (PackStat.first_version, COMPACT gc_hint / not_clean). Accumulate a fixed-size histogram in the same pass:

• min_version / max_version of the pack
• 32 linear buckets (or 32 monotone CDF sample points) of row counts

count(version < S) is a prefix sum plus in-bucket interpolation. Convert to bytes with existing PackStat.bytes / PackStat.rows; a second byte-weighted histogram is not worth it.

Cost: ~32 × 4B ≈ 128B per pack; ~10KB for an 80-pack segment, negligible next to an 80–200MB DMFile.

2. Secondary (recommended): per-pack CDF of per-key gc_hint_version

We already compute each key’s gc_hint while writing the pack (second-oldest version, or oldest version if it is a delete) and persist only the min. Store a 16–32 bucket CDF of those per-key hints as well:

keys_gcable(S) = count(keys whose gc_hint <= S)

This fixes “one key poisons the pack” (1/8000 keys expired → key-gcable ratio ≈ 0). It does not estimate reclaimable rows: a gcable key may drop one version or many. Use it only as a cheap prefilter; ROI still comes from the version CDF.

3. Segment rollup

calculateStableProperty already walks pack properties. Sum the pack histograms there so shouldCompactStableWithTooManyInvalidVersion does not iterate packs on every check.

How it plugs into the existing checks

Keep the current 1.2 ratio as a cheap first filter. After it passes:

1. gc_hint_version > S → skip (existing)
2. num_versions / num_rows (or / num_puts) below ratio → skip (existing)
3. New: sum(estimated_reclaim_bytes(S)) / stable_bytes < θ → skip
4. Otherwise MergeDelta

Suggested new setting: dt_bg_gc_min_reclaim_ratio (float, default 0.1), not a reuse of dt_bg_gc_ratio_threhold_to_trigger_gc. With the numbers from this issue, 0.65% reclaim is skipped and 18%+ still compact.

Old DMFiles without the new fields: keep today’s heuristic until the next compact rewrites the file. Do not backfill by reading the version column at GC time; that scan is more expensive than a false-positive rewrite.

Why pack-level gc_hint_version alone is insufficient

PackProperty.gc_hint_version is the min of per-key hints in that pack. safepoint > gc_hint_version means some obsolete data exists in the pack — the same comment as today’s segment-level hint.

• One expired key in an 8K-row pack marks the whole pack eligible.
• MergeDelta still rewrites every pack, not only eligible ones. Pack-level hint can only be a do / don’t-compact gate, not a partial rewrite.
• On the 10:18 log, ~0.65% of ~627K versions across 78 packs is ~50 reclaimable rows per pack if spread uniformly. Almost every pack would still be eligible, and we would still pay the 80–150MB rewrite.

Pack-level min is a reasonable extra prefilter when obsolete data is concentrated in a few packs. It does not provide estimated_reclaim(S).

What not to add

Candidate	Why not
Only narrowing gc_hint_version from segment to pack	Same existence check; uniform 0.6% reclaim still triggers
Reclaimable-row count at write-time safepoint	Stale on the next safepoint
Per-row version index / inverted index	Not lightweight
A few version quantiles (p50 / p90)	Stable is often bimodal (base versions + recent extras); quantiles destroy the prefix mass we query
Byte-weighted second histogram	Average row size per pack is enough for ROI
Reusing not_clean / deleted_rows	Write-time layout flags, not reclaimable-at-S

Write path and compatibility

• Increment histogram buckets in DMFileWriter::write / the COMPACT version filter while the version column is already in hand.
• Extend dtpb.PackProperty with optional fields so old files remain readable.
• Optional: log estimated_reclaim_ratio next to compact_reason at INFO so the next production incident is diagnosable without TRACE StableProperty.