HIVE-29580: CBO: Ambiguous column reference not detected in some CTE/CTAS/other queries

[konstantinb]

What changes were proposed in this pull request?

HIVE-29580: under CBO, detect ambiguous by-name references to duplicate-named columns that
escape a subquery/CTE boundary, instead of silently binding to an arbitrary candidate.

Mechanism. When the outer RowResolver is built at a subquery boundary and a duplicate alias
collides (CalcitePlanner block 9), the surviving ColumnInfo is marked (ambiguousName); the
duplicate remains bound under its internal name, so positional use keeps working. A later
by-name reference through the boundary alias fails with the existing
AMBIGUOUS_COLUMN (Error 10007). Check sites: TypeCheckProcFactory (qualified and unqualified
column resolution) and JoinCondTypeCheckProcFactory (both join-condition processors). The
marker is propagated through intermediate projections (genColListRegex, windowing projection)
and excluded from ColumnInfo.equals/hashCode/isSameColumnForRR.

Hive-internal SQL generators fixed along the way (their own output tripped the check —
both regressions were caught by precommit on earlier iterations of this PR and fixed here):

• MergeRewriter emitted the target's partition columns twice into the rewritten MERGE source
  projection; appendNonPartitionColsOfTargetTable now omits them for natively partitioned
  tables (non-native tables keep all columns). No plan/golden churn: the optimizer already
  pruned the duplicates before EXPLAIN.
• The UNION DISTINCT rewrite (SELECT DISTINCT * over an internal alias) synthesizes one
  group-by reference per RowResolver entry — unique by construction — so the markers are
  cleared on that rewrite-private projection (genColListRegex copies; the subquery's own
  RowResolver keeps its markers for user-written references).

Why are the changes needed?

This completes the reference-time model documented in HIVE-20215. Its author noted in 2018 that
a query of exactly this shape "compile[s] successfully but it should throw an error since
reference to t.c1 is ambiguous" and expected HIVE-19770 to fix it. HIVE-19770 shipped the
tolerance (duplicate renamed to its internal name at the boundary) and a reference-time guard,
but the rename destroys the ambiguity information at exactly the boundary the guard needs it,
so the promised rejection never fired.

The silent binding is a wrong-results hazard, not a cosmetic one: limit_join_transpose.q
contained a derived table joining on value whose two escaped key columns hold different
data — whatever src2.key returned was an arbitrary pick between unequal columns. A CTAS over
such a reference silently persists the arbitrary choice into a table. Standard SQL
(e.g. PostgreSQL) rejects these references outright.

Design notes and deliberate choices

• Name collision is the only criterion. It is decidable and stable across planner phases.
  Semantic equality of the candidates is deliberately not considered: it is undecidable in
  general and unstable (constant folding can flip it). This means a reference whose candidates
  are provably equal via a join predicate (cross_prod_3.q's on A.key = B.key) is still
  rejected — PostgreSQL rejects the same shape for the same reason.
• Unreferenced duplicates stay tolerated (star expansion, positional use, set operations),
  preserving HIVE-19770's behavior. Pinned by the *_tolerated tests, including master's own
  pre-existing clientpositive/ambiguous_col.q, which is untouched.
• Non-CBO paths are untouched. CBO/non-CBO parity is explicitly not a goal;
  HIVE-19770/HIVE-20215 deliberately diverged the planners in 2018.
  ambiguous_col_noncbo_baseline.q documents the non-CBO baseline;
  ambiguous_col_unreferenced_tolerated.q is its CBO mirror, so the divergence reads
  shape-by-shape across the pair.
• The 2018 RowResolver guard remains and still covers same-block collisions (e.g. lateral
  view alias reuse — ambiguous_col_lateral_view_alias.q is its only coverage). For
  boundary shapes the marker now fires earlier, which is why the four pre-existing negative
  goldens (ambiguous_col.q, ambiguous_col_2.q, cbo_ambiguous_colref_in_gby.q,
  cte_col_alias_clash.q) change message format: same rejection, now with the 10007 error code.
• A CTAS-specific duplicate-column check from earlier iterations was removed: it regressed
  CTE materialization (Hive-generated CTAS). The reference-time case is still covered by the
  marker (ambiguous_col_ctas.q — kept as a guard that a user CTAS is never exempted should
  an authorship-based exemption ever be added). Rejecting duplicate definitions in CTAS
  remains open under HIVE-12825/HIVE-18568.
• No config gate. The check ships ungated; a flag is a one-line addition at the throw site
  if reviewers prefer one.
• Known residual holes are tolerated: paths the marker does not reach fall back to
  master-equivalent silent binding — the patch never behaves worse than master.

Does this PR introduce any user-facing change?

Yes (backward incompatible): queries that reference a duplicate-named column through a
subquery/CTE boundary by name now fail to compile under CBO with Error 10007. Two existing
tests relied on such references and were updated with explicit column aliases, preserving their
plans and result data: cross_prod_3.q (candidates provably equal — the harmless instance) and
limit_join_transpose.q (candidates genuinely unequal — the wrong-results instance). The JIRA
should carry the incompatible-change label.

How was this patch tested?

• Rejected shapes: ambiguous_col_rejected.q (CBO) and ambiguous_col_noncbo_baseline.q
  (non-CBO baseline), consolidated in the hive.cli.errors.ignore format (resourceplan.q
  precedent) so each file reads as a boundary spec with one FAILED line per statement.
• Mechanism/role-specific clientnegative tests: join-condition checks (qualified and
  unqualified), unqualified reference site, USING-clause expansion, DISTINCT + windowing
  (marker must survive the windowing projection), lateral view alias reuse (RowResolver guard),
  CTAS persistence case.
• Tolerated shapes: ambiguous_col_tolerated.q, ambiguous_col_unreferenced_tolerated.q,
  ambiguous_col_union_distinct_tolerated.q (pins the UNION DISTINCT fix value-sensitively:
  branches differing only in the duplicate-named column must not collapse).
• Unit tests: TestColumnInfo (marker plumbing), TestTypeCheckProcFactory (check helper;
  file refactored to the Enclosed runner to host parameter-free tests alongside the existing
  parameterized ones), TestJoinCondTypeCheckProcFactory (driven through the production walker
  entry point), TestSemanticAnalyzer (compile-level CBO rejection/tolerance plus direct
  genColListRegex propagation tests), TestMultiInsertSqlGenerator (MERGE projection).
• Mutation-verified: removing any marker/propagation/check line fails a specific named
  test; 13 of the 16 new unit tests fail without this patch (the other 3 are negative controls
  that pin the check against over-reach).
• MERGE paths: TestDbTxnManager2/TestTxnCommands2 variants and the MERGE qtests run green
  with zero golden churn.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.2% Duplication on New Code

See analysis details on SonarQube Cloud