[SPARK-59183][SDP] SCD2 Ignore-null support; Populate version map per event row at ingest - #58645

Open

AnishMahto wants to merge 15 commits into

apache:masterfrom

AnishMahto:SPARK-59183-populate-verison-map-at-ingest

Commits on Sep 10, 2026

[SPARK-59358][SDP] Project missing columns during SCD2 preprocessing
AnishMahto
committed
bd07e65
[SPARK-59358][SDP] Test preprocessing column casing
AnishMahto
committed
8a0ec2b
cleanup
AnishMahto
committed
ded99f1
cleanup whitespace
AnishMahto
committed
a563f15
cleanup and unify helpers to Scd2VersionMap
AnishMahto
committed
7761b72
delete rows in microbatch receive null version maps
AnishMahto
committed
668f2d9
document risk for changing ignore-null selection between microbatch retries
AnishMahto
committed
b15794d
use withField
AnishMahto
committed
f796137
use JSON serialization for version map keys
AnishMahto
committed
7a9e4ac
cleanup
AnishMahto
committed
b642a25
fix delete detection
AnishMahto
committed
1800e18
self-review
AnishMahto
committed
a463835
refactor out generic extractLeafPaths
AnishMahto
committed
1bfd55d
[SPARK-59183][SDP] Handle reductive evolution in version maps
AnishMahto
committed
7e61ec0
[SPARK-59183][SDP] Pin version map key casing
AnishMahto
committed
eb76dc4