feat(blob): support placeholder fallback for partial updates - #453
feat(blob): support placeholder fallback for partial updates#453SteNicholas wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end support for data-evolution partial updates on BLOB columns by introducing placeholder entries (bin_length == -2) in the blob format and implementing a multi-layer fallback read path that merges newer/older blob layers row-by-row.
Changes:
- Extend blob write/read format to recognize placeholder entries and (optionally) emit an in-band placeholder sentinel for downstream fallback merging.
- Update
DataEvolutionSplitRead::BlobBunchto retain all blob files across max-sequence “layers” and introduce a newBlobFallbackBatchReaderto resolve placeholders across those layers. - Wire an internal per-reader format option through
AbstractSplitRead::CreateRawFileReadersand add unit + integration coverage for placeholder fallback, gaps, null-wins behavior, and row-range pushdown.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/inte/blob_table_inte_test.cpp | Adds end-to-end integration tests for partial-update placeholder fallback, null-wins semantics, descriptor mode, and row-range pushdown. |
| src/paimon/format/blob/blob_reader_builder.h | Plumbs internal option to enable placeholder-sentinel emission from the blob reader. |
| src/paimon/format/blob/blob_format_writer.cpp | Detects in-band placeholder sentinel and writes bin_length = -2 placeholder entries (no payload). |
| src/paimon/format/blob/blob_format_writer_test.cpp | Adds golden-bytes and strict vs placeholder-aware read-mode tests, including selection-bitmap coverage. |
| src/paimon/format/blob/blob_file_batch_reader.h | Extends blob reader API to optionally emit placeholder sentinel bytes instead of failing on placeholders. |
| src/paimon/format/blob/blob_file_batch_reader.cpp | Implements placeholder handling in offsets/contents building and strict-mode failure behavior. |
| src/paimon/core/operation/data_evolution_split_read.h | Updates BlobBunch to model layered blob files and declares fallback-reader construction. |
| src/paimon/core/operation/data_evolution_split_read.cpp | Keeps layered blob files, chooses concat vs fallback path, and builds padded per-layer segments for fallback merging. |
| src/paimon/core/operation/data_evolution_split_read_test.cpp | Updates/expands BlobBunch tests to validate layering rules, row-count behavior, and optimize-path selection. |
| src/paimon/core/operation/abstract_split_read.h | Adds extra_format_options plumbing to allow per-reader format option overrides (used for placeholder emission). |
| src/paimon/core/operation/abstract_split_read.cpp | Merges extra_format_options over table options when instantiating file formats/readers. |
| src/paimon/common/reader/blob_fallback_batch_reader.h | Introduces the fallback batch reader interface and contracts for layered placeholder resolution. |
| src/paimon/common/reader/blob_fallback_batch_reader.cpp | Implements row-wise fallback merging across sequence layers with gap padding and schema validation. |
| src/paimon/common/reader/blob_fallback_batch_reader_test.cpp | Adds unit tests covering multi-layer fallback, gaps, all-placeholder => null, null-wins, and validation/misalignment failures. |
| src/paimon/common/data/blob_defs.h | Defines placeholder bin length, sentinel bytes, internal option key, and sentinel predicate helper. |
| src/paimon/CMakeLists.txt | Registers the new reader source and unit test in the build. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
bc858d0 to
03c2d22
Compare
A data-evolution partial update rewrites only the touched rows of a blob column and records every untouched row as a placeholder entry (bin_length -2, no data bytes). The blob format writes such entries only for partial updates: a data-evolution blob-only column write enables the placeholder protocol (BlobDefs::kWritePlaceholderKey), under which a value equal to the placeholder sentinel is persisted as a bin_length -2 entry and a sentinel-prefixed real value is unescaped; any other write stores blob bytes verbatim, so user data can never be turned into a placeholder. The reader fails on placeholder entries unless the internal option BlobDefs::kEmitPlaceholderSentinelKey switches it to emit the sentinel for them, escaping stored values that start with the sentinel bytes so the fallback merge can never mistake real data for placeholders. BlobBunch keeps all max-sequence layers of a bunch, and the new BlobFallbackBatchReader resolves each row to the newest layer holding a real value: an explicitly written null wins over older layers, a row that is a placeholder in every layer degrades to a null blob while keeping its _ROW_ID and reporting -1 as its _SEQUENCE_NUMBER, resolved rows report their layer's sequence number, and row-range pushdown is honored including the row id ranges a layer does not cover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| /// | ||
| /// A union `SplitRead` to read multiple inner files to merge columns, note that this class | ||
| /// does not support filtering push down and deletion vectors, as they can interfere with the | ||
| /// process of merging columns. |
There was a problem hiding this comment.
Please add BlobFallbackBatchReader here.
| /// Both channels escape legitimate values so user bytes can never collide with the | ||
| /// sentinel: a real value whose bytes start with the sentinel travels with one extra copy | ||
| /// of the sentinel prepended, and the consuming end strips it (an exact match is a | ||
| /// placeholder, a longer match is unescaped). Sentinel bytes are never stored in blob files |
There was a problem hiding this comment.
The S + S escape/unescape protocol introduces quite a bit of complexity across the writer, BlobFileBatchReader, and BlobFallbackBatchReader, and the behavior is not easy to understand or maintain.
Could we simplify this by using a sufficiently distinctive fixed placeholder marker, for example _PAIMON_BLOB_PLACEHOLDER, and only checking for exact equality? This would remove the sentinel-prefix detection and the producer-side escape requirement. If we accept the negligible collision probability, the marker can be documented as an internal reserved value.
| } | ||
| PAIMON_ASSIGN_OR_RAISE(uint64_t file_row_id, | ||
| reader_->GetPreviousBatchFileRowId(idx_in_array)); | ||
| return first_row_id_.value() + file_row_id; |
There was a problem hiding this comment.
BlobFileBatchReader advertises SupportPreciseBitmapSelection() == true, so a row-range selection is applied directly inside the BLOB reader. However, GetPreviousBatchFileRowId() returns NotImplemented whenever the selection removes any rows. CompleteRowTrackingFieldsBatchReader relies on this method to synthesize _ROW_ID, so reading {"b0", "_ROW_ID", "_SEQUENCE_NUMBER"} with a proper subset of row ranges fails.
The reader already maintains target_blob_row_indexes_, which maps every selected position back to its original file row index. Please use that mapping for the previous batch instead of rejecting selected reads
| } else { | ||
| return file_schema_result.status(); | ||
| } | ||
| PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema> arrow_schema, |
There was a problem hiding this comment.
For this wrapper, we only need to determine whether _ROW_ID and _SEQUENCE_NUMBER are physically present. DataFileMeta.write_cols is sufficient for a BLOB file and pretending that NotImplemented means an empty schema.
Could we pass the BLOB file’s known physical field names from file_meta.write_cols into CompleteRowTrackingFieldsBatchReader? Self-describing formats can continue to query GetFileSchema(), while the BLOB path explicitly declares its physical fields. This keeps unrelated NotImplemented errors visible and avoids fabricating a full Arrow file schema from column names alone.
Purpose
Linked issue: close #452
A data-evolution partial update rewrites only the touched rows of a blob column and records every untouched row as a placeholder entry (
bin_length -2, no data bytes). This PR makes the whole chain understand such entries:blob.internal.write-placeholder(BlobDefs::kWritePlaceholderKey,falseby default), enabled only for data-evolution partial updates, i.e. blob-only column writes of a table with data evolution enabled. Under the protocol, a value equal to the 9-byte placeholder sentinel (0x01+"BLOBPLHD", exposed asBlobDefs::PlaceholderSentinelView()) is persisted as abin_length -2entry without payload bytes, and a sentinel-prefixed real value is unescaped by stripping one leading sentinel copy. Outside the protocol the writer never interprets values, so arbitrary user bytes can never be turned into a placeholder entry.blob.internal.emit-placeholder-sentinel(BlobDefs::kEmitPlaceholderSentinelKey,falseby default, set only by the fallback read path) switches it to emit the sentinel bytes for-2entries, escaping stored values that start with the sentinel by prepending one extra sentinel copy — so after the batch has passed through schema-mapping readers, the fallback merge can identify placeholders without ever mistaking real data for them. Sentinel bytes are never stored in blob files and never returned to users.BlobBunch: keeps all blob files of a bunch, where files sharing amax_sequence_numberform one non-overlapping layer; overlaps across layers are the expected shape of partial updates. When every file shares a singlemax_sequence_number,SequentialReadOptimize()keeps the existing concat fast path.BlobFallbackBatchReader(new, aligned with Java'sBlobFallbackRecordReader/AllPlaceholdersRecordReader): merges the layers of one blob bunch ordered bymax_sequence_numberdescending; row-id ranges a layer does not cover are padded with placeholder gap segments so all layers step in lockstep; each output row takes the newest non-placeholder layer, and an explicitly written null wins over older layers. A row that is a placeholder in every layer degrades to a null blob while keeping its_ROW_IDand reporting-1as its_SEQUENCE_NUMBER; resolved rows report their layer's sequence number. Row-range pushdown is honored per layer, including inside gaps.DataEvolutionSplitRead: builds the fallback reader when a blob bunch spans multiple sequence layers, and passes the internal format options throughAbstractSplitRead::CreateRawFileReaders(explicitextra_format_optionsparameter, no default arguments).Tests
blob_format_writer_test.cpp: golden-bytes layout of placeholder entries aligned with the Java writer (TestWritePlaceholderGoldenBytes), strict vs placeholder-aware read modes including descriptor mode (TestReadPlaceholderStrictAndAwareModes), selection-bitmap reads over placeholders (TestReadPlaceholderWithSelectionBitmap), sentinel-equal user bytes stored verbatim outside placeholder mode (TestSentinelBytesVerbatimWithoutPlaceholderMode), and the escape round trip (TestEscapedSentinelRoundTrip).blob_fallback_batch_reader_test.cpp(new): fallback merge across layers with batch-size and file-batch-size sweeps — basic fallback, leading/middle/trailing gaps and disjoint gap ranges, all-placeholder rows degrading to null, null-wins semantics, escaped sentinel values (TestEscapedSentinelIsNotPlaceholder), three layers, row-tracking projections{blob, _ROW_ID, _SEQUENCE_NUMBER}/{blob, _ROW_ID}/{blob, _SEQUENCE_NUMBER}including the all-placeholder-1semantics (TestRowTrackingFieldsPreserved), misaligned-group and creation-argument failures.data_evolution_split_read_test.cpp:BlobBunchkeeps all sequence layers, rejects overlaps within one layer, and reportsRowCount/SequentialReadOptimizeaccordingly.blob_table_inte_test.cpp(IT): end-to-end partial-update reads — placeholder fallback with null overwrite,_ROW_IDalignment and ablob_as_descriptorread-mode variant (TestDataEvolutionBlobPartialUpdateFallback), multi-layer layouts with per-row and gap-crossing row-range reads (TestDataEvolutionBlobPartialUpdateMultipleLayers), a compacted four-layer scenario mirroring Java'sBlobUpdateTest.testReadCompactedBlobSequenceGroupswith per-row range reads (TestDataEvolutionBlobPartialUpdateCompactedLayers), row-range pushdown over updated and untouched rows (TestDataEvolutionBlobPartialUpdateWithRowRanges), a sequence layer covering only a strict subrange of the row-id range with{b0, _ROW_ID, _SEQUENCE_NUMBER}projection (TestDataEvolutionBlobPartialUpdateRowTrackingWithSubrangeLayer), an all-placeholder row keeping its_ROW_IDwith_SEQUENCE_NUMBER -1(TestDataEvolutionBlobPartialUpdateAllPlaceholderRowTracking), and a user blob whose bytes exactly equal the sentinel surviving write, read and a later fallback unchanged (TestBlobValueEqualToPlaceholderSentinelBytes).API and Format
include/paimon.bin_length -2marks a placeholder entry occupying no file space, aligned with Java'sBlobFormatWriter.PLACE_HOLDER_LENGTH. Readers without placeholder support already reject such entries with an explicit error, and files without placeholders are unaffected.falseby default:blob.internal.write-placeholder(set only for data-evolution blob-only column writes) andblob.internal.emit-placeholder-sentinel(set only by the data-evolution fallback read path).Documentation
Internal read/write behavior of data-evolution partial updates; no user-facing documentation change.
Generative AI tooling
Generated-by: Claude Code (Claude Fable 5)
🤖 Generated with Claude Code