-
Notifications
You must be signed in to change notification settings - Fork 0
Integrate bounded CDC deduplication with durable publication #82
Copy link
Copy link
Open
Labels
area:catalogCatalog generations, publication, and lookupCatalog generations, publication, and lookuparea:chunkingContent-defined chunking and profilesContent-defined chunking and profilesarea:segmentPhysical immutable segment representationPhysical immutable segment representationkind:benchmarkMeasured performance evidenceMeasured performance evidencekind:implementationProduction implementation workProduction implementation workkind:testTest infrastructure or failure-mode evidenceTest infrastructure or failure-mode evidencepriority:P1Must fix before closing out the current milestoneMust fix before closing out the current milestone
Description
Metadata
Metadata
Assignees
Labels
area:catalogCatalog generations, publication, and lookupCatalog generations, publication, and lookuparea:chunkingContent-defined chunking and profilesContent-defined chunking and profilesarea:segmentPhysical immutable segment representationPhysical immutable segment representationkind:benchmarkMeasured performance evidenceMeasured performance evidencekind:implementationProduction implementation workProduction implementation workkind:testTest infrastructure or failure-mode evidenceTest infrastructure or failure-mode evidencepriority:P1Must fix before closing out the current milestoneMust fix before closing out the current milestone
Outcome
Provide one bounded production path from an unknown-length source through the registered Keep CDC profile, chunk verification and deduplication, immutable segment publication, catalog admission, and an exact receipt.
This is the Keep-native counterpart of default CAS deduplication. It preserves
keep.fastcdc-gear64/v1; it does not add Buzhash merely to imitate another implementation.Dependencies
Invariant impact
For a named logical identity, the pipeline must publish exactly the admitted bytes or refuse. Deduplication, batching, concurrency, and backpressure must not change
BlobId,ChunkId,LayoutId, publication order, recovery behavior, or error precision.Acceptance criteria
BlobIdreconstruction are verified before visible admission.Non-goals