Skip to content

Integrate bounded CDC deduplication with durable publication #82

Description

@flyingrobots

Outcome

Provide one bounded production path from an unknown-length source through the registered Keep CDC profile, chunk verification and deduplication, immutable segment publication, catalog admission, and an exact receipt.

This is the Keep-native counterpart of default CAS deduplication. It preserves keep.fastcdc-gear64/v1; it does not add Buzhash merely to imitate another implementation.

Dependencies

Invariant impact

For a named logical identity, the pipeline must publish exactly the admitted bytes or refuse. Deduplication, batching, concurrency, and backpressure must not change BlobId, ChunkId, LayoutId, publication order, recovery behavior, or error precision.

Acceptance criteria

  • Unknown-length ingestion remains single-pass and bounded by explicit count-and-byte limits.
  • Existing chunks are reused only after exact identity and representation verification.
  • Missing chunks stream into immutable segment stages without materializing the whole blob.
  • Layout and complete BlobId reconstruction are verified before visible admission.
  • Publication returns a receipt binding profile, blob, layout, segment, catalog generation, and exact byte counts.
  • Every write boundary has deterministic short-write, interruption, process-death, and restart evidence.
  • Benchmarks report throughput, peak memory, allocation, sync count, and deduplication ratio.
  • The reference adapter and durable adapter satisfy one backend-neutral ingestion contract where their durability claims overlap.

Non-goals

  • A new CDC algorithm.
  • Convergent encryption.
  • Application naming, retention policy, or causal meaning.
  • Hiding the fact that the non-durable reference adapter remains non-durable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:catalogCatalog generations, publication, and lookuparea:chunkingContent-defined chunking and profilesarea:segmentPhysical immutable segment representationkind:benchmarkMeasured performance evidencekind:implementationProduction implementation workkind:testTest infrastructure or failure-mode evidencepriority:P1Must fix before closing out the current milestone

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions