Repository navigation

Add batched ingestion with local state and replay-safe Lance output - #325

Open

beinan wants to merge 16 commits into

mainfrom

codex/streaming-ingestion

Commits on Oct 6, 2026
- Add experimental bounded streaming ingestion primitives
  Beinan Wang
  committed
  829cb5a
- Add Lance sink with atomic ingestion coverage and version fencing
  Beinan Wang
  committed
  976361e
- Bound durable ingestion backlog by required consumer progress
  Beinan Wang
  committed
  5f34060
- Allow coalesced session checkpoint reduction
  Beinan Wang
  committed
  034819e
Commits on Oct 7, 2026
- Recover cold ingestion sessions from checkpoints and committed WAL
  Beinan Wang
  committed
  09590f7
- Scope macro-generated must-use lint allowance to async ingestion traits
  Beinan Wang
  committed
  d14c089
- Run session alignment lanes with supervised pipeline stages
  Beinan Wang
  committed
  c9344a1
- Scope async trait generated must-use lint on store registry
  Beinan Wang
  committed
  6f6f011
- Resolve source receipts through independently indexed WAL history
  Beinan Wang
  committed
  2ff5ee0
- Admit continuous source requests with stable receipt recovery
  Beinan Wang
  committed
  d366a73
- Keep batched source receipt lookup Send across task boundaries
  Beinan Wang
  committed
  4474e47
- Allow pipeline memory pools above four GiB without serializing admission
  Beinan Wang
  committed
  9860fb0
- Release aligned input and history reservations before WAL publication
  Beinan Wang
  committed
  669fc9f
- Preserve bulk source batches through WAL publication
  Beinan Wang
  committed
  9d61c1f
- Persist local alignment cells and typed output in atomic Lance batches
  Beinan Wang
  committed
  7e3112d
- Validate immutable Lance output ranges and bounded local recovery batches
  Beinan Wang
  committed
  8f4583f

Navigation Menu

Repository navigation

Add batched ingestion with local state and replay-safe Lance output - #325

Add batched ingestion with local state and replay-safe Lance output#325
beinan wants to merge 16 commits into
mainfrom
codex/streaming-ingestion

Commits on Oct 6, 2026

Add experimental bounded streaming ingestion primitives

Add Lance sink with atomic ingestion coverage and version fencing

Bound durable ingestion backlog by required consumer progress

Allow coalesced session checkpoint reduction

Commits on Oct 7, 2026

Recover cold ingestion sessions from checkpoints and committed WAL

Scope macro-generated must-use lint allowance to async ingestion traits

Run session alignment lanes with supervised pipeline stages

Scope async trait generated must-use lint on store registry

Resolve source receipts through independently indexed WAL history

Admit continuous source requests with stable receipt recovery

Keep batched source receipt lookup Send across task boundaries

Allow pipeline memory pools above four GiB without serializing admission

Release aligned input and history reservations before WAL publication

Preserve bulk source batches through WAL publication

Persist local alignment cells and typed output in atomic Lance batches

Validate immutable Lance output ranges and bounded local recovery batches