Task iceberge - #33

Merged

buzzcrow merged 253 commits into

mainfrom

Sep 28, 2026

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 22, 2026

Close completed Iceberg catalog foundation work
buzzcrow
committed
6cd0b85
Add bounded Iceberg namespace records and storage primitives
buzzcrow
committed
fbb3264
Separate Iceberg writer credentials from catalog administration
buzzcrow
committed
7748a40
Persist Iceberg namespace phases and bounded operation payloads
buzzcrow
committed
c03edc7
Implement recoverable namespace property updates and authoritative reads
buzzcrow
committed
118dcbc
Implement durable namespace creation and parent admission recovery
buzzcrow
committed
9237681
Implement bounded namespace drop and lifecycle recovery
buzzcrow
committed
ddacb5f
Harden namespace recovery bounds and verify drop restart replay
buzzcrow
committed
e2af397
Unify namespace property recovery with lifecycle helping
buzzcrow
committed
8f801ef
Recover abandoned namespace operations with bounded background sweeps
buzzcrow
committed
885098a
Add authoritative namespace listing with authenticated continuation tokens
buzzcrow
committed
780b6e1
Expose namespace reads with bounded complete-list spooling
buzzcrow
committed
e34b8d4
Integrate namespace mutation retries and record latency acceptance blocker
buzzcrow
committed
b6d8006
Repair stale namespace mappings with bounded background sweeps
buzzcrow
committed
9cfc354
Define canonical native Iceberg file locations
buzzcrow
committed
204419b
Publish immutable file authorities with bounded inline storage
buzzcrow
committed
63707e0

Commits on Sep 23, 2026

Add opt-in durable completion for shared chunk objects
buzzcrow
committed
9d9a64b
Stream native files through bounded chunk directories
buzzcrow
committed
c79f9a3
Record standard FileIO content-kind authority decision
buzzcrow
committed
5f93105
Add bounded table-scoped FileIO delegation credentials
buzzcrow
committed
35a0bee
Validate metadata JSON with bounded streaming workers
buzzcrow
committed
ba45004
Probe immutable file footer framing with bounded canonical reads
buzzcrow
committed
2e66a5e
Resolve ORC footer ranges from bounded postscripts
buzzcrow
committed
a83b466
Stream Avro container blocks with independent resource bounds
buzzcrow
committed
128c73a
Bound Avro block decompression independently of encoded size
buzzcrow
committed
02bfa7c
Resolve manifest sequence and row lineage inheritance in constant space
buzzcrow
committed
5ff38f5
Authenticate delegated file requests with isolated signature credentials
buzzcrow
committed
fe94480
Stream native file HTTP bodies with bounded pull admission
buzzcrow
committed
ae0f9a4
Restrict native file routing to exact immutable object operations
buzzcrow
committed
999b6d3
Checkpoint immutable file writers with resumable bounded digest state
buzzcrow
committed
6e20807
Read staged byte trees without inventing complete file formats
buzzcrow
committed
2d1a833
Assemble staged files through resumable bounded byte windows
buzzcrow
committed
fd03f45
Define bounded multipart session and completion phase invariants
buzzcrow
committed
135a157
Persist bounded multipart session and part authority records
buzzcrow
committed
4964939
Journal multipart part replacements behind a recoverable session fence
buzzcrow
committed
6ae0c8f
Freeze multipart selections and recover bounded assembly progress
buzzcrow
committed
7eb303c
Recover multipart sessions in bounded pages without physical deletion
buzzcrow
committed
8d9d1f0
Verify multipart checkpoint recovery through native storage restart
buzzcrow
committed
849a72e
Recover multipart publication from a frozen seal and retain exact outcomes
buzzcrow
committed
3891e0d
Schedule multipart recovery with independent session deadlines
buzzcrow
committed
a3fd24f
List multipart parts with bounded scans and stable session snapshots
buzzcrow
committed
9deae31
Journal global multipart credits and recover terminal releases
buzzcrow
committed
482e428
Bound native file upload streams and verify transport integrity
buzzcrow
committed
b7982b2
Validate bounded Avro writer schemas and streamed binary records
buzzcrow
committed
aca2815
Decode bounded Puffin footers and bind deletion vector descriptors
buzzcrow
committed
1ff6ebe
Stream and validate deletion vector bitmaps with bounded state
buzzcrow
committed
e51519a
Project bounded Avro scalar records by Iceberg field identity
buzzcrow
committed
30cea6f
Decode manifest list fields with versioned scalar semantics
buzzcrow
committed
414337a
Project nested Avro record fields with bounded traversal
buzzcrow
committed
b68b85b
Resolve typed manifest entries across bounded Avro blocks
buzzcrow
committed
af163d0
Decode bounded manifest equality IDs
buzzcrow
committed
48f940f
Validate manifest writer metadata from Avro headers
buzzcrow
committed
d7ae138
Bind manifest inheritance to list metadata
buzzcrow
committed
8336c6e
Decode and validate bounded manifest metric maps
buzzcrow
committed
5f9fbd8
Validate contextual manifest semantics and bound streaming reads
buzzcrow
committed
29fd721
Add bounded metadata projections with canonical fallback
buzzcrow
committed
e4b2fae
Persist multipart part times and enforce FileIO limits
buzzcrow
committed
93ae9c2
Validate manifest partition summaries against streamed entries
buzzcrow
committed
53c6ecf
Validate bounded Variant manifest statistics
buzzcrow
committed
5ab91bf
Bind snapshot deletion vectors to canonical data files
buzzcrow
committed
3dd263c
Connect native Iceberg FileIO HTTP routes
buzzcrow
committed
1499a62
Support standard Iceberg FileIO streaming clients
buzzcrow
committed
f6db416
Add delegated Iceberg credential wire contract
buzzcrow
committed
d190646
Validate selected unbound manifest streams
buzzcrow
committed
57f4ec5
Validate selected snapshot manifest enumeration
buzzcrow
committed
328278f
Read historical manifest lists using compatible field defaults
buzzcrow
committed
1a51250
Validate snapshot row ID allocation across manifests
buzzcrow
committed
1ff8482
Check bounded snapshot file and deletion vector identities
buzzcrow
committed
89da6d4
Decode bounded canonical Parquet footer metadata
buzzcrow
committed
a0e561b
Decode typed Parquet logical annotations
buzzcrow
committed
a9d0801
Validate Parquet column paths and flat value counts
buzzcrow
committed
5d8cdac
Bind selected Parquet footer reads to manifest descriptors
buzzcrow
committed
4e396f2
Clarify remaining selected format validation steps
buzzcrow
committed
076a9d3
Validate selected Parquet schemas and position delete pages
buzzcrow
committed
c6ccd66
Validate snapshot files and add generation-qualified table selection
buzzcrow
committed
f8600b7
Validate bounded table metadata documents and snapshot linkage
buzzcrow
committed
338f34c
Validate metadata defaults and add generation-consistent table reads
buzzcrow
committed
e7dad74
Validate selected metadata transitions and commit requirements
buzzcrow
committed
ba8247c
Decode bounded commit requests with closed update variants
buzzcrow
committed
088f9aa
Validate scalar table update parameters before evaluation
buzzcrow
committed
ccb4eb8
Preserve retained table definition identities across generations
buzzcrow
committed
0d3a198
Build manifest contexts from selected table definitions
buzzcrow
committed
744a5f7
Verify nonempty table metadata against official SDK fixtures
buzzcrow
committed
c3ced04
Compose bounded generation-qualified table HTTP reads
buzzcrow
committed
b99d05b
Verify table reads with the official REST catalog client
buzzcrow
committed
98cf0fb
Record remaining name mapping interoperability decision
buzzcrow
committed
62e370e
Evaluate ordered Iceberg metadata updates with bounded canonical output
buzzcrow
committed
79e40b0
Validate deletion vector replacement coverage across snapshots
buzzcrow
committed
553a5c4

Commits on Sep 24, 2026

Commits on Sep 25, 2026

Commits on Sep 26, 2026

Advance journal recovery and Iceberg reclamation
buzzcrow
committed
e4ae722
Reclaim retired multipart parts and expired catalog records
buzzcrow
committed
b7d19db
Bound reclamation discovery to file and part scopes
buzzcrow
committed
9575a62
Reclaim terminal multipart assembly checkpoints safely
buzzcrow
committed
32a07eb
Track native block ownership and finalize retired catalog reclamation
buzzcrow
committed
39cdf80
Add Iceberg reclamation controls and bounded runtime
buzzcrow
committed
dfc2ff3
Bound Iceberg reclamation I/O and verify capacity recovery
buzzcrow
committed
d847f8d
Record live reclamation fence decision
buzzcrow
committed
ab77283
Add durable candidate sealing admission
buzzcrow
committed
4d4ddf0
Admit durable purge work from background scheduler
buzzcrow
committed
8488382
Constrain Iceberg reclamation to inactive authorities
buzzcrow
committed
d22122c
Keep inactive reclamation resumable under pressure
buzzcrow
committed
f30597f
Bound Iceberg GC scheduling and validate foreground progress
buzzcrow
committed
4a6defa
Define pre-alpha development version
buzzcrow
committed
a0d85c0
Define single-node preview deployment
buzzcrow
committed
9a4ebee
Verify Iceberg reclamation capacity and draft file protection
buzzcrow
committed
fd18150
Preserve active Iceberg responses and correct native test ports
buzzcrow
committed
7dc1f99
Close validated Iceberg reclamation requirement
buzzcrow
committed
6f563e5
Define reusable monitor profile and single-node preview topology
buzzcrow
committed
0830556
Persist replay-safe monitor bootstrap manifests
buzzcrow
committed
5050ed3
Store private preview credentials and expose explicit retrieval
buzzcrow
committed
16fa6a4
Render single-node service configs from validated profile
buzzcrow
committed
3f1a16c
Add bounded reusable monitor health probes
buzzcrow
committed
ebbe027
Publish monitor status and health commands
buzzcrow
committed
6e4a1eb
Persist monitor lifecycle events and rotate child logs
buzzcrow
committed
1a4c93e
Supervise preview children and log recovery transitions
buzzcrow
committed
d7ef91f
Verify dependent restart and transient probe recovery
buzzcrow
committed
9ef5303
Reconcile preview KV groups through management APIs
buzzcrow
committed
3f67785
Log preview KV bootstrap transitions
buzzcrow
committed
e096114
Provision preview sparse disks without truncation
buzzcrow
committed
5c11434
Reset preview restart budget after stable health
buzzcrow
committed
61d7992
Reconcile preview hardware through Group Zero
buzzcrow
committed
c671b1b
Verify preview KV bootstrap against a real process
buzzcrow
committed
26d07d7
Fence preview listeners before child replacement
buzzcrow
committed
5aa0c38
Bind preview disks to a stable owner and data group
buzzcrow
committed
0c7a6e9
Verify preview disks through real DiskIO processes
buzzcrow
committed
6064dc8
Default Docker Iceberg access to host port 80
buzzcrow
committed
7694b2d
Probe preview chunk startup and preserve DiskIO read errors
buzzcrow
committed
07254e8
Listen for preview Iceberg on container port 80
buzzcrow
committed
2045f08
Align direct DiskIO reads for byte-range recovery
buzzcrow
committed
224ea28
Add replay-safe preview S3 credential bootstrap
buzzcrow
committed
13b2ac1
Add replay-safe preview Iceberg catalog bootstrap
buzzcrow
committed
184e8b1
Authenticate preview Iceberg health probes
buzzcrow
committed
1370309
Stage preview services through monitor run
buzzcrow
committed
77a6037
Define strict preview web configuration boundary
buzzcrow
committed
2cb6ad4
Probe monitor event-loop liveness over local socket
buzzcrow
committed
52aa8c1
Verify real preview Iceberg catalog restart
buzzcrow
committed
33fba5e
Fail closed on preview web process configuration
buzzcrow
committed
24bdf5c
Align preview plan with staged bootstrap
buzzcrow
committed
58882d7

Commits on Sep 27, 2026

Build single-node preview image
buzzcrow
committed
5517da7
Guard web startup against local topology replay
buzzcrow
committed
4160605
Record web write authority decision
buzzcrow
committed
b126a99
Set preview S3 and Web ports
buzzcrow
committed
63937ed
Add Docker web authority snapshot
buzzcrow
committed
e6b7381
Define authenticated Docker logical management
buzzcrow
committed
d583866
Wire management bearer to Docker web
buzzcrow
committed
ec19ed7
Track shared Group 0 authority cleanup
buzzcrow
committed
8f14275
Clarify one-way Group 0 bootstrap transfer
buzzcrow
committed
5194933
Verify chunk service authority during preview bootstrap
buzzcrow
committed
522a801
Log runtime authority bootstrap probes
buzzcrow
committed
1337fde
Fail closed on malformed bare-metal web registry
buzzcrow
committed
b3584a6
Enable preview container boot and manual release gates
buzzcrow
committed
ea94707
Complete preview container acceptance and Group 0 readiness
buzzcrow
committed
ae78116
Route web KV data through Group 0 discovery
buzzcrow
committed
61ef025
Resolve web KV endpoints from Group 0
buzzcrow
committed
881f37a
Project web stores from Group 0 topology
buzzcrow
committed
4cdbb88
Read web group lists from Group 0
buzzcrow
committed
3b424c0
Project web groups from Group 0 membership
buzzcrow
committed
355ee96
Seed nonmember KV servers from Group 0
buzzcrow
committed
15fc0fd
Restart KV nodes with retained Group 0 seeds
buzzcrow
committed
28d07f5
Bootstrap Group 0 before deploying KV peers
buzzcrow
committed
57c791b
Close Docker preview CI task
buzzcrow
committed
bc42d5b
Harden single-node container recovery and packaging
buzzcrow
committed
feec89b
Verify bounded Iceberg traversal at large logical sizes
buzzcrow
committed
42bb910
Bound monitor logs across child restarts
buzzcrow
committed
2e7e2dd
Show live monitor recovery when cluster authority is unavailable
buzzcrow
committed
6ecec11
Preserve authorized file reads after logical table drop
buzzcrow
committed
a27d413
Close native Iceberg catalog conformance milestone
buzzcrow
committed
bd8f28d
Package host-built artifacts into the single-node runtime image
buzzcrow
committed
3617616
Discover Group 0 for prestarted nonmember servers
buzzcrow
committed
c6badf3
Preserve KV registration identity across process restarts
buzzcrow
committed
47edb95
Record child exits after preceding probe failures
buzzcrow
committed
481000f
Record verified single-node runtime image acceptance
buzzcrow
committed
d669dae
Reject incomplete logical group and replica wiring
buzzcrow
committed
83bf5e5
Clean logical descendants after confirmed node deletion
buzzcrow
committed
adb4102
Report and clean failed replica creation
buzzcrow
committed
602587a
Use real-process election settings for restart coverage
buzzcrow
committed
e06f18b
Confirm conditional logical membership publication
buzzcrow
committed
a2a9105
Publish new group membership atomically
buzzcrow
committed
4963773
Reconcile lost logical deletion responses
buzzcrow
committed
28b6472

Commits on Sep 28, 2026

Separate launch policy from local and SSH process identity
buzzcrow
committed
9f07074
Wire launch policy into bare-metal Web and CLI lifecycle
buzzcrow
committed
5fb6e46
Read bare-metal snapshots from live Group 0 authority
buzzcrow
committed
77de69f
Share configured process controls across CLI service commands
buzzcrow
committed
4e2800e
Retain selected ORC validation as an independent follow-up
buzzcrow
committed
c21987a
Preserve system groups across interrupted bootstrap retries
buzzcrow
committed
bf29a26
Target the Iceberg Docker Hub repository and drop optional size experiments
buzzcrow
committed
217479a
Clarify Apport collection limits for container crashes
buzzcrow
committed
c6c9dde
Discover elected leaders for explicitly rejected conditional writes
buzzcrow
committed
5a67228
Use the versioned Docker Hub image in the end-user quick start
buzzcrow
committed
14b3d12
Use the DockerHub environment for container publication
buzzcrow
committed
957acf3
Rely on environment review to authorize container publication
buzzcrow
committed
882df87
Keep fault-injection routing stable across topology refreshes
buzzcrow
committed
da135a7
Record the confirmed Docker Hub release configuration
buzzcrow
committed
b55626e
Rename the Docker container publication workflow
buzzcrow
committed
4cfff64
Add project links and maintainer metadata to the Iceberg image
buzzcrow
committed
3a25333
Confirm bootstrap metadata without overwriting committed authority
buzzcrow
committed
beffafa
Defer container crash diagnostics to the deployment follow-up
buzzcrow
committed
63f68db
Scope runner artifact paths to container test steps
buzzcrow
committed
197702a
Validate upload MD5 while consuming decoded file bytes
buzzcrow
committed
b57409f
Reuse the namespace initialized by native file test setup
buzzcrow
committed
56af574
Track a complete review of shared access I/O and reclamation
buzzcrow
committed
cf37456
Specify twenty parts for the multipart acceptance case
buzzcrow
committed
dbaa0cd
Prepare public documentation for the first development release
buzzcrow
committed
68f5fbe
Install RPC connection handlers before starting reads
buzzcrow
committed
ac85574
Organize test tooling and verify Iceberg client suites
buzzcrow
committed
8ecc06f
Close the verified single-node container requirement
buzzcrow
committed
9ae946f