buzzcrow · buzzcrow · Sep 28, 2026 · Sep 28, 2026 · Sep 28, 2026 · Sep 28, 2026
diff --git a/.agents/skills/doc-design/SKILL.md b/.agents/skills/doc-design/SKILL.md
@@ -21,7 +21,7 @@ stable IDs such as `I1`.
 Describe current state only. Omit requirement numbers, change history,
 before/after prose, file paths, and line numbers. Refer to searchable symbols.
 Keep architecture in root docs, detail in sub-designs, and operations in the
-user guide; link rather than repeat.
+relevant design or component README; link rather than repeat.
 
 When explicitly promoting a working draft, remove temporary scaffolding and
 requirement references, rewrite as current state, update `doc/doc_index.md`,

diff --git a/.agents/skills/doc/SKILL.md b/.agents/skills/doc/SKILL.md
@@ -15,7 +15,7 @@ Start at `doc/doc_index.md`; open only the row and section matching the task.
 - `doc/design/<area>/design-crowdb-<area>.md`: architecture and rationale.
 - `doc/design/<area>/design-crowdb-<area>-<topic>.md`: permanent topic detail.
 - `doc/design/kv/kv-*-flow-analysis.md`: permanent KV path analysis.
-- `doc/user-manual/user-guide.md`: operations; generated HTML is not hand-edited.
+- `doc/design/`: permanent architecture and operational behavior.
 - `doc/backlog/`: requirement index and analysis.
 - `doc/working/`: implementation plans and explicitly requested design drafts.
 
@@ -29,9 +29,6 @@ Start at `doc/doc_index.md`; open only the row and section matching the task.
 - Split independent topics; delete working files when their work completes.
 - Prefer concrete, tight prose and raw-readable bullets. Remove filler,
   repetition, rhetorical openings, adjective lists, and excessive em dashes.
-- Rebuild `user-guide.html` with
-  `pixi run -- python doc/user-manual/build_html.py` whenever its Markdown
-  source changes.
 
 When the target is a design, backlog requirement, design draft, or working
 plan, use only its matching `/doc-*` guide instead of this router.
diff --git a/AGENTS.md b/AGENTS.md
@@ -45,4 +45,4 @@ and transport exposed through FFI.
   the backlog index only for selection, ordering, or status.
 - Pre-push or explicitly requested code review: `/review`.
 - Design questions: one section selected through `doc/doc_index.md`.
-- Operations/user behavior: `doc/user-manual/user-guide.md`.
+- Operations/user behavior: the relevant `doc/design/` or component README.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -35,5 +35,5 @@ The intended image is `crowdb/crowdb-iceberg:v0.1.0-dev`; no published digest is
 recorded yet. The GUI is not ready for this container. Multi-node deployment,
 production hardening and data-format upgrades are outside this release.
 
-See the [container guide](doc/user-manual/docker-single-node-user-guide.md) for
-supported startup, persistence, credentials and recovery behavior.
+See the container deployment files for supported startup, persistence,
+credentials and recovery behavior.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -76,9 +76,10 @@ pixi run test-single-node-container
 - Add dependencies through the owning package manager and avoid newly published
   versions until they have had time for ecosystem review.
 
-Start documentation work at `doc/doc_index.md`. Permanent architecture belongs
-under `doc/design/`, user behavior in `doc/user-manual/user-guide.md`, future
-contracts in `doc/backlog/`, and temporary execution plans in `doc/working/`.
+Start documentation work at `doc/doc_index.md`. Permanent architecture and
+operational behavior belong under `doc/design/` or the relevant component
+README, future contracts in `doc/backlog/`, and temporary execution plans in
+`doc/working/`.
 
 ## Code and tests
 

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/README.md b/README.md
@@ -6,166 +6,60 @@
 [![CI](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml/badge.svg)](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
 
-CROWDB is a distributed storage platform for objects, tables, and AI datasets.
-It owns the data path from S3, Iceberg, and native Dataset access through
-distributed metadata and chunk storage to disk—and eventually GPU memory.
-
-Version `0.1.0-dev` is the first development release being prepared for public
-evaluation. Use disposable data; production use and on-disk upgrade compatibility
-are not supported. Dataset and direct GPU delivery remain planned work.
-
-- Use **S3** for familiar object access.
-- Use **Iceberg** for native catalogs, tables, snapshots, and immutable files.
-- **Dataset** is planned for samples, shards, tensors, batches, and direct data access.
-
-## Three Layers, One Data Path
-
-```text
-                              Applications
-                    S3 tools   Table engines   AI runtimes
-                        |            |             |  |
-                        | HTTP       | HTTP        |  | native
-                        v            v             v  v
-+--------------------------------------------------------------------------+
-| LAYER 3 — ACCESS                                                         |
-|                                                                          |
-|   S3                     Iceberg                 Dataset                 |
-|   [implemented]          [implemented]           [design]                |
-|   HTTP objects           HTTP tables             HTTP + native client    |
-|                                                                          |
-|   Access Server serves HTTP. Dataset native access can bypass it.        |
-+------------------------------------+-------------------------------------+
-                                     |
-                                     v
-+--------------------------------------------------------------------------+
-| LAYER 2 — CHUNK                                                          |
-|                                                                          |
-|   Distributed structures:     Chunk Stream          Chunk-KV             |
-|  |  |  |
-|   Data path: chunk client -> Chunk I/O -> ChunkDB -> DiskIO -> DiskDB    |
-|  |  |
-|   Accelerated path: DiskIO buffer -- RDMA / GDS -------> GPU memory      |
-+------------------------------------+-------------------------------------+
-                                     |
-                                     v
-+--------------------------------------------------------------------------+
-| LAYER 1 — REUSABLE KV                                                    |
-|                                                                          |
-|   crowdb-kv   multi-group Paxos   WAL   Group 0   crowdb-tree   RPC      |
-|                                                                          |
-|   A standalone distributed layer—not the product-level data model.       |
-+--------------------------------------------------------------------------+
-```
+CROWDB is a distributed storage platform for Iceberg tables, AI datasets, and
+S3 objects. Each access model keeps its own semantics while sharing one storage
+core for distributed state, placement, protection, streaming, and recovery.
 
-## Why CROWDB?
-
-Storage systems are usually assembled by stacking one system on another: table
-metadata over object storage, dataset libraries over table or object APIs, and
-new accelerators behind paths designed for disks and CPUs. Every boundary adds
-another namespace, lifecycle, RPC, copy, and recovery model. When that boundary
-becomes the bottleneck, the layers above it can only work around it.
-
-CROWDB exists to own the complete data path. S3 objects, Iceberg tables, and AI
-datasets are intended as native access models over the same distributed storage core. They
-share durability, placement, protection, and reclamation without pretending
-that one model is merely a convention inside another.
-
-That control matters because both hardware and workloads keep changing. NVMe,
-RDMA, GPUDirect Storage, and accelerator offload reshape more than one isolated
-module. AI training and inference also need data to reach GPU memory without an
-HTTP gateway or client CPU becoming the permanent middleman. Supporting those
-changes cleanly requires control from protocol semantics down to buffers and
-disk layout.
-
-The goal is a storage foundation that can evolve as one system: simple enough
-to reason about, fast enough to justify owning the stack, and composed of
-layers that remain useful independently.
-
-## Technical Foundation
-
-- **Parallel consensus:** crowdb-kv runs multiple independent Multi-Paxos slots
-  concurrently, with WAL durability, lease reads, and pluggable engines.
-- **One protected chunk layer:** bounded streaming, mirrored small data,
-  strip-level erasure coding, shard repair, placement, and reclamation serve
-  every access model.
-- **Chunk-based distributed structures:** Chunk Stream provides durable ordered
-  append. Chunk-KV uses range partitions that split and rebalance online while
-  reads and writes continue.
-- **Native access models:** S3, Iceberg, and Dataset share the core without
-  being wrappers around one another. The planned Dataset model targets direct
-  topology access, RDMA and GPU delivery.
-
-## Where It Stands
-
-| Access model | Status      | What it means                                        |
-| ------------ | ----------- | ---------------------------------------------------- |
-| S3           | Implemented | Core HTTP object operations and bounded streaming    |
-| Iceberg      | Implemented | Native catalog, FileIO and core v1/v2/v3 semantics   |
-| Dataset      | Design      | HTTP, topology-aware native client, and GPU delivery |
+- **Iceberg:** native catalog and FileIO, implemented.
+- **S3:** core HTTP object operations, implemented.
+- **Dataset:** native access and direct GPU delivery, in design.
 
-The KV, tree, DiskDB, ChunkDB, chunk I/O, Chunk Stream, Chunk-KV, RPC,
-operations console, and core S3 foundation have working implementations. See
-the [backlog](doc/backlog/backlog.md) for current delivery scope.
-
-## Quick Start
-
-The first Linux amd64 image is being prepared for manual publication. Once
-`v0.1.0-dev` is published, start the Iceberg catalog and storage with Docker:
-
-```sh
-docker run -d --name crowdb-iceberg \
-  -p 127.0.0.1:80:80 \
-  crowdb/crowdb-iceberg:v0.1.0-dev
-```
-
-Follow the [single-node Docker guide](doc/user-manual/docker-single-node-user-guide.md)
-for startup checks, client credentials, persistent volumes and recovery.
-For source builds and development with Pixi, see
-[CONTRIBUTING.md](CONTRIBUTING.md).
-
-## Explore
-
-- [Access architecture](doc/design/access-server/design-crowdb-access-server.md)
-  — S3, Iceberg, Dataset, native access, and GPU delivery.
-- [Documentation index](doc/doc_index.md) — every permanent architecture and
-  subsystem design.
-- [User guide](doc/user-manual/user-guide.md) — setup, console, CLI, and
-  supported operations.
-- [Single-node Docker guide](doc/user-manual/docker-single-node-user-guide.md)
-  — preview image, volume, credentials, clients, and recovery.
-- [Backlog](doc/backlog/backlog.md) — what is implemented, in progress, and
-  planned.
-
-<details>
-<summary><b>Current cluster demos</b></summary>
-
-### Cluster lifecycle
+## Why CROWDB?
 
-Bootstrap a cluster, register physical topology, create stores and Paxos groups,
-and watch replicas elect a leader.
+Storage bottlenecks move—from disks to CPUs, networks, and data movement—but
+system boundaries tend to stay. A change that crosses a metadata service, an
+object gateway, and a separate storage engine can become a negotiation between
+systems rather than an improvement to one data path.
 
-<video src="https://github.com/user-attachments/assets/974d4a44-2446-462e-a9d0-9d9a82d07146" autoplay muted loop></video>
+CROWDB owns enough of that path to change it when workloads and hardware change.
+S3 objects, Iceberg tables, and planned AI datasets are native access models
+built over shared infrastructure, not conventions layered on top of one
+another. The goal is not to claim novelty for Paxos, WALs, trees, or erasure
+coding; it is to make their contracts agree on durability, placement, bounded
+buffers, and recovery.
 
-### KV operations
+Read the full motivation in
+[Why we’re building CROWDB](https://buzzcrow.github.io/blog/why-we-are-building-crowdb/).
 
-Put, get, scan, and delete through a selected distributed group.
+## Development preview
 
-<video src="https://github.com/user-attachments/assets/1fbdcf4e-255a-47e6-b4db-6a1fa0fb0df8" autoplay muted loop></video>
+Version `0.1.0-dev` is being prepared for public evaluation on Linux amd64.
+Use disposable data. Production use and on-disk upgrade compatibility are not
+supported, and Dataset and direct GPU delivery are not available yet.
 
-### Failover and replica management
+- [Project homepage](https://crowdb.dev/)
+- [Quick start](https://crowdb.dev/docs/quickstart/)
+- [Documentation](https://crowdb.dev/docs/)
+- [Architecture](https://crowdb.dev/docs/architecture/)
+- [Demos](https://crowdb.dev/demo/)
 
-Expand a group, remove its leader, and continue operations after re-election.
+## Repository
 
-<video src="https://github.com/user-attachments/assets/63298646-6eaa-4253-b96a-6f5eb420ad91" autoplay muted loop></video>
+The repository contains the storage implementation, tests, and source design
+documents. Start with:
 
-</details>
+- [Contributing](CONTRIBUTING.md) for the development environment and workflow.
+- [Documentation index](doc/doc_index.md) for subsystem designs.
+- [Backlog](doc/backlog/backlog.md) for current delivery scope.
+- [Single-node container](container/single-node-container/README.md) for image
+  development and packaging.
 
-## Notes on AI-Assisted Development
+## AI-assisted development
 
 The code in this project was written with AI assistance. The architecture,
 naming, module boundaries, and trade-offs remain human choices. AI is the
 compiler. The intent is mine.
 
 ## License
 
-See [LICENSE](LICENSE).
+Licensed under the [Apache License 2.0](LICENSE).
diff --git a/app/crowdb-access-server/Cargo.toml b/app/crowdb-access-server/Cargo.toml
@@ -16,7 +16,6 @@ test-util = []
 s3-e2e = ["s3"]
 iceberg-e2e = []
 s3 = [
-    "dep:crowdb-common",
     "dep:futures",
 ]
 
@@ -27,7 +26,7 @@ crowdb-access-iceberg = { path = "../../lib/crowdb-access-iceberg" }
 crowdb-access-s3 = { path = "../../lib/crowdb-access-s3" }
 crowdb-chunk-client = { path = "../../lib/crowdb-chunk-client" }
 crowdb-chunk-kv-client = { path = "../../lib/crowdb-chunk-kv-client" }
-crowdb-common = { path = "../../lib/crowdb-common/rust", optional = true }
+crowdb-common = { path = "../../lib/crowdb-common/rust" }
 crowdb-kv-client = { path = "../../lib/crowdb-kv-client" }
 crowdb-protocol = { path = "../../lib/crowdb-protocol" }
 futures = { version = "0.3", optional = true }

diff --git a/app/crowdb-access-server/conf/crowdb_access_server_config.toml b/app/crowdb-access-server/conf/crowdb_access_server_config.toml
@@ -0,0 +1,62 @@
+# Canonical configuration for the S3 and Iceberg access processes.
+# Credentials and bearer tokens are supplied through environment variables.
+
+[common]
+management_seeds = ["http://127.0.0.1:10000"]
+diskio_connections_per_endpoint = 2
+diskio_rpc_workers = 2
+
+[read]
+stream_window_bytes = 1048576
+stream_slots = 3
+global_stream_bytes = 268435456
+recovery_memory_bytes = 268435456
+
+[small_write]
+threshold_ratio = 0.9
+disk_block_bytes = 1048576
+conversion_enabled = true
+ec_data = 8
+ec_code = 4
+memory_budget_bytes = 1342177280
+queue_capacity = 1024
+min_pipelines = 1
+max_pipelines = 32
+max_batch_bytes = 1048576
+
+[s3]
+listen = "127.0.0.1:8081"
+tenant = "preview"
+region = "us-east-1"
+trusted_network = false
+small_object_limit = 8388608
+list_scan_items = 1001
+list_scan_bytes = 4194304
+continuation_ttl_seconds = 900
+native_budget_bytes = 268435456
+cleanup_backlog_limit = 10000
+ec_data = 8
+ec_code = 4
+max_chunk_size = 1073741824
+
+[iceberg]
+listen = "127.0.0.1:8181"
+native_budget_bytes = 268435456
+
+[iceberg.gc]
+enabled = false
+interval_ms = 1000
+step_bytes = 8388608
+step_ms = 1000
+page_items = 64
+page_bytes = 49152
+concurrency = 1
+kv_bytes = 67108864
+kv_requests = 128
+chunk_bytes = 8388608
+chunk_requests = 128
+retry_base_ms = 1000
+retry_max_ms = 60000
+corruption_attempts = 3
+minimum_retention_ms = 604800000
+catalogs = []