Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,10 @@ jobs:
# `cargo build` to verify the floor still works — catches any new code
# that requires a newer Rust without an accompanying release-notes call-out.
#
# I61 (ISSUES.md, 2026-05-22): `-p chisel` scopes the build to the
# library crate ONLY. The MSRV promise is about what users see when
# they depend on `chisel` — bench is internal tooling whose heavy
# I61 (ISSUES.md, 2026-05-22): `-p chisel-storage` scopes the build to the
# library crate ONLY (published package name; the library itself is still
# `chisel` via [lib].name in Cargo.toml). The MSRV promise is about what
# users see when they depend on it — bench is internal tooling whose heavy
# deps (criterion, rusqlite, redb, rand → getrandom, clap → clap_lex,
# ...) are now adopting `edition2024` faster than we want to bump our
# floor. Bench can lag the library's MSRV without affecting any
Expand All @@ -115,7 +116,7 @@ jobs:
# above the 1.82 library floor for the same reason bench/python do (I61).
# The MSRV promise is about the published library, which this verifies.
- name: Build chisel library with MSRV toolchain
run: cargo build --verbose -p chisel
run: cargo build --verbose -p chisel-storage

# I58 (ISSUES.md, 2026-05-22): a dedicated `bench-tests` job once
# ran `cd bench && cargo test --verbose` because bench/ was a
Expand Down
60 changes: 60 additions & 0 deletions .github/workflows/publish-crate.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Publish crate

# Fires on a published GitHub Release (a deliberate "ship this" action),
# not on every tag push -- so a stray or draft tag never triggers a publish.
on:
release:
types: [published]

jobs:
# Mirrors wheels.yml's cargo-test-gate: a GitHub Release can in principle be
# cut against a commit that never went through ci.yml (e.g. a hand-pushed
# tag), so re-verify test + audit here rather than trusting the release
# event alone. crates.io has no "unpublish", only yank -- this gate exists
# specifically because that makes the publish step below unusually
# expensive to get wrong.
cargo-test-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- run: cargo test
- name: Install cargo-audit
run: cargo install cargo-audit --locked
- run: cargo audit

publish:
needs: cargo-test-gate
runs-on: ubuntu-latest
# Requires a "release" GitHub Actions environment (Settings > Environments)
# with protection rules (e.g. required reviewers) -- this is what actually
# gates the publish on human approval; the workflow trigger alone does not.
environment: release
permissions:
id-token: write # required to mint the OIDC token trusted-publishing exchanges
contents: read
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2

# Catches packaging-level issues (stray large files, missing includes)
# that `cargo test`/`clippy` don't -- cheap insurance before an
# irreversible publish.
- name: Dry-run package verification
run: cargo publish -p chisel-storage --dry-run

# Requires a Trusted Publisher Configuration for "chisel-storage" to
# already exist on crates.io (crate owner sets this up in the crate's
# settings after its first, manual publish -- crates.io has no
# pending-publisher mechanism for a crate's very first release, unlike
# PyPI). See README.md's release process notes.
- name: Authenticate with crates.io
id: auth
uses: rust-lang/crates-io-auth-action@v1

- name: Publish to crates.io
run: cargo publish -p chisel-storage
env:
CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
37 changes: 37 additions & 0 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,40 @@ jobs:
with:
name: sdist
path: dist/*.tar.gz

# Only runs when this workflow was itself tag-triggered (push: tags: v*),
# never on a manual workflow_dispatch smoke-build -- so re-running the
# build by hand can never accidentally publish.
publish-pypi:
if: github.event_name == 'push'
needs: [wheels, sdist]
runs-on: ubuntu-latest
# Requires a "release" GitHub Actions environment (Settings > Environments)
# with protection rules (e.g. required reviewers). Shares the same
# environment name as publish-crate.yml's crates.io job so one set of
# protection rules gates both registries.
environment: release
permissions:
id-token: write # required for PyPI Trusted Publishing (OIDC)
steps:
# Collects wheels-ubuntu-latest, wheels-ubuntu-24.04-arm,
# wheels-macos-latest, and sdist into one flat dist/ directory --
# gh-action-pypi-publish expects everything under one packages-dir.
- uses: actions/download-artifact@v8
with:
pattern: "wheels-*"
path: dist
merge-multiple: true
- uses: actions/download-artifact@v8
with:
name: sdist
path: dist

# Requires a Trusted Publisher configured on PyPI for "chisel-storage"
# (Project > Publishing), naming this repo, this workflow file, and the
# "release" environment -- unlike crates.io, PyPI supports configuring
# this as a "pending publisher" before the project's first-ever
# release, so this job can carry that first release too.
- uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: dist
54 changes: 27 additions & 27 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 13 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@
# Each member retains its own [package] block (name, version, edition,
# rust-version, etc.) — workspace inheritance via [workspace.package]
# is a separate refactor (cosmetic, not functional). The path-deps
# `chisel = { path = ".." }` in python/Cargo.toml and bench/Cargo.toml
# work unchanged.
# `chisel-storage = { path = ".." }` in python/Cargo.toml and bench/Cargo.toml
# work unchanged (the root package is named "chisel-storage" on crates.io;
# see [lib] below for why `use chisel::...` is unaffected).
[workspace]
members = [".", "python", "bench"]
# `default-members` excludes `python` from `cargo build` / `cargo test`
Expand All @@ -32,8 +33,8 @@ default-members = [".", "bench"]
resolver = "2"

[package]
name = "chisel"
version = "0.1.0"
name = "chisel-storage"
version = "1.0.0"
edition = "2021"
# MSRV: bump only with a release-notes call-out; CI pins this same version.
# 1.82 is a conservative pin; the true stdlib floor is ~1.74 (io::Error::other,
Expand All @@ -53,6 +54,14 @@ keywords = ["database", "storage", "embedded", "transactional", "shadow-paging"]
# radix tree / freemap / handle table types we expose.
categories = ["database-implementations", "data-structures"]

# Published on crates.io as "chisel-storage" (the plain "chisel" name is
# already taken by an unrelated project). The library/import name stays
# "chisel" — Cargo resolves `use chisel::...` for any consumer depending on
# `chisel-storage = "..."` from this [lib].name override alone, no renaming
# needed on the consumer's side.
[lib]
name = "chisel"

[dependencies]
xxhash-rust = { version = "0.8", features = ["xxh3"] }
libc = "0.2"
Expand Down
20 changes: 14 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
# Chisel

[![CI](https://github.com/pgexperts/chisel/actions/workflows/ci.yml/badge.svg)](https://github.com/pgexperts/chisel/actions/workflows/ci.yml)
[![Crates.io](https://img.shields.io/crates/v/chisel-storage.svg)](https://crates.io/crates/chisel-storage)
[![docs.rs](https://docs.rs/chisel-storage/badge.svg)](https://docs.rs/chisel-storage)
[![PyPI](https://img.shields.io/pypi/v/chisel-storage.svg)](https://pypi.org/project/chisel-storage/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A transactional, crash-durable key-value storage engine written in Rust. Chisel uses **shadow paging** (copy-on-write) to guarantee that the database file is always in a consistent state. There is no write-ahead log and no recovery procedure — after a crash, you just open the file and it's correct.

Chisel is designed for single-writer embedded use: one process holds the file via `flock`, all mutations go through `&mut self`, and the API is synchronous. A PyO3 binding ships alongside the Rust crate; see [`python/README.md`](python/README.md).

## Status

Pre-1.0. Current release: `0.1.0`. The API is stable-by-intent but subject to revision until 1.0 ships. The on-disk format is likewise pre-stable; see [On-disk format compatibility](#on-disk-format-compatibility) for the 1.0-and-onward promise.
1.0. Current release: `1.0.0`. Both the API and the on-disk format are now frozen per the compatibility promise in [On-disk format compatibility](#on-disk-format-compatibility).

## Features

Expand All @@ -29,10 +35,12 @@ Add to your `Cargo.toml`:

```toml
[dependencies]
chisel = "0.1"
chisel-storage = "1.0"
```

(While Chisel is pre-1.0 and not yet on crates.io, use a path or git dependency: `chisel = { path = "path/to/chisel" }`.)
The published crate is named `chisel-storage` (plain `chisel` was already taken); the library itself is still `chisel`, so your code imports it as `use chisel::{Chisel, Options};` regardless.

(Not yet published to crates.io; use a path or git dependency in the meantime: `chisel-storage = { path = "path/to/chisel" }`.)

## Quick Start

Expand Down Expand Up @@ -362,11 +370,11 @@ Encryption introduces the second major: an encrypted database is stamped MAJOR =

Write safety across minors is a narrower guarantee: a binary at MINOR = *m* opening a file at MINOR = *m' > m* cannot safely commit without risking overwriting fields it doesn't know about. The open gate is MAJOR-only by design, so minor variants coexist — same-major files of any minor open successfully, and the chunk-tags MINOR = 1 variant is the first such case. The write-refusal arm (refuse writes when file MINOR > binary MINOR) is implemented (I29): opening a newer-minor file forces the handle read-only, so any mutation returns `ReadOnlyMode` rather than risking a write that clobbers fields the binary doesn't know about. The post-1.0 cross-minor read-compatibility guarantee is absolute; write-compatibility requires binary MINOR ≥ file MINOR.

### Pre-1.0 caveat
### Format history

Until Chisel reaches 1.0, the on-disk format may change between pre-release builds without a major-version bump. Any such pre-1.0 change will be called out in release notes. The first 1.0 release freezes the plaintext format at MAJOR = 1 for the entire 1.x line; encrypted databases carry MAJOR = 2 (see above), and each major's on-disk format is sacred within that major.
Before 1.0, the on-disk format changed between pre-release builds without a major-version bump; such changes were called out in release notes. The 1.0 release freezes the plaintext format at MAJOR = 1 for the entire 1.x line; encrypted databases carry MAJOR = 2 (see above), and each major's on-disk format is sacred within that major going forward.

Files written by prior development builds (pre-1.0 flat `format_version`, which decodes as MAJOR = 0) are rejected at open time — recreate the database. No production-grade migration is provided for pre-release files.
Files written by prior pre-1.0 development builds (flat `format_version`, which decodes as MAJOR = 0) are rejected at open time — recreate the database. No production-grade migration is provided for pre-1.0 files.

## How durability works

Expand Down
4 changes: 2 additions & 2 deletions bench/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "chisel-bench"
version = "0.1.0"
version = "1.0.0"
edition = "2021"
# MSRV mirrors the root crate (1.82). The path-dep on chisel inherits its
# floor; bench-specific deps (criterion, rusqlite, clap) all support older
Expand All @@ -16,7 +16,7 @@ repository = "https://github.com/pgexperts/chisel"
publish = false

[dependencies]
chisel = { path = ".." }
chisel-storage = { path = ".." }
redb = "2"
rusqlite = { version = "0.31", features = ["bundled"] }
rand = "0.8"
Expand Down
4 changes: 2 additions & 2 deletions python/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "chisel-py"
version = "0.1.0"
version = "1.0.0"
edition = "2021"
# MSRV mirrors the root crate (1.82 conservative pin; the root crate's true
# stdlib floor is ~1.74 from io::Error::other). PyO3 has a lower MSRV of its
Expand All @@ -23,7 +23,7 @@ name = "_chisel"
crate-type = ["cdylib"]

[dependencies]
chisel = { path = ".." }
chisel-storage = { path = ".." }
# I75 (ISSUES.md, 2026-05-22): bumped from 0.22 to clear
# RUSTSEC-2025-0020 (Risk of buffer overflow in PyString::from_object;
# patched in 0.24.1+). Bumped again to 0.29 to clear RUSTSEC-2026-0176
Expand Down
13 changes: 11 additions & 2 deletions python/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,23 @@
# chisel (Python binding)

[![PyPI](https://img.shields.io/pypi/v/chisel-storage.svg)](https://pypi.org/project/chisel-storage/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](../LICENSE)

Python binding for [Chisel](..), a transactional slot-based storage engine written in Rust with shadow-paging durability. The engine is embedded, single-writer, and fully synchronous.

## Status

Pre-1.0. Current release: `0.1.0`. The API is stable-by-intent but subject to revision until 1.0 ships. The on-disk format is likewise pre-stable; see [On-disk format compatibility](#on-disk-format-compatibility) for the 1.0-and-onward promise.
1.0. Current release: `1.0.0`. Both the API and the on-disk format are now frozen per the compatibility promise in the [root README](../README.md#on-disk-format-compatibility).

## Install

Not yet published to PyPI; build locally with [maturin](https://www.maturin.rs/):
```bash
pip install chisel-storage
```

The distribution is named `chisel-storage` (plain `chisel` was already taken on PyPI); the importable module is still `chisel`, so your code reads `import chisel` regardless.

Not yet published to PyPI; build locally with [maturin](https://www.maturin.rs/) in the meantime:

```bash
pip install maturin
Expand Down
14 changes: 12 additions & 2 deletions python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,12 @@ requires = ["maturin>=1.5,<2.0"]
build-backend = "maturin"

[project]
name = "chisel"
version = "0.1.0"
# Published on PyPI as "chisel-storage" (plain "chisel" is already taken by
# an unrelated package). The importable module stays `chisel` — module-name
# and python-source below are untouched by this, so `import chisel` keeps
# working regardless of the distribution name pip installs.
name = "chisel-storage"
version = "1.0.0"
description = "Python binding for the Chisel transactional storage engine"
readme = "README.md"
requires-python = ">=3.11"
Expand All @@ -20,6 +24,12 @@ classifiers = [
"Operating System :: MacOS",
]

[project.urls]
Homepage = "https://github.com/pgexperts/chisel"
Repository = "https://github.com/pgexperts/chisel"
Documentation = "https://github.com/pgexperts/chisel/blob/main/python/README.md"
Issues = "https://github.com/pgexperts/chisel/issues"

[project.optional-dependencies]
test = ["pytest>=8", "hypothesis>=6"]

Expand Down
Loading