Skip to content

docs(python): correct the binding docs and make the type stub live (#97) - #132

Merged
Xof merged 2 commits into
fix/96-sqlite-cold-read-journal-modefrom
docs/97-python-binding-docs
Jul 31, 2026
Merged

docs(python): correct the binding docs and make the type stub live (#97)#132
Xof merged 2 commits into
fix/96-sqlite-cold-read-journal-modefrom
docs/97-python-binding-docs

Conversation

@Xof

@Xof Xof commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Closes #97 (6 findings, DESIGN/docs).

Stacked on #131#130#129main.

Every claim below was verified against a built wheel (maturin develop --release,
Python 3.13) before and after the change — including one of my own that turned
out to be wrong (see the last section).

PYTHON-2 — the savepoint comment described the opposite of the engine

savepoint.rs said the engine "pops the savepoint stack down to AND including
this savepoint, so the mark itself is gone after one successful call". It does
the opposite: rollback_to_inner ends with savepoints.truncate(idx + 1),
which retains the named savepoint, and the engine's own doc says so.

This mattered because the wrong claim was the stated justification for the
Python-side guard — "turns that into a cleaner, more specific
AlreadyFinishedError" — reading as if the guard merely re-labels an error the
engine would raise anyway. It doesn't: a second engine-level rollback_to
would succeed, so the guard is the only thing enforcing the documented
single-use contract. Comment corrected and the guard re-justified on its own
terms. Two test comments repeating the same wrong model are fixed.

PYTHON-4 — Tags section documented the pre-I126 API

The README described tag 0 as an "untagged" sentinel and tag() as returning
an int. Neither is true: require_tag rejects 0 with
ValueError: tag must be non-zero on every tagged method, and tag() returns
Optional[int]. python/chisel/__init__.py already documented it correctly, so
the two docs contradicted each other.

A reader following the old text writes if db.handles_with_tag(0): and gets an
unhandled ValueError, or if tx.tag(h) == 0: which never matches — silently
misclassifying every untagged handle.

PYTHON-5 — the only defrag example could not be run

DefragOptions(sparse_threshold=0.25, max_pages=0) raises TypeError before
reaching the engine; the field is max_values, and max_pages appears nowhere
in the codebase. Also dropped the pointer to "DefragOptions.max_pages's
docstring" (does not exist) and the claim that defrag "lives on the Chisel
object, not the Transaction object" — PyTransaction::defrag exists and is
tested by test_transaction_defrag_mid_tx.

PYTHON-6 — the type stub was inert; now it is checked in CI

Named chisel/chisel.pyi, PEP 561 resolves the stub to module chisel.chisel,
which does not exist. No checker ever read its 244 lines, and drift accumulated
undetected — __exit__ was declared -> None in all three classes while every
implementation returns bool (db.rs:311, transaction.rs:92,
savepoint.rs:65, each Ok(false)).

  • renamed to chisel/__init__.pyi, pyproject include and header comment updated
  • fixed the three __exit__ return types
  • added a CI step that type-checks it, which is what stops it going inert again

Verified: mypy now resolves import chisel through the stub (it reports errors
against the stub's declarations, which it previously could not see) and is
--strict clean at 3.13. The step is pinned to --python-version 3.13 because
the stub uses collections.abc.Buffer (PEP 688, 3.12+) while the package still
runs on 3.11 — the stub's content doesn't vary by interpreter, so one pinned
check is enough.

PYTHON-8 — Thread safety forbade what the test suite certifies

The README said two threads "must never call into the same Chisel at the same
time", while test_two_thread_mutex_contention runs 1600 concurrent read()
calls across two threads and asserts no error, no corruption, no poison.
Meanwhile the property that actually bites was documented only in a Rust
comment users never see.

Rewritten to state what holds: calls serialize (GIL + Mutex) so concurrency
cannot corrupt; transaction state is what must not be shared across threads;
and since only open() releases the GIL, a long commit or defrag() blocks
every Python thread in the process — not just ones touching this database.

PYTHON-9 — encryption was named only in the error tables

The README listed NoEncryptionKeyError, InvalidEncryptionKeyError,
EncryptionNotSupportedError, NoFreeKeySlotError and LastKeySlotError
without ever mentioning the encryption_key kwarg or
add_key/rotate_key/remove_key. Combined with the inert stub, there was no
in-tree source short of the Rust code. Adds an Encryption section and puts
encryption_key in the open() signature block.

Verification

Every example and claim in the new section was executed against the built wheel:
the encryption round-trip, all three open-time mismatch errors, the 8-slot limit
(NoFreeKeySlotError raised with 8 slots occupied), LastKeySlotError,
InvalidEncryptionKeyError on each of the three key methods, and
TransactionInProgressError on each mid-transaction.

That caught an error of my own: I first wrote that a raw key "must be 32 bytes".
It doesn't — derive_kek uses the bytes as HKDF input keying material and
rejects only empty input, so any non-empty length works. I also wrote that
b"hunter2" and "hunter2" derive different keys; they don't, because KDF
dispatch is on the slot's recorded kdf_id rather than the Key variant. Both
claims are corrected in the shipped text.

cargo clippy --workspace -- -D warnings clean, cargo fmt --check clean,
138 Python tests passing, 682 Rust tests passing.

Noted for later in this stack

max_pages also appears stale in README.md:188 and ARCHITECTURE.md:609
(the Rust DefragOptions field is max_values too). Those belong to #99 and
#98 respectively and are handled there, to keep this PR to the Python binding.

Xof added 2 commits July 29, 2026 13:08
Six findings from the 2026-07-29 review, all verified against the
running binding before and after.

savepoint.rs said the engine "pops the savepoint stack down to AND
including this savepoint, so the mark itself is gone after one
successful call". The engine does the opposite: `rollback_to_inner`
ends with `savepoints.truncate(idx + 1)`, retaining the named savepoint
so it can be rolled back to again — its own doc says so. That mattered
because the wrong claim was the stated justification for the Python
guard ("a cleaner, more specific AlreadyFinishedError"), inviting a
maintainer to delete a guard that is in fact the only thing enforcing
the single-use contract. The comment now says what the engine does and
re-justifies the guard on its own terms. Two test comments repeating
the same wrong model are corrected.

README Tags documented the pre-I126 API: tag 0 as an "untagged"
sentinel and `tag()` returning an int. Tag 0 raises ValueError on every
tagged method and `tag()` returns None. A reader following the old text
would write `if db.handles_with_tag(0)` (ValueError) or
`if tx.tag(h) == 0` (never matches, so untagged handles are silently
misclassified).

The README's only defrag example could not be run: `max_pages` is not a
field — `DefragOptions(max_pages=0)` raises TypeError before reaching
the engine, and the name appears nowhere in the codebase. Corrected to
`max_values`, dropped the pointer to a nonexistent docstring, and
removed the claim that defrag "lives on the Chisel object, not the
Transaction object" — `PyTransaction::defrag` exists and is tested.

The type stub was inert. Named `chisel/chisel.pyi`, PEP 561 resolves it
to module `chisel.chisel`, which does not exist, so no checker ever read
its 244 lines and drift accumulated undetected — `__exit__` was declared
`-> None` in all three classes while every implementation returns
`bool`. Renamed to `chisel/__init__.pyi` (pyproject include and header
updated), fixed the `__exit__` return types, and added a CI step that
type-checks it, which is what keeps it from going inert again. Verified:
mypy now resolves `import chisel` through the stub and reports
`--strict` clean at 3.13; the step is pinned to 3.13 because the stub
uses `collections.abc.Buffer` (PEP 688, 3.12+) while the package itself
still runs on 3.11.

README "Thread safety" forbade what the suite certifies —
`test_two_thread_mutex_contention` runs 1600 concurrent reads across
two threads and asserts no error — while omitting the property that
actually bites. Rewritten to state what holds: calls serialize (GIL +
Mutex) so concurrency cannot corrupt, transaction state is what must
not be shared, and since only `open()` releases the GIL, a long commit
or defrag blocks every Python thread in the process.

Encryption was undocumented: the README named five encryption errors in
its tables without ever mentioning the `encryption_key` kwarg or
add_key/rotate_key/remove_key. Adds an Encryption section covering the
bytes-vs-str key vocabulary, the three open-time mismatch errors, the
8-slot table, and the between-transaction restriction.

Every claim in the new section was executed against a built wheel,
which caught one of my own: a raw key is HKDF input keying material of
any non-empty length, not a fixed 32 bytes.

Closes #97.
The preceding commit staged only the chisel.pyi -> __init__.pyi rename:
the `git add` that should have carried the rest listed the old stub
path, which no longer existed after `git mv`, so the whole invocation
failed and added nothing. This commit is the content that belongs with
it — the README corrections (tags, defrag, thread safety, the new
Encryption section), the savepoint comment fix and its two test
comments, the stub's `__exit__` return types and header, and the CI
type-check step.

No change in intent from the message on the previous commit; see it for
the reasoning on each finding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python binding: README and docstrings contradict the shipped API

1 participant