Repository navigation
feat(stack-encrypt): order text over ADR-0002's folded order key - #1235
Conversation
|
|
Warning Review limit reachedOnly developers with an assigned seat can use this organization's usage-based review budget, and seats here are assigned manually. Ask an admin to assign a seat, or change the review continuation mode in Billing. Next included review available in 59 minutes. View limit details
Comment |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 425 | 0 | 342 | 0 |
Every mutant in the changed lines was caught by a test.
There was a problem hiding this comment.
🔵 Needs a closer look
Changes to cryptographic term derivation, key framing, and stored-byte compatibility warrant final human review.
No open issues 🎉
2 files not reviewed
languages/golang/encrypt/internal/testusers/kinds_stash.go: Generated filelanguages/golang/encrypt/internal/testusers/probe_stash.go: Generated file
What changed in this PR
Enables folded text ordering in the Rust encryption engine and Go SDK, preparing for EQL text-order types.
Changes:
- Derives text order terms using Unicode 16 case-and-accent folding.
- Enables text comparisons and validates folded range bounds.
- Updates generated Go methods, regression tests, documentation, and dependencies.
| File | Description |
|---|---|
| packages/stack-encrypt/tests/ui/pass/plan_chain.rs | Adds compiling text queries. |
| packages/stack-encrypt/tests/ui/op_order_on_text.stderr | Removes obsolete rejection snapshot. |
| packages/stack-encrypt/tests/ui/op_order_on_text.rs | Removes text-order rejection test. |
| packages/stack-encrypt/tests/ui/op_order_on_bytes.stderr | Adds bytes-order rejection snapshot. |
| packages/stack-encrypt/tests/ui/op_order_on_bytes.rs | Checks bytes ordering remains unavailable. |
| packages/stack-encrypt/tests/ui/op_between_takes_a_pair.stderr | Updates ordering diagnostics. |
| packages/stack-encrypt/tests/term_bytes.rs | Pins folded terms and rejection behavior. |
| packages/stack-encrypt/tests/op.rs | Tests folded comparisons and ranges. |
| packages/stack-encrypt/src/sem/mod.rs | Folds text and separates its key framing. |
| packages/stack-encrypt/src/op/mod.rs | Enables typed text ordering. |
| packages/stack-encrypt/src/dynamic/term.rs | Updates text-order documentation. |
| packages/stack-encrypt/src/dynamic/query.rs | Enables and validates dynamic text queries. |
| packages/stack-encrypt/fuzz/Cargo.lock | Locks the folding dependency. |
| packages/stack-encrypt/docs/adr/0009-operators-select-the-index-and-a-query-term-is-not-the-stored-term.md | Records text ordering as decided. |
| packages/stack-encrypt/CHANGELOG.md | Documents changed text term bytes. |
| packages/stack-encrypt/Cargo.toml | Adds vitaminc-ore. |
| packages/eql/Cargo.lock | Updates engine dependencies. |
| languages/golang/stashgen/testdata/cases/orders/order_stash.go.golden | Adds expected string ordering methods. |
| languages/golang/stashgen/refusal_test.go | Updates unused-index notice coverage. |
| languages/golang/stashgen/enginetest/enginetest.go | Mirrors text-order capability. |
| languages/golang/stashgen/emit.go | Updates unused-index explanations. |
| languages/golang/encrypt/README.md | Explains folded ordering and refusals. |
| languages/golang/encrypt/order_test.go | Tests folded ordering and generated queries. |
| languages/golang/encrypt/internal/testusers/probe_stash.go | Regenerates string ordering methods. |
| languages/golang/encrypt/internal/testusers/kinds_stash.go | Adds named-string ordering methods. |
| languages/golang/encrypt/guest/Cargo.lock | Updates guest dependencies. |
| languages/golang/encrypt/doc.go | Documents text ordering semantics. |
| languages/golang/cmd/stashgen/README.md | Documents generated string comparisons. |
| languages/golang/CHANGELOG.md | Records text ordering support. |
| Cargo.toml | Declares workspace folding dependency. |
| Cargo.lock | Locks vitaminc-ore and dependencies. |
Review effort: Balanced
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
cipherstash-agent
left a comment
cipherstash-agent
left a comment
There was a problem hiding this comment.
Automated review attestation
Source: Claude Code scheduled review (/code-review medium + /cross-check-review-findings) at head 54a204d. This is an automated review, not a human approval.
Critical / blocking issues: none.
Findings: none survived cross-check.
Reviewed (packages/stack-encrypt/src):
sem/mod.rs:OrderSourceandtext_order_key. Text goes through the folded key. The PRF context adds the frozenTEXTdomain label only for text, so integer, float and bytes terms keep their bytes. The key is zeroizing and moved, not copied.op/mod.rs: sealedOrdered::inverted. TheStringimpl compares folded keys. Integers keeplo > hi.dynamic/query.rs:ordered(kind),bound_orderand thecheck_inputfold refusal. They agree with the typed path (("b","A")inverted,("A","a")a tie, an unassigned code point refused before a keyset).- Docs, ADR-0009, trybuild snapshots, lockfiles and the Go generated/golden files, checked for consistency with the Rust change.
Cross-check against CI on this head:
- fmt/clippy/nextest/doctests/rustdoc, the Go matrix (Linux/macOS/Windows), Go lint and live tests, the EQL suites and the fuzz regressions are green.
- The mutants gate caught 24 of 24 mutants (0 missed).
- Copilot raised no issues.
- The Codex code and security reviews were still running when this review was posted.
Notes (non-blocking):
OrderSource::ofselects the text path by runtimeAnydowncast onString,Zeroizing<String>and&'static str. A new owned text type passed to an order term would silently take the non-foldedValuepath. This is fine today becauseOrderedis sealed, and the PR notes it. A test asserting everyOrderedtext form takes the folded path would guard it.- Text order terms change bytes (the CHANGELOG notes this as Breaking, with none deployed). Confirm no stored text ORE/OPE terms exist outside tests before merge.
- Changeset-bot reports no changeset. That looks acceptable here since the touched crates and the Go module use their own CHANGELOGs, but a maintainer should confirm.
Generated by Claude Code
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 54a204d924
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Parked until new This PR folds text inside Stack Encrypt ( No released or built version of either crate implements vitaminc-ore 0.6 yet. Once they do, this PR gets reworked:
The engine surface stays: |
54a204d to
f915cf5
Compare
|
Unparked, and reworked onto
The Go surface and the tests from the first draft stay. The description is rewritten to match. |
An `f32` or `f64` order term is cllw-ore's vitaminc encryption of the value at the `F64` target, whose vitaminc-ore encoding folds `-0.0` into `0.0` and every NaN into one value above `+Infinity`: Postgres's `float8` order. So the ordering operators are now answered over floats, on the typed path (`op::Ordered` for `f32` and `f64`) and over a data plan's `float32` and `float64` fields, and stashgen writes `Lt`, `Lte`, `Gt`, `Gte` and `Between` on a float field with `ore` or `ope`. An `f32` is ordered as the `f64` it widens to: its `Orderable` target is `F64` and vitaminc widens it exactly, so an `f32` and its `f64` are one term and `f32` can join `op::Ordered`. That settles what the earlier draft refused with `TermError::NarrowFloat`, which is gone. `Between` judges a float range in the same order (`(0.0, -0.0)` is one value, `(NaN, 1.0)` is inverted), by the comparison #1235 added. Float pins, the float ordering baseline, the integer-width lengths and the `score` field in the cross-path fixture carry over from the earlier draft; the fixtures and UI snapshots are re-recorded. Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
|
Mutants re-run after removing the equivalent |
|
Removed the inverted-range check from A range whose low bound is above its high bound is now encrypted like any other and matches nothing, as
|
ccd4431 to
ec45e28
Compare
|
Reworked onto the current The branch is now 3 commits:
What changed beyond the rebase:
Needs your decision: Verification:
|
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Review at ec45e28
Reviewed the full diff against next (merge-base c612ed402), the cllw-ore 0.5.2-rc.1 vitaminc module and vitaminc-ore 0.6.0 sources the terms now go through, PR #1243's diff for the inverted-range claim, and CI at this head. Findings first, most severe first; then what I checked and found sound.
Findings
1. High: the frozen stored-rows fixture fails at this head, and CI is red because of it.
stack-encrypt::stored_rows every_frozen_row_opens_and_seals_as_it_did fails in the nextest job and the CRAP gate, and the mutants gate exits 4 because its unmutated baseline fails (the "0 / 0 / 0" comment above is that). tests/fixtures/stored_rows_v1.json was frozen on next by 7ca7ba3, which is in this PR's merge-base, and it holds 37 ORE and 28 OPE terms. This PR changes the bytes of every order term (the /v2 labels, the column salt, the marker byte), so the "sealing the source today stores the same terms" half of that test fails for every case. The PR's verification section (644 passed) predates the rebase onto this next. The fixture's README says: never regenerate or edit v1; for a deliberate change to the stored format add stored_rows_v2.json and a write_stored_rows_v2 beside it, keep v1, and say in the CHANGELOG how v1 rows migrate. That rule and this PR's "nothing was deployed, re-pin everything" stance pull in different directions, and that is a call for the maintainers, but it has to be made in this PR: as it stands the PR cannot merge green, and the mutants result the description quotes was not produced at this head.
2. Medium: PR #1243 and two docs in this repo still say the client refuses lo > hi.
Commit 35b382d removes TermError::InvertedRange and both Supports::<Between>::check and check_input now let an inverted range through. PR #1243 (open, also based on next) was reviewed on the opposite claim: its range_functions.sql.j2 template (rendered into all ten query_<T>_range_functions.sql) says "The client refuses a range whose low bound is above its high bound; one that reaches the database anyway matches no row", packages/eql/docs/reference/eql-functions.md says "the client refuses to build one", and tests/encryption/tests/v4_range.rs has the same comment. The SQL semantics agree (an inverted range matches no row), so there is no correctness bug, but whichever PR merges second needs to fix the other's wording, and #1243's README hunk sits a few lines below this PR's README edit, so expect a conflict there too. Within this PR's own scope the same stale claim survives in languages/golang/encrypt/query.go:112 ("a range the wrong way round, and match text that cuts no token, are [ErrTerm]") and languages/golang/encrypt/guest/src/abi.rs:477; both files' tests were updated here, the doc comments were not.
3. Medium: "ADR-0002's fold" is not the fold that ships.
ADR-0002's "Text fold, v1" says the accent step removes "every code point with General_Category Mn". vitaminc-ore 0.6.0's fold.rs (what OrderTarget::Text encodes, domain vitaminc/ore/value/text-folded/unicode-16/v1) removes a mark of category Mn, Mc or Me with a non-zero canonical combining class and keeps class-zero marks. Thai U+0E31 MAI HAN-AKAT and Devanagari U+0901 CANDRABINDU are both Mn with class 0: the ADR's transform drops them, the shipped one keeps them, so two Thai strings the ADR predicts tie do not. The PR tells users in six places (Go README, doc.go, op::Ordered, dynamic/query.rs, both CHANGELOGs) that text orders by ADR-0002's fold, and amends the paragraph right after the step list, so the step list should be amended in the same way to describe what vitaminc-ore actually pins. The encoding belongs to vitaminc-ore, as this PR says; the ADR just has to stop describing a different one.
4. Low: the Go CHANGELOG understates the break and files it under Added.
languages/golang/CHANGELOG.md:73-77 says "Text order terms change bytes from the alpha's" under ### Added. Every ORE and OPE term changes, integers included (value_kinds.json re-pins i8/i16/u8 ORE and u16/u128 OPE), and the Go file's header says breaking changes go under ### Breaking. The stack-encrypt CHANGELOG gets this right.
5. Nit: the typed path refuses unfoldable order text one keyset load later than the data-plan path.
check_input refuses a text bound with an unassigned code point before any keyset; the orders! check is a no-op, so cipher.query(op::Lt, "a\u{378}").on(&plan).keyset("tenant") loads the tenant keyset first and refuses in the visitor. Nothing leaks and no term is produced, so this is consistency only.
Checked and sound
- Key derivation and separation. The per-index key is unchanged in shape: HMAC PRF over
PAE(label, context), with the labels bumped toore-key/v2andope-key/v2and still distinct. The block is copied into aProtected<[u8; 32]>, the visitor's copy is zeroized, and cllw'sKeywipes on drop with the scheme. The column saltPAE("vitaminc/ore/v1", scheme, target domain, "")goes into cllw-ore's keyed BLAKE3 as a prefix, so ORE and OPE terms are separated twice (key label and scheme name), and text and bytes of the same UTF-8 are separated by the target domain;a_text_order_term_is_keyed_apart_from_bytespins that. - The ORE crate owns the encoding.
Orderable::encrypt_inhands the value toOrderColumn::encrypt; stack-encrypt no longer folds, frames or derives a text key. cllw-ore's vitaminc schemes use its byte-slice entry points, so cllw's ownorderize_secretcollation is not applied on top of the fold. The only stack-encrypt call into the fold ischeck_input's pre-check throughvitaminc_ore::text, whose result is dropped. - Fold applied identically on every path. Typed (
sem::ore/ope), data-plan (ore_of/ope_ofthroughZeroizing<String>) and Go (through the guest) all reach the same visitor. Pinned bytext_order_terms_fold_case_and_accents(every owned form and&'static str), the data-planLt("CAFÉ") == Lt("cafe")operand check, and Go'sTestTextOrderingMethodsFoldTheValue(Lt(v)equals the stored ORE term for every folded form). - Order is actually pinned.
text_orders_by_its_folded_keychecks OPE byte order and ORE comparison overalice < eclair < Élan < emu < Zoe; the Go property test orders hundreds of random strings over the folded alphabet and the adjacency list covers"" < "a" < "aa" < "ab" < "b", Cyrillic, Greek and CJK, with ties for case, accent forms andß/ss/SS. - Boundaries. The empty string is keyed thanks to the marker byte (cllw-ore's own
an_empty_value_is_bound_to_its_key_and_column). No truncation anywhere; a term is 8 x (folded UTF-8 length + 1) bytes, so the length leak is of the same kind as before (folded length rather than raw length).u8goes in asRaw(Int::U8)at targetU8, matching how the other widths are placed.boolis refused before any PRF call. Unassigned code points are refused on write and on query, in line with hownextalready treats equality and match terms, and the ADR-0002 amendment says so. - Breaking-change marking. The stack-encrypt CHANGELOG entry sits under
### Breaking, names the three causes, and the pins interm_bytes.rs,frozen_bytes.rsand the three regenerable fixtures are re-recorded. See finding 1 for the one fixture that is not regenerable. - Changeset. None needed: no npm package consumes stack-encrypt (protect-ffi's
Cargo.lockcarries cllw-ore 0.4.2 and no stack-encrypt), and the crate and the Go module keep their own CHANGELOGs. The root, EQL, encrypt-guest and stack-encrypt fuzz lockfiles are updated. - Codex's earlier P2 (pass unassigned code points through) is addressed by the ADR-0002 amendment in ec45e28; the earlier automated review's note about the
Anydowncast is moot now thatOrderSourceis gone.
Checks run
- Read the diff, cllw-ore 0.5.2-rc.1
src/vitaminc.rsandsrc/impls/string.rs, vitaminc-ore 0.6.0src/scheme.rs,src/order.rsandsrc/fold.rs, and PR #1243's diff. cargo nextest run -p stack-encrypt --all-featureslocally at this head: 677 tests run, 676 passed, 1 failed, 1 skipped. The one failure isstored_rows::every_frozen_row_opens_and_seals_as_it_did(every_kind: everyoreandopepassthrough differs from the frozen row; everyeqandmatchterm is unchanged), the same failure CI reports.- CI at this head: everything green except the three jobs in finding 1.
|
Force-pushed: a fix folded into ba29e03. An order term takes its plaintext by value: the ORE and OPE visitors carried it into |
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Re-review of the update to ba29e03.
Compared with 4392b57, only commit 3 changed, and the change is heap hygiene for order terms. It moves no bytes: stored_rows_v1.json and the term fixtures are untouched.
Orderablegains a required,#[doc(hidden)]wipe. The owned sources (String,Vec<u8>and theirZeroizingforms) zeroize. Scalars and'staticborrows are no-ops. Because the method is required on a sealed trait, any new source type has to say which kind it is.OreVisitorandOpeVisitornow hold the plaintext asCarried<T>, which wipes it on drop.ore_termandope_termwrap the value beforeorder_target()can fail, so a refused value is wiped too.encrypt_inreads through&self, so the drop is the last owner.Vec'sZeroizealso clears spare capacity, which covers aStringbuilt with extra capacity.- The new
an_order_term_of_owned_text_or_bytes_leaves_no_plaintext_on_the_heaptest covers both terms, forStringandVec<u8>.
This closes the unwiped plaintext clone on every OPE derivation. It was found while reviewing #1274, where the eql heap-hygiene test had excluded stack_encrypt::sem::OpeVisitor by backtrace. #1274's latest push removes that exclusion, so the leak is now covered by a test rather than excused.
Unchanged from review 5565622638: the fixture-exception premise and trigger, re-freeze tooling, the stale "never edit" wording, the InvertedRange docs conflict with #1243, the ADR-0002 fold step list, and the Go CHANGELOG section. CI is green at this head.
ba29e03 to
7a8c8ae
Compare
|
Rebased onto |
7a8c8ae to
e086fcf
Compare
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
Re-review at e086fcf
Incremental review against the previous head ba29e03. The branch was rebased from next at c612ed4 onto next at 5887218ce (the #1267 smallint merge), and the top commit was then amended twice (7a8c8aeee, where the review fixes landed, then e086fcf, which only narrows one README sentence). git range-diff across the two heads shows commit 1 unchanged, commit 2 gaining four doc fixes, and commit 3 gaining the fixture README exception, the re-freeze path in stored_rows.rs, the ADR-0002 bullet, both CHANGELOG entries, the _comment and the term_bytes.rs doc. Status of the open items first, then one new finding.
Earlier findings
| # | Was | Now |
|---|---|---|
| 1 | Medium: fixture-exception premise false, trigger unobservable | Fixed. The README names stack-encrypt 0.1.0, 0.2.0 and Go v0.1.0-alpha.1 as having shipped order terms, says the exception rests on those being pre-1.0 and pre-release rather than on nobody holding a term, ends it at stack-encrypt 1.0 (a crate version, so observable), and says why a stale order term is worse than a stale leaf. The stack-encrypt CHANGELOG's Breaking entry names the three releases and says to encrypt the column again. One loose end, Low, below. |
| 2 | Medium: no re-freeze tooling, "nothing else moved" unenforced | Fixed, and I watched it fail. STACK_ENCRYPT_REFREEZE_ORDER_TERMS=1 rewrites only ore/ope passthroughs after without_order_terms has asserted the rest of the tree is identical; the write is after the loop, so any other difference panics first. Checked by hand at this head: one corrupted ore term fails the normal run with the hint and is restored byte for byte by the re-freeze run; one corrupted eq term fails the re-freeze run at the "same shape, terms and passthroughs" assert and nothing is written; re-freezing the committed file is a no-op. Nothing in .github or the mise files sets the variable. |
| 3 | Low: stale "never edit" wording | Fixed. The JSON _comment (both copies), the #[ignore] reason (stored_rows.rs:589) and the writer's refusal (:594) now defer to the README. |
| 4 | Medium: InvertedRange docs |
Fixed within this PR (query.go, guest/src/abi.rs, ADR-0007, ADR-0009, all in 5024415ed). The #1243 side is declined with "whichever merges second fixes the other", which is sound; #1243 is still open (now ready for review, base next), so that is still owed by one of the two. |
| 5 | Medium: ADR-0002 fold step list | Fixed. The bullet now says a mark of Mn/Mc/Me with non-zero canonical combining class is removed and a class-zero mark is kept, with a dated amendment. Checked against vitaminc-ore 0.6.0 src/fold.rs: is_mark is canonical_combining_class(c) != 0 && category in {Mn, Mc, Me}, and the pipeline is NFD, case fold, NFD, drop marks, NFC. |
| 6 | Low: Go CHANGELOG under Added | Fixed. languages/golang/CHANGELOG.md:38 is a ### Breaking entry covering every width, floats, text and bytes, naming the alpha, and saying the miss is silent. |
| 7 | Nit: typed path refuses unfoldable text one keyset later | Declined: the typed Eq path has the same shape. Agreed. |
New finding
1. High (blocks merge, not a code defect): the branch does not merge into the current next.
The rebase landed on next at #1267. Since then #1262, #1264 (remove FfiValue), #1263 and the vitaminc 0.6.1 bump (b5baedba0) merged, and GitHub reports the PR as conflicting. git merge-tree origin/next e086fcf conflicts in five files: Cargo.toml (the new cllw-ore/vitaminc-ore lines sit beside the lines the 0.6.1 bump rewrote), src/dynamic/query.rs (four hunks: #1264 spelled the test inputs as Tree, this PR as ValueFrame::of, and next still carries the integer_order test that commit 2 deletes), src/dynamic/term.rs (one), tests/heap_hygiene.rs (the import block: use stack_encrypt::nonempty; next to the FfiValue import #1264 replaced) and guest/tests/native_ops.rs (one). All of it is test inputs and imports, so mechanical, but the green CI at this head was produced against the pre-#1264 base. Two things worth knowing for the rebase: vitaminc-ore 0.6.0 and cllw-ore 0.5.2-rc.1 both take caret 0.6.0 requirements, so they resolve against vitaminc 0.6.1, and vitaminc-ore 0.6.1's src/ is byte-identical to 0.6.0's, so the fold and the text term bytes should not move; stored_rows will say.
2. Low: the PR description and the last comment lag the tree.
The description's Breaking bullet still says "Nothing was deployed (ADR-0002)", which the CHANGELOG no longer claims; Verification quotes "644 passed" (678 at this head); and nothing in it mentions the fixture exception or the re-freeze path. The comment at the last push says the exception ends at "stack-encrypt 1.0 or the Go module's first non-pre-release, whichever comes first", but the e086fcf amend dropped the Go clause, so the README now ends it at stack-encrypt 1.0 alone. That is a defensible choice (a Go v0.x is unstable by Go's own convention, and the Go CHANGELOG carries the Breaking entry), but the comment should say which it is, since a Go v0.x release holding order terms could now be invalidated under the exception.
3. Nit: var_os(..).is_some() enables the rewrite on any value. STACK_ENCRYPT_REFREEZE_ORDER_TERMS= (empty) re-froze the corrupted fixture in my run just as =1 did. The README says =1; matching that (== Some("1")) is a one-liner.
Rebase adaptations, checked
- The only conflict the author reports,
languages/golang/CHANGELOG.md, is resolved with both entries kept: the Smallint entry at:80and the text-ordering entry at:84, in### Added. nextgained nothing underpackages/stack-encryptbetween the two bases, so no smallint rows or terms entered any stack-encrypt fixture; smallint lives ineql-bindingsand is equality-only, so there is nothing of it to re-freeze.stored_rows_v1.jsonagainstorigin/next's copy, by structural diff: exactly 19oreand 14opepassthroughs plus the_commentdiffer; no leaf,eq,match, plain passthrough, key or shape difference. Againstba29e03the only change is the_comment.record_lowering.json,value_kinds.jsonandquery_operators.jsonare byte-identical toba29e03; against the base onlycleaves (fresh nonce),ore/opeterms and the fiveageorder queries differ.
Checks run
git range-diffacross the two heads;git merge-tree --write-tree origin/next refs/prwatch/1235; structural JSON diffs of the four fixtures with a script.- vitaminc-ore 0.6.0 and 0.6.1 and cllw-ore 0.5.2-rc.1 sources from the local registry.
mise x --env test -- cargo nextest run -p stack-encrypt --all-featuresat this head: 678 passed, 1 skipped (stored_rows,term_bytes,frozen_bytesandheap_hygieneincluded); then the three re-freeze experiments above, with the fixture restored afterwards.gh pr checks 1235: every job passes at this head;mergeableisCONFLICTING.
A non-superuser install skips the ore_block_256 btree operator class, and ore_fallback.sql then poisons every single-bound ORE domain so a value cast to one fails loudly with an OPE hint. It could not reach the nine query_<T>_range_ore domains: it is shared with EQL v3 and they are v4-only. So on such an install a range operand was accepted, and eql_v4.between ran as an unindexed seq scan with no hint. Each v4-only range types file now adds the same NOT VALID poison CHECK to its own ORE range, behind the same opclass check, calling the function ore_fallback.sql creates, and names the family's OPE range in the hint. A new PostgreSQL test installs the built v4 bundle as a NOSUPERUSER role in a database of its own and checks every ORE range is refused with that hint while its OPE range is accepted. Stripping the new ALTER DOMAINs from the bundle makes it fail. Also: the generated ORE between note now lists the family's own OPE column domains and OPE range instead of a fixed `_ord`, `_ord_ope`, `_search` (only text has `_search`), and the inverted-range wording no longer says the client refuses such a range, which #1235 makes untrue; it says what holds either way: such a range matches no row and the SQL does not reorder it.
e086fcf to
b98bad2
Compare
|
Rebased onto Conflicts, and how each was resolved:
No production logic changed in the resolution; only imports and tests. |
b98bad2 to
21659d1
Compare
|
Force-pushed b98bad2 → 21659d1; the content changed, all in the text-ordering commit:
At the tip: fmt, nextest (1218), clippy |
21659d1 to
7840c3a
Compare
|
Force-pushed 21659d1 → 7840c3a, two changes inside the text-ordering commit: the |
Text ordering is decided: the order key is the system ADR-0002's text fold, which vitaminc 0.6 ships in vitaminc-ore, and the ORE and OPE indexes answer the orderings on text over it. The ADR no longer says "if the catalog keeps text ordering" or that the paragraph goes if text ordering is dropped, and decision 7 names text beside the integer types.
A `Between` whose low bound is above its high bound was refused with `TermError::InvertedRange` before either term was derived. SQL's own `BETWEEN` refuses nothing: `x BETWEEN 20 AND 10` is a valid query that matches no row, and an encrypted range now behaves the same way. The empty result leaks nothing and corrupts nothing, so a refusal bought little. It would also cost more than it buys once text and floats are ordered. Judging their ranges correctly means reproducing, before encryption, the order the index compares in: text by its fold, floats by their canonical encoding. That order belongs to the ORE crates, not to this one. So `TermError::InvertedRange` (code `stack_encrypt::inverted_range`) goes, with the typed path's `lo > hi` check (and the `PartialOrd` bound `op::Ordered` carried for it) and the data path's `integer_order`. The variant is under [Unreleased] and never shipped.
… cllw-ore's vitaminc schemes Text could not be ordered: no ordering operator was answered on a text field, and the ORE and OPE terms stored for text went through cllw-ore's own collation, which keeps case, so Zoe ordered before alice and é after z. ADR-0002 defines text's order key as its fold (Unicode 16 NFC with case and accents folded), and vitaminc 0.6 ships it in vitaminc-ore. cllw-ore 0.5.2-rc.1 implements vitaminc-ore's `OrderScheme` behind its `vitaminc` feature, so Stack Encrypt encodes nothing for ordering itself. An ORE or OPE term is now cllw-ore's `vitaminc::Ore` or `vitaminc::Ope` encryption of the value: the engine names the value's order target through the new sealed `sem::Orderable` (integers at their own width, `f32`/`f64` as `F64`, text as `Text`, bytes as `Bytes`) and passes the value straight through, on the typed path and for each scalar of a dynamic value alike; cllw-ore encodes it with vitaminc-ore, text by the fold. Café, cafe, CAFÉ and cafe with a combining accent are one term. Text holding a code point Unicode 16 does not assign has no fold and no order term, as it has no equality or match term; ADR-0002 is amended to say vitaminc 0.6 refuses such a code point rather than passing it through. The per-index key derivation is kept: the PRF block keys the scheme inside the visitor and is wiped there, and the column is opened with the value's target and the empty context, since the key is already the index's own. The key labels move to `/v2` because every order term changes bytes (new column salt, cllw-ore's marker byte for text and bytes). Every source type's term is now one ciphertext type (`OreCllw8VariableV1`, `OpeCllw8VariableV1`); `OreTerm<T>`/`OpeTerm<T>` keep `T` as a marker so typed callers compile unchanged, and `sem::CllwOreEncrypt`/`CllwOpeEncrypt` are no longer re-exported. The term pins and the record, query and value-kinds fixtures are re-recorded. op::Ordered covers String, and the data path answers the orderings on text too, refusing order text with no fold at its boundary, for one value and for each bound of a range, through `vitaminc_ore::text`, the encoding the `Text` target uses. The Go generator writes Lt, Lte, Gt, Gte and Between on a string field with ore or ope; its notice is left for order indexes on bytes or a float, and for a second order index. Closes #1234.
7840c3a to
3004048
Compare
|
Rebased onto |
Summary
Stack Encrypt is CipherStash's Rust encryption engine; the Go SDK runs it as a WebAssembly guest. Next to each ciphertext it can store order terms, which let the database answer
<,>,BETWEENandORDER BYwithout decrypting. There are two kinds: ORE (order-revealing encryption, compared by a special function) and OPE (order-preserving encryption, compared as plain bytes).Until now Stack Encrypt could not order text at all. This PR makes text orderable, so a Go string field with
index=oreorindex=openow getsLt,Lte,Gt,GteandBetween. Text orders by ADR-0002's fold: case and accents are folded, under Unicode 16. SoCafé,cafeandCAFÉtie,alicecomes beforeZoe, andésorts witherather than afterz.How it works changed since the first draft. The ORE crate
cllw-orenow encodes and encrypts each type for ordering itself, throughvitaminc-ore. That shipped as a pre-release, 0.5.2-rc.1, with avitamincfeature. Stack Encrypt says which type a value is and passes it straight through. It no longer folds text or builds order keys of its own.Changes
Engine (
packages/stack-encrypt)Dependency.
cllw-oreis now=0.5.2-rc.1with thevitamincfeature, in the root workspace. The root,packages/eql, encrypt guest and stack-encrypt fuzz lockfiles are refreshed and pass--locked.sem::Orderableis one constant. A sealed trait with a single item,const TARGET: OrderTarget, thevitaminc-oretarget a type's values are encoded as. That is the one thing vitaminc'sOrderEncrypt::column(target, context)leaves to the caller; everything else goes through vitaminc's ownEncrypttrait. (The first draft had three methods; the other two re-implemented whatEncryptandZeroizingalready do, and are gone.) The table:u16tou128,i16toi128f32,f64F64StringTextVec<u8>Bytesbool,u8,i8OreTerm<bool>,OreTerm<u8>and anOre/Opeindex on either do not compile (a trybuild case holds it)A value is encrypted by reference, as the
Encryptsource it already is (&String,&i32, ...), and the engine holds the one copy it was handed inzeroize::Zeroizingand wipes it when the term is derived or refused. There is no customDrop. The heap-hygiene test fails if that wrap is removed. Because the source must own its bytes to be wiped,&'static str,&'static [u8],Zeroizing<String>andZeroizing<Vec<u8>>are no longer order sources; pass aStringorVec<u8>.8-bit integers have no order term. An order term over 256 values ranks every row among 256, which a few known rows resolve to the values: the leak the crate already refuses for a boolean's two. So
u8andi8leaveop::Orderedandsem::Orderable,dynamic::admitsrefusesoreandopeonint8anduint8(as onbool), a data plan naming the pair is refused when built, and a frame with an 8-bit scalar under an order index yields no term. Both keep their equality term. Thevalue_kinds.jsonfixture'si8andu8fields carry equality alone (regenerated), and the frozenstored_rows_v1.jsonloses its four 8-bit order terms under the README's order-term exception, with every other byte as written (the README records the edit). Go: stashgen refusesore/opeonint8,uint8andbyteas it does onbool, with tests against the engine and the fake; the README's type table and the Go CHANGELOG say so.Order terms.
cllw_ore::vitaminc::OreorOpeand is wiped. The visitor opens a column with the value's target and an empty context, since the key already belongs to this index, and encrypts the value.ore-key/v2andope-key/v2.Removed.
OrderSource,text_order_key, theAnydowncasts, theCllwOreEncrypt/CllwOpeEncryptre-exports, andTermError::NoOrderTerm(codestack_encrypt::no_order_term): the boolean refusal it carried at run time is a compile error now.Term types.
OreCllw8VariableV1orOpeCllw8VariableV1.OreTerm<T>andOpeTerm<T>keepTto name the source, so typed callers compile unchanged.KeysetCipher::ore_termandope_termreturn the ciphertext types.Between.op::OrderedcoversString.Betweenno longer refuses a range whose low bound is above its high bound. As in SQL,BETWEEN 20 AND 10is a valid query that matches nothing.TermError::InvertedRange(codestack_encrypt::inverted_range) is gone; it was unreleased.vitaminc_ore::text, the encoding theTexttarget uses.Unencodable text. Text with a code point Unicode 16 doesn't assign has no order term (
TermError::Normalize), on write and on query.Breaking: every order term changes bytes. The causes are the
/v2labels, the column salt (which binds the scheme and the target), and cllw-ore's one marker byte in front of text and bytes. stack-encrypt 0.1.0 and 0.2.0 and the Go module's v0.1.0-alpha.1 shipped the old terms, and nothing refuses a stale one: a<,BETWEENorORDER BYover a column holding them silently misses rows, so the CHANGELOGs say to encrypt such a column again. The pins interm_bytes.rsandfrozen_bytes.rsare re-recorded, as are both cross-path fixtures; the frozenstored_rows_v1.jsonre-freezes only itsoreandopepassthroughs, underSTACK_ENCRYPT_REFREEZE_ORDER_TERMS=1, which the check accepts only when nothing else moved. That exception runs until stack-encrypt 1.0 (tests/fixtures/README.md).Docs. ADR-0009's text-ordering section names the crate, and the CHANGELOG has the Breaking entry.
Go (
languages/golang)ore/ope. Generated files and goldens are regenerated, and the README,doc.goand the stashgen README describe the fold. These are unchanged from the first draft.alice, not 40.Verification
mise x --env test -- cargo nextest run --workspace --all-features: 1218 passed.cargo nextest run -p stack-encrypt --all-features: 679 passed. This includes the trybuild UI binary: one new case (order_term_of_an_8_bit_integer), and two snapshots re-recorded for the changedOrdereddiagnostic.tests/heap_hygiene.rs: passes, and fails when the order visitor'sZeroizingwrap is removed (checked by hand).tests/stored_rows.rspasses without the re-freeze variable at the final head.cargo clippy --workspace --all-targets --all-features -- -D warnings,cargo fmt,mise run doc(rustdoc with warnings as errors) andmise run test:doc(stack-encrypt: 40 doctests): all clean.cargo mutants --no-shuffle -p stack-encrypt --in-diffagainstnext(CI's form), at the final head: 38 mutants, 25 caught, 13 unviable, none missed.cargo test --locked -p eql-bindings --all-features(197 passed), andpackages/eql/tests/encryption(37 passed, 3 ignored; those need a database).mise run wasm:guest:testpasses (68 and 71 tests).mise run go:testpasses,mise run go:lintreports 0 issues, andgo generate ./...leaves no diff.u32and for text;Lt/Betweenon text, with folded ties;DATABASE_URL).Related
obterm stores, still waits on anore-rsrelease (stashgen refuses TextOrdOre and TextSearchOre — EQL stores block ORE terms and the engine derives CLLW ORE #1132).Review notes
sem/mod.rs's ORE/OPE section:Orderable(one constant) and the visitors, which hold the plaintext inZeroizingand encrypt it by reference through vitaminc'sEncrypt.Orderable::TARGETstays here until vitaminc-ore owns the rule (vitaminc-ore: order index widths are only checked at runtime, and every call needs a target and context — add typed targets and a chained API vitaminc#470); then it goes.cllw-oreis pinned to the pre-release on purpose. When 0.5.2 (or later) is final, bump the pin; the terms shouldn't change.stack-authandstack-kmsfuzz lockfiles are already stale onnext, so--lockedfails there. This PR leaves them alone.https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL