Repository navigation
feat(eql)!: produce the block ORE EQL v4 types through ore-rs 6-bit block ORE - #1280
Conversation
🦋 Changeset detectedLatest commit: 45a79a0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 63 | 0 | 29 | 0 |
Every mutant in the changed lines was caught by a test.
coderdan
left a comment
There was a problem hiding this comment.
Review of 4799dd1 against feat/eql-v4-text-search-optional-terms (4430c0c). No blocking defect found. The SQL comparator reproduces ore-rs 0.9.0's comparators exactly, and every check I could run without credentials passes, including the PostgreSQL-backed suite and the timing harness on a scratch PostgreSQL 15.12. Three low-severity notes follow, each inline; nothing here needs to hold the PR.
Findings (most severe first)
-
Low — the shape check is about half of every comparison.
ore_block_v2_schemeruns twice percompare_ore_block_v2_termcall and each run issues two SPI sub-selects (the symbol scan and the padding-bucketEXISTS). Measured here on ani32-shaped term: scheme check 3.6 µs, hash bit 2.6 µs, whole comparison 14.6 µs (20,000 calls each, PG 15.12, Apple silicon); v3's comparator on a 408-byte term takes 3.9 µs on its equal path. A sort or btree build re-validates the same stored term at every comparison it takes part in. Both scans can be plain expressions; details inline. -
Low — the 1023-byte ceiling is enforced at encryption only; the comparator admits any count the policy allows, up to the wire maximum. That is what ore-rs does too, so this is a note rather than a divergence. A well-formed chained term of 65535 blocks (1,638,398 bytes) needs no key to craft, since every shape check is on public structure; one comparison of it takes ~20 ms here, and every comparison a btree takes against that row pays it. It needs write access to the table, which already allows worse, so this is informational unless you want
ore_block_v2_schemeto bound the count to what Stack Encrypt can write (1,366 blocks for a 1,024-byte bucket). -
Low — residual data-dependent work in the comparator loop, already acknowledged in the doc comment.
substr(...) <> substr(...)is amemcmpthat stops at the first differing byte, and the two post-loop reads atfirst_diffare at a secret-dependent offset. Your harness and my run of it both put every |t| under 5, so this is a suggestion: the tags can be compared as twobiginthalves XORed (constant work), and the right block and tag can be latched arithmetically inside the loop so nothing after it is indexed by the first difference, which is what ore-rs'sFirstDiffdoes. Inline on the loop.
Housekeeping, not a defect: tests/timing/ore_block_v2/ctgen is a fourth detached Cargo workspace with its own Cargo.lock; AGENTS.md's list of detached workspaces (three fuzz crates and two Go guests) and .github/dependabot.yml do not know about it yet. Inline on its Cargo.toml.
What I verified as sound
Comparator against ore-rs 0.9.0 (read src/scheme/bit2_w6.rs, chained.rs, width.rs, primitives/hash.rs, ciphertext.rs, vitaminc.rs from the registry copy the lockfile names, and the head functions.sql side by side):
- Layout and offsets: header 4 (
02 02) or 7 (02 03+ 3-byte policy), thencountsymbols,count16-byte tags, 16-byte nonce,count8-byte right blocks;nonce_b = header + 17*cb, right block k at 1-basednonce_b + 17 + 8k, tag k ofaatheader + ca + 16k + 1. All matchtotal_len,f_at,nonce_at,right_at. - Lewi-Wu split:
a'sxt/f,b's nonce and right block, asFirstDiff::latch(a[n], left_block(a_f,n), right_block(b_right,n), first). - Bit extraction: PostgreSQL
get_bit(bytea, n)is byten/8, bitn%8LSB-first, which isct_select_byte(right, xt/8)thenct_bit(byte, xt%8), and the same orderlsb_mask/indicator_mask_xorwrite. - σ-MMO: GF(2^128) doubling big-endian with the lo top bit carried into hi and
0x87folded on hi's top bit, XOR nonce, AES-128-ECB under the publicORE-rs.v2.H-pi.1,lsb(π(m)) ⊕ lsb(m);aes-ecb/pad:noneon an exact 16-byte block. The 12 vectors pass here. - Refusal set equals ore-rs's: version/scheme id,
count ≥ 1,≤ 14for the fixed scheme, exact length,Padding::from_wire(Noneonly withn = 0,padded_len(0) ≤ 49151) andadmits_count(kin0..=15,bucket > min_bytes,ceil(8n/6) = (8n+5)/6), every symbol< 64, different schemes, different fixed counts, different policies. ore-rs returnsNone; the SQL raises, naming only the length. - Chained: lexicographic over
min(ca, cb), shorter first on a full match (ca.cmp(&cb)=sign(ca - cb)); for the fixed schemeca = cbis enforced so that term is 0. - Result selection and the latch are arithmetic; the hash runs on every path.
Security
- Key derivation:
ORE_BLOCK_KEY_DOMAIN(stack-encrypt/sem/ore-block-key/v1) is distinct from the CLLW domains; the 32-byte index key goes toBit6BlockOre::new/ChainedBlockOre::new, which derive their 16-byte ORE keys under separate labels (block-ore-bit6/key/v2,chained-block-ore/key/v2) and bind scheme, target and context into the term salt through ore-rs'sinit_with_saltstages. Consistent with ADR-0002 as amended;left_halves_are_pinnedandterms_are_keyed_by_their_contexthold it. - Nonce: 16 bytes from
ChaCha20Rng::from_entropy()inside ore-rs, per term. - Wiping: the plaintext travels as
Carried<T>; ore-rs frames text/bytes in aZeroizingcopy and scrubs the stack;heap_hygienereads the producible text targets from the table, soTextOrdOreandTextSearchOreare covered without an edit. - Limits: 1023 =
DEFAULT_MAX_STR_BYTES− the marker byte; trailing NUL refused because NUL padding would alias"a\0"and"a". Both are enforced at derivation (ore-rs'sTooLong/TrailingNul, mapped to the newTermErrors) and by the key-freeore_block_textcheck on the query path, both bounds included. Neither message names the value. - Error messages in SQL name a length or a cardinality, never bytes; the test asserts it.
Dependencies and release
ore-rs = "0.9.0"from crates.io (default-features = false,vitaminc), not git or path. TheCargo.lockchurn in the root, EQL, Go encrypt guest and stack-encrypt fuzz workspaces is ore-rs and its dependencies (block-modes,byteorder,num*,orderable-bytes,subtle-ng) plusvitaminc-*0.6.0→0.6.1, which ore-rs 0.9.0 requires. protect-ffi's and the auth guest's lockfiles are untouched, so no WASM-unknown build picks upgetrandom.cargo metadata --lockedpasses in the root,packages/eql, both Go guests,packages/stack-encrypt/fuzzandctgen;packages/stack-auth/fuzzfails on the base too, as the PR says.- eql-bindings pins
stack-encrypt = "0.3.0", which crates.io does not have yet (max 0.2.0); pre-existing on the base, and theIndexSpec::OreBlockbreak sits under[Unreleased]in stack-encrypt's CHANGELOG, so the ordering stays "stack-encrypt 0.3.0 before eql-bindings".
v3 bundle and v4 objects
mise run --force buildat base and head:cipherstash-encrypt.sqlandcipherstash-encrypt-uninstall.sqlare byte-identical (8eccb1cb…,b1b5131b…). The v4 bundle namesore_block_256only in@filepath comments (5) andore_block_v2173 times; installed,eql_v4_internalhas noore_block_256object, andore_block_v2_operator_classis the default btree opclass for the type.{{ore_block}}refusal:the_block_ore_name_follows_the_versionpasses (literal refused except assem/ore_block_256/).
Breaking surface and docs: CHANGELOGs (stack-encrypt breaking entry under Unreleased, eql-bindings, Go), ADR-0002 and ADR-0009 amendments, the @cipherstash/eql minor changeset. The skills name only v3 objects, so leaving them is right. stored_rows.rs and record_lowering.rs are not in the diff and the suite passes.
Checks run
mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 695 passed, 1 skipped.packages/eql:cargo test --locked -p eql-codegen(162),-p eql-bindings --all-features,-p eql-encryption-tests;mise run types:check.go test ./stashgen/... ./cmd/stashgen/... ./internal/...: ok.- Scratch PostgreSQL 15.12 (mise's; Docker was down) with the head v4 bundle: the
--ignoredsuitesv4_ore_block(5),text_eq,v4_types,v4_order,v4_text_searchall pass, includingEXPLAINshowing the index for<andORDER BYthroughord_term_ore. tests/timing/ore_block_v2/run.sh 20000: 0 wrong of 1,500; t = +0.93/+1.21 (fixed first vs last), −2.07/−3.41 (equal vs first), +2.48/+2.41 (chained first vs last), −2.14/−0.73 (prefix vs first).- CI: all checks green at this head. No prior reviews or review comments on the PR.
4430c0c to
9048302
Compare
4799dd1 to
59087de
Compare
|
Restacked onto the new #1274 and force-pushed. Besides hashes: smallint has a Rust plaintext now (#1267), so |
59087de to
2189d5a
Compare
9048302 to
30d3a35
Compare
coderdan
left a comment
There was a problem hiding this comment.
Incremental re-review of the restack to 2189d5a (onto #1274's 30d3a35). Range-diff against 4799dd1: commits 1 (the engine) and 3 (Go) are identical; commit 2 (the SQL comparator and harness) and commit 4 (the bindings) changed.
Substantive, commit 2, responses to review 5566990967.
- Housekeeping, the
ctgendetached workspace: addressed. AGENTS.md lists it,test-eql.yml'srust-cratesjob runscargo metadata --lockedagainst its manifest (reachable:packages/eql/tests/**is in both the push filter and therelevant:block, and the job's working directory ispackages/eql), andrun.shbuilds it with--locked. One sentence of the AGENTS.md text is wrong; finding 1 below. - The three LOWs on the comparator (the shape check runs twice with two SPI sub-selects each; no upper bound on a chained term's block count; the residual
memcmp/substrdata dependence) are unchanged. The author left each to the maintainer (1, 2, 3). Still open; the comparator body infunctions.sqlis byte-identical to4799dd1.
Rebase adaptations, commit 4 (smallint). SmallintOrdOre is producible, deliberately and consistently. The engine already takes i16 for a block ORE term (sem/mod.rs:1217, i16 => I16, under Bit6BlockOre for targets of at most ten bytes); ENCRYPTION_DOMAINS gains ("smallint","ord_ore") in the same position as the other families; v4/smallint.rs derives it with #[stash(plaintext = i16)]; v4/targets.rs has the TARGETS row (indexes: ["ore_block"], producible) and all three _named arms; eql_gen.go agrees (mise run types:check no drift at this head); the guest's targets.rs and native_ops.rs, eql_test.go, query_test.go (families now includes Smallint), the codegen tests, tests/targets.rs, both READMEs, both CHANGELOGs and the changeset list it. The v4_ore_block.rs changes are this adaptation, not a review response: SmallintOrdOre joins the shape table, and an i16 PostgreSQL agreement check runs over [MIN, -300, -1, 0, 1, 63, 64, 300, MAX], which crosses a 6-bit block boundary at 63/64. The refusal tests move to DateOrdOre with the base's "no Rust plaintext type" wording. The run.sh change is the --locked flag only. No fixture changed.
New findings
- LOW, docs (inline on AGENTS.md): "None of them has a Dependabot entry" is wrong for five of the six detached workspaces.
.github/dependabot.yml(lines 254–260) lists/packages/stack-auth/fuzz,/packages/stack-kms/fuzz,/packages/stack-encrypt/fuzz,/languages/golang/encrypt/guestand/languages/golang/auth/guestunder a lockfile-only cargo entry, held byscripts/__tests__/dependabot-cargo-cross-workspace.test.mjs. Onlyctgenis unlisted, and that file's comment still counts "five detached workspaces". A reader of AGENTS.md would either add a duplicate entry for the fuzz crates or take ctgen's omission as the convention. Either add/packages/eql/tests/timing/ore_block_v2/ctgento thatdirectories:list (it shares no crate by path, so the cross-workspace rule does not bite) and update the count, or reword the sentence to say ctgen alone is unlisted and why. - LOW, coverage (inline on
query_test.go): no Go test seals or queries a block ORE type through the guest.TestTheCatalogListsEachTypesOperatorsreads the catalog only, and notestusersmodel carries an*OrdOreorTextSearchOrefield, so there is no Go-sealed row and noLtterm forobchecked from Go, where the OPE types haveRankingandTestV4Ordered*andTextSearchhasTestV4TextSearchStoresEveryTermAndQueriesCarryOne. The guest's own tests (native_ops.rs) run natively, so ore-rs is never executed on wasm32 anywhere in CI; a wasm32-only panic or a term the guest writes differently from native Rust would surface in a customer program first. A model with anIntegerOrdOreand aTextOrdOrefield, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL the wayRankingis, would close it (the nonce makesobnon-deterministic, so the fixture would compare everything butcandoband checkobthrough the comparator).
Nit: the PR body still says "Smallint's OrdOre is not produced here" and "the six types are producible"; the restack comment has it right.
Checks (at this head, no credentials): cargo test --locked -p eql-codegen -p eql-bindings --all-features (162 + 116 unit, all integration suites pass), -p eql-encryption-tests (v4_ore_block 1 passed, 5 ignored without a database); mise run types:check no drift; mise x --env test -- cargo nextest run -p stack-encrypt --all-features 695 passed, 1 skipped; go test ./stashgen/... ./cmd/stashgen/... ok; go vet clean. CI is green at this head.
| three fuzz crates, the two Go guests and the EQL v4 block ORE timing | ||
| harness's ciphertext generator | ||
| (`packages/eql/tests/timing/ore_block_v2/ctgen`) are detached workspaces | ||
| too, each with its own `Cargo.lock`. None of them has a Dependabot entry. |
There was a problem hiding this comment.
LOW (docs): this is wrong for five of the six. .github/dependabot.yml lines 254–260 list /packages/stack-auth/fuzz, /packages/stack-kms/fuzz, /packages/stack-encrypt/fuzz, /languages/golang/encrypt/guest and /languages/golang/auth/guest under a lockfile-only cargo entry (held by scripts/__tests__/dependabot-cargo-cross-workspace.test.mjs). Only ctgen is unlisted, and that file's own comment still says "five detached workspaces". Either add /packages/eql/tests/timing/ore_block_v2/ctgen to that directories: list (it shares no crate by path, so the cross-workspace rule does not apply) and fix the count there, or reword this to say ctgen alone has no entry and why.
| for _, family := range families { | ||
| orderIndex[family+"Ord"] = record.Ope | ||
| orderIndex[family+"OrdOpe"] = record.Ope | ||
| orderIndex[family+"OrdOre"] = record.OreBlock |
There was a problem hiding this comment.
LOW (coverage): this test reads the catalog only, and no testusers model carries an *OrdOre or TextSearchOre field, so no Go test seals or queries a block ORE type through the guest. The OPE types have Ranking and TestV4Ordered*, and TextSearch has TestV4TextSearchStoresEveryTermAndQueriesCarryOne; the block ORE types have nothing on the Go side. The guest's native_ops.rs tests run natively, so ore-rs is never executed on wasm32 in CI; a wasm32-only panic, or a term the guest writes differently from native Rust, would show up in a customer program first. A model with an IntegerOrdOre and a TextOrdOre field, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL as Ranking is, would close it (the nonce makes ob non-deterministic, so the fixture would compare everything but c and ob and check ob through the comparator).
2189d5a to
b4acdbf
Compare
|
Force-pushed b4acdbf (from 2189d5a). Two content changes, folded into their owning commits: the v4 shape check no longer uses sub-selects, which makes a comparison about 1.5–2× faster, and it now refuses chained terms over 1,366 blocks (79b5689). There are two new PostgreSQL tests (b4acdbf). Details are in the review threads. v3's install and both uninstall bundles are byte-identical. |
30d3a35 to
61eb180
Compare
b4acdbf to
b079a84
Compare
|
Rebased onto #1274's new head (61eb180), after #1262, #1263 and #1264 merged into
The lockfile conflicts resolved to the new base plus ore-rs 0.9 and its dependencies. Every |
61eb180 to
422f4a1
Compare
b079a84 to
380c49a
Compare
|
Rebased onto #1274's new head (b079a84 → 380c49a). The block ORE commit changed with it: |
422f4a1 to
bd9091a
Compare
380c49a to
0d96d0a
Compare
bd9091a to
92b0e1c
Compare
0d96d0a to
488a100
Compare
|
Rebased onto |
…index `IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block ORE under its own key domain, over the order keys the CLLW indexes use: `ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and the 128-bit integers. The legacy 8-bit scheme is not used. `sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so it compares by the scheme's comparator, never by its bytes, and `from_bytes` checks a stored term's structure. A query is the full ciphertext too: EQL v4's comparator reads the right half of the operand a btree index passes second. Text whose fold is longer than 1023 bytes or ends in NUL is refused, on the query path before any key is touched. The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte. BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a `match` over it needs an arm for it.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
92b0e1c to
213d998
Compare
488a100 to
4e8bd65
Compare
|
Rebased onto #1274's new head 213d998 (488a100 → ba28e6d), itself on
Verified at the tip: every |
4e8bd65 to
ba28e6d
Compare
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:
- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
doubling on two bigint halves with no branch on the data, then AES under
the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
(header, block count, exact length, padding policy and the block counts
it can produce, every symbol below 64). A malformed term raises, naming
its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
fixed-width terms of different block counts and chained terms padded
under different policies. For terms of one shape it runs a constant
operation count: one statement per block evaluates both comparisons and
latches the first difference arithmetically, the hash runs once on every
path, and the result is selected without a branch. Chained terms compare
lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.
The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.
`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block` (`record.OreBlock`) in `se_targets`, and the engine answers its orderings through it. The checker accepts it in a type's indexes and as an operator's index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a field reaches it only by naming such a type with `encrypt_into=`, so it has no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still refuse it. The static test engine produces `TextOrdOre`, `TextSearchOre` and `IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`, `DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their `ob` term is Stack Encrypt's block ORE term, one hex element holding the whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its stored value takes the empty filter, and its query carries only the term its operator reads. In the target table an `ob` term is the index `ore_block`, not `ore`, which is CLLW ORE and has no EQL domain, and the block-ORE reason in `encryption_gap` is gone. The unproducible types are the date, timestamp and numeric families, which have no Rust plaintext type yet, and the two JSON types. `tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts in PostgreSQL, both ways round, against the plaintext order and ore-rs's own comparator: equal values, a first difference in every block, shared prefixes and text in three padding buckets. It checks the hash bit against ore.rs's vectors, that malformed and mismatched terms raise, and that a btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres` runs it. The Go catalog, the guest's tests and the Go operator test follow. ADR-0009 is amended. The EQL v4 range tests (tests/encryption/tests/v4_range.rs) now derive their order terms from the engine. They arrived from the range domains with cipherstash-client's ORE and OPE indexers writing the `ob` and `op` of every row and bound, which was the only source of block ORE terms while EQL's ordered types were not producible. The comparator this series adds refuses that 8-bit layout, and this commit is where the `OrdOre` types become producible, so the rows are real `IntegerOrdOre`, `TextOrdOpe`, ... values and each bound is the term the single-bound query of that type carries. The smoke test ranges over the families whose ordered types are producible and records that the date, timestamp and numeric ones are not.
|
Pushed 45a79a0 (ba28e6d → 45a79a0). The only change is the EQL 3.1 upgrade note, |
ba28e6d to
45a79a0
Compare
Summary
Stack Encrypt can now write the block ORE EQL v4 types:
IntegerOrdOre,BigintOrdOre,RealOrdOre,DoubleOrdOre,TextOrdOreandTextSearchOre. Before this, those column types were listed but refused, because EQL stored one kind of order-revealing ciphertext and the engine made another. (ORE, order-revealing encryption, lets PostgreSQL sort and range-query encrypted values without decrypting them. EQL is the SQL extension that stores and compares those values;eql_v4is the version Stack Encrypt writes.)The engine now derives ore-rs's 6-bit block ORE, and the EQL v4 SQL gets a comparator for that format, under names of its own (
ore_block_v2), so nothing in v4 reuses the name of EQL v3's older 8-bit format. The EQL v3 SQL bundle is unchanged, byte for byte.Changes
stack-encrypt (breaking)
IndexSpec::OreBlock(wire keyore_block), termsem::OreBlockTerm:ore_rs::vitaminc::Bit6BlockOrefor values up to ten bytes (integers to 64 bits, floats),ore_rs::vitaminc::ChainedBlockOrefor text, bytes and 128-bit integers. Never the legacy 8-bitBlockOre.TermErrorvariants.IndexSpecis exhaustive, so the new variant breaks amatchover it.OreandOpeindexes are untouched.EQL SQL (v4 only)
sem/ore_block_256/functions.sqlholds the v4 comparator,compare_ore_block_v2_term: shape checks, the σ-MMO hash bit through pgcrypto, a constant operation count for terms of one shape, and errors for malformed or mismatched terms.{{ore_block}}. It rendersore_block_256in v3 andore_block_v2in v4, and a literalore_block_256in the source is refused. The hand-written SQL, the pin script, the generated extractors and the ORE fallback use it.tests/timing/ore_block_v2/: a hand-run timing harness with a detached ciphertext generator.eql-bindings / eql-codegen
OreBlock256implementsEncryptFrom, andTextSearchOrefollowsTextSearch(empty filter when stored, one term per query).obterm maps to the indexore_block, notore(CLLW).Go
record.OreBlockandstashgen.IndexOreBlock(not a tag word), checker parsing, the static test engine, and the regeneratedencrypt/eqlcatalog.Docs
@cipherstash/eqlchangeset.stash-postgresandstash-indexingdescribe the v3 surface only and name no v4 object, so they are unchanged.Verification
Run locally on macOS (Apple silicon), on the branch as pushed:
mise run --force buildat the base (4e44e5ba6, before the latest test-only base move) and at this branch:cmpofrelease/cipherstash-encrypt.sqlandrelease/cipherstash-encrypt-uninstall.sqlpasses (SHA-2568eccb1cb…andb1b5131b…both sides). In the v4 bundleore_block_256now appears only in source file paths, andore_block_v2in 173 places.mise run test:encryption:postgrespasses, including the newtests/encryption/tests/v4_ore_block.rs:compare_ore_block_v2_termboth ways round, and must match both the plaintext order and ore-rs's own comparator. Fixed-width:i32andi64with a first difference in every block, extremes and neighbours; floats including ±0, ±∞ and NaN; two writes of each value, which compare equal. Chained: text with shared prefixes across three padding buckets and a difference at each of 40 positions; folded equivalents compare equal.Malformed ORE block term: N bytes, naming no bytes: truncated, too long, empty, header only, wrong version, unknown scheme, symbol 64, a count/length mismatch, an unknown padding kind, a count the padding cannot produce. Fixed-width against chained, 6 blocks against 11, two padding policies and a two-term array each raise.<,>=,=andORDER BYoninteger_ord_oreandtext_ord_orecolumns return the right rows through a btree index, andEXPLAINshows the index.tests/timing/ore_block_v2/run.sh 100000, PostgreSQL 17.5 native, two runs; 0 wrong of 1,500 checks): Welch t for first- vs last-block difference, fixed −1.22 / +0.33 and chained +0.74 / +0.09; equal vs first block −0.65 / −0.40; chained prefix (block counts settle it) vs first block −2.33 / −4.01. All under |t| = 5. The prefix figure is about 40 ns a comparison, below one statement; see the harness README.tests-crates.yml):cargo fmt --check, clippy-D warnings,cargo nextest run --workspace --all-features,mise run test:doc,mise run doc, the examples build and thecargo publish --dry-runof the stack-* crates all pass.cargo mutants --in-diffon stack-encrypt found 91 mutants, 21 missed. Tests were added for every one (OreBlockTerm's views,DebugandPartialOrd, the 1023-byte and trailing-NUL bounds, the query path's early refusal,IndexSpec::parse,indexes()). A re-run of those: 48 mutants, 33 caught, 15 unviable, 0 missed.test-eql.ymltasks):test:crates,test:encryption,check:encryption:wasi,types:check,codegen:parity,test:schemas:parity,test:self_contained_v3,test:installer_complete,test:symbol_order_v3,test:v4_isolation,test:build_ordering_helpers, the four matrix inventories,test:matrix:catalog-coverage,docs:validate:sourceandtest:schemapass.test:docs_v3_grepdid not run here: it needs bash 4 (mapfile), and this machine has bash 3.2, so CI covers it.heap_hygienepasses withTextOrdOreandTextSearchOrethrough every index and no new exclusion. The block ORE visitor carries its plaintext inCarried<T>, as the CLLW visitors do.go:test,go:lintandwasm:guest:testpass, andgo generate ./...leaves the tree clean.cargo metadata --lockedpasses on everyCargo.lockit touches.packages/stack-auth/fuzzandpackages/stack-kms/fuzzalready fail--lockedon the base; this branch does not touch them.test:docs_v3_grep(bash 4).pnpm run test:scriptshit 5-second timeouts in release-gate and Go-proxy tests under load, a different set each run, in files this branch does not touch.Related
Closes #1132
Review notes
ore_block_256, the name of cipherstash-client's 8-bit format (256-way blocks). v4 now holds ore-rs's 6-bit format, wire version 2, so it isore_block_v2:eql_v4_internal.ore_block_v2_term,compare_ore_block_v2_term(s),ore_block_v2_hash_bit,ore_block_v2_schemeand the operator class. The name is a render-time placeholder, so the v3 output stays identical. The source directory keeps its name (sem/ore_block_256/), since it is a path, not an object. No released v4 bundle stored anobterm, so nothing installed is renamed under data.compare_{{ore_block}}_term. Each term's shape is checked first (header, block count, exact length, padding policy and the block counts it allows, every symbol below 64). It raises on different schemes, on fixed-width terms of different block counts and on different padding policies. Then one statement per compared block evaluates both comparisons (non-short-circuiting|) and latches the first difference arithmetically. The hash runs exactly once on every path, equal terms included, and the result is selected arithmetically. Chained terms compare lexicographically: when the shorter one's blocks all match, it sorts first. PL/pgSQL is not constant-time at the C level; the timing results are below.OrdOreis not produced here. The smallint family has no plaintext in this branch, and the work that adds it is separate.https://claude.ai/code/session_01NmUu6jin2Kna7qF8WFSfft