Skip to content

feat(eql)!: produce the block ORE EQL v4 types through ore-rs 6-bit block ORE - #1280

Merged
coderdan merged 4 commits into
nextfrom
feat/eql-v4-block-ore
Oct 19, 2026
Merged

coderdan merged 4 commits into
nextfrom
feat/eql-v4-block-ore

Conversation

@coderdan

Copy link
Copy Markdown
Contributor

Summary

Stack Encrypt can now write the block ORE EQL v4 types: IntegerOrdOre, BigintOrdOre, RealOrdOre, DoubleOrdOre, TextOrdOre and TextSearchOre. Before this, those column types were listed but refused, because EQL stored one kind of order-revealing ciphertext and the engine made another. (ORE, order-revealing encryption, lets PostgreSQL sort and range-query encrypted values without decrypting them. EQL is the SQL extension that stores and compares those values; eql_v4 is the version Stack Encrypt writes.)

The engine now derives ore-rs's 6-bit block ORE, and the EQL v4 SQL gets a comparator for that format, under names of its own (ore_block_v2), so nothing in v4 reuses the name of EQL v3's older 8-bit format. The EQL v3 SQL bundle is unchanged, byte for byte.

Changes

stack-encrypt (breaking)

  • New index IndexSpec::OreBlock (wire key ore_block), term sem::OreBlockTerm: ore_rs::vitaminc::Bit6BlockOre for values up to ten bytes (integers to 64 bits, floats), ore_rs::vitaminc::ChainedBlockOre for text, bytes and 128-bit integers. Never the legacy 8-bit BlockOre.
  • Each term carries a random nonce, so terms compare by the scheme's comparator, not by bytes. A query is the full ciphertext.
  • Text whose fold is over 1023 bytes or ends in NUL is refused, with new TermError variants.
  • IndexSpec is exhaustive, so the new variant breaks a match over it.
  • The CLLW Ore and Ope indexes are untouched.

EQL SQL (v4 only)

  • A version block in sem/ore_block_256/functions.sql holds the v4 comparator, compare_ore_block_v2_term: shape checks, the σ-MMO hash bit through pgcrypto, a constant operation count for terms of one shape, and errors for malformed or mismatched terms.
  • The renderer gains a placeholder, {{ore_block}}. It renders ore_block_256 in v3 and ore_block_v2 in v4, and a literal ore_block_256 in the source is refused. The hand-written SQL, the pin script, the generated extractors and the ORE fallback use it.
  • tests/timing/ore_block_v2/: a hand-run timing harness with a detached ciphertext generator.

eql-bindings / eql-codegen

  • The six types are producible. OreBlock256 implements EncryptFrom, and TextSearchOre follows TextSearch (empty filter when stored, one term per query).
  • In the target table an ob term maps to the index ore_block, not ore (CLLW).

Go

  • record.OreBlock and stashgen.IndexOreBlock (not a tag word), checker parsing, the static test engine, and the regenerated encrypt/eql catalog.

Docs

  • ADR-0002 and ADR-0009 amended (2026-10-19).
  • CHANGELOGs for stack-encrypt (breaking), eql-bindings and Go.
  • @cipherstash/eql changeset.
  • Skills checked: stash-postgres and stash-indexing describe the v3 surface only and name no v4 object, so they are unchanged.

Verification

Run locally on macOS (Apple silicon), on the branch as pushed:

  • EQL v3 is byte-identical. mise run --force build at the base (4e44e5ba6, before the latest test-only base move) and at this branch: cmp of release/cipherstash-encrypt.sql and release/cipherstash-encrypt-uninstall.sql passes (SHA-256 8eccb1cb… and b1b5131b… both sides). In the v4 bundle ore_block_256 now appears only in source file paths, and ore_block_v2 in 173 places.
  • PostgreSQL 17.5 (mise's, a local cluster; Docker was down): mise run test:encryption:postgres passes, including the new tests/encryption/tests/v4_ore_block.rs:
    • Pairs of real ore-rs ciphertexts are compared through compare_ore_block_v2_term both ways round, and must match both the plaintext order and ore-rs's own comparator. Fixed-width: i32 and i64 with a first difference in every block, extremes and neighbours; floats including ±0, ±∞ and NaN; two writes of each value, which compare equal. Chained: text with shared prefixes across three padding buckets and a difference at each of 40 positions; folded equivalents compare equal.
    • The hash bit matches all 12 of ore.rs's vectors.
    • Malformed terms raise Malformed ORE block term: N bytes, naming no bytes: truncated, too long, empty, header only, wrong version, unknown scheme, symbol 64, a count/length mismatch, an unknown padding kind, a count the padding cannot produce. Fixed-width against chained, 6 blocks against 11, two padding policies and a two-term array each raise.
    • <, >=, = and ORDER BY on integer_ord_ore and text_ord_ore columns return the right rows through a btree index, and EXPLAIN shows the index.
  • Timing (tests/timing/ore_block_v2/run.sh 100000, PostgreSQL 17.5 native, two runs; 0 wrong of 1,500 checks): Welch t for first- vs last-block difference, fixed −1.22 / +0.33 and chained +0.74 / +0.09; equal vs first block −0.65 / −0.40; chained prefix (block counts settle it) vs first block −2.33 / −4.01. All under |t| = 5. The prefix figure is about 40 ns a comparison, below one statement; see the harness README.
  • Crates (as tests-crates.yml): cargo fmt --check, clippy -D warnings, cargo nextest run --workspace --all-features, mise run test:doc, mise run doc, the examples build and the cargo publish --dry-run of the stack-* crates all pass.
  • Mutation testing: cargo mutants --in-diff on stack-encrypt found 91 mutants, 21 missed. Tests were added for every one (OreBlockTerm's views, Debug and PartialOrd, the 1023-byte and trailing-NUL bounds, the query path's early refusal, IndexSpec::parse, indexes()). A re-run of those: 48 mutants, 33 caught, 15 unviable, 0 missed.
  • EQL (database-free test-eql.yml tasks): test:crates, test:encryption, check:encryption:wasi, types:check, codegen:parity, test:schemas:parity, test:self_contained_v3, test:installer_complete, test:symbol_order_v3, test:v4_isolation, test:build_ordering_helpers, the four matrix inventories, test:matrix:catalog-coverage, docs:validate:source and test:schema pass. test:docs_v3_grep did not run here: it needs bash 4 (mapfile), and this machine has bash 3.2, so CI covers it.
  • Heap hygiene: heap_hygiene passes with TextOrdOre and TextSearchOre through every index and no new exclusion. The block ORE visitor carries its plaintext in Carried<T>, as the CLLW visitors do.
  • Go: all five guests build; go:test, go:lint and wasm:guest:test pass, and go generate ./... leaves the tree clean.
  • Per commit: each commit builds (root and EQL workspaces, both guest builds, stashgen's tests), and cargo metadata --locked passes on every Cargo.lock it touches. packages/stack-auth/fuzz and packages/stack-kms/fuzz already fail --locked on the base; this branch does not touch them.
  • Not run here: the SQLx suites (they need Docker, so CI runs them) and test:docs_v3_grep (bash 4). pnpm run test:scripts hit 5-second timeouts in release-gate and Go-proxy tests under load, a different set each run, in files this branch does not touch.

Related

Closes #1132

Review notes

  • Naming. v4 used to render every block-ORE object as ore_block_256, the name of cipherstash-client's 8-bit format (256-way blocks). v4 now holds ore-rs's 6-bit format, wire version 2, so it is ore_block_v2: eql_v4_internal.ore_block_v2_term, compare_ore_block_v2_term(s), ore_block_v2_hash_bit, ore_block_v2_scheme and the operator class. The name is a render-time placeholder, so the v3 output stays identical. The source directory keeps its name (sem/ore_block_256/), since it is a path, not an object. No released v4 bundle stored an ob term, so nothing installed is renamed under data.
  • Comparator design. See the doc comments on compare_{{ore_block}}_term. Each term's shape is checked first (header, block count, exact length, padding policy and the block counts it allows, every symbol below 64). It raises on different schemes, on fixed-width terms of different block counts and on different padding policies. Then one statement per compared block evaluates both comparisons (non-short-circuiting |) and latches the first difference arithmetically. The hash runs exactly once on every path, equal terms included, and the result is selected arithmetically. Chained terms compare lexicographically: when the shorter one's blocks all match, it sorts first. PL/pgSQL is not constant-time at the C level; the timing results are below.
  • Leakage. A chained comparison reveals the common-prefix length in 6-bit blocks, and a chained term reveals its text's length bucket (16, 32, 64… bytes). Both are ore-rs's documented properties of the scheme.
  • Smallint's OrdOre is not produced here. The smallint family has no plaintext in this branch, and the work that adds it is separate.

https://claude.ai/code/session_01NmUu6jin2Kna7qF8WFSfft

@changeset-bot

changeset-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 45a79a0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@cipherstash/eql Minor
stash Patch
@cipherstash/stack-prisma Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/prisma-example Patch
@cipherstash/stack-drizzle Patch
@cipherstash/stack-supabase Patch
@cipherstash/stack Patch
@cipherstash/wizard Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5c27cc97-cfe6-43b9-806d-6de8da28f53a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
63 0 29 0

Every mutant in the changed lines was caught by a test.

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 4799dd1 against feat/eql-v4-text-search-optional-terms (4430c0c). No blocking defect found. The SQL comparator reproduces ore-rs 0.9.0's comparators exactly, and every check I could run without credentials passes, including the PostgreSQL-backed suite and the timing harness on a scratch PostgreSQL 15.12. Three low-severity notes follow, each inline; nothing here needs to hold the PR.

Findings (most severe first)

  1. Low — the shape check is about half of every comparison. ore_block_v2_scheme runs twice per compare_ore_block_v2_term call and each run issues two SPI sub-selects (the symbol scan and the padding-bucket EXISTS). Measured here on an i32-shaped term: scheme check 3.6 µs, hash bit 2.6 µs, whole comparison 14.6 µs (20,000 calls each, PG 15.12, Apple silicon); v3's comparator on a 408-byte term takes 3.9 µs on its equal path. A sort or btree build re-validates the same stored term at every comparison it takes part in. Both scans can be plain expressions; details inline.

  2. Low — the 1023-byte ceiling is enforced at encryption only; the comparator admits any count the policy allows, up to the wire maximum. That is what ore-rs does too, so this is a note rather than a divergence. A well-formed chained term of 65535 blocks (1,638,398 bytes) needs no key to craft, since every shape check is on public structure; one comparison of it takes ~20 ms here, and every comparison a btree takes against that row pays it. It needs write access to the table, which already allows worse, so this is informational unless you want ore_block_v2_scheme to bound the count to what Stack Encrypt can write (1,366 blocks for a 1,024-byte bucket).

  3. Low — residual data-dependent work in the comparator loop, already acknowledged in the doc comment. substr(...) <> substr(...) is a memcmp that stops at the first differing byte, and the two post-loop reads at first_diff are at a secret-dependent offset. Your harness and my run of it both put every |t| under 5, so this is a suggestion: the tags can be compared as two bigint halves XORed (constant work), and the right block and tag can be latched arithmetically inside the loop so nothing after it is indexed by the first difference, which is what ore-rs's FirstDiff does. Inline on the loop.

Housekeeping, not a defect: tests/timing/ore_block_v2/ctgen is a fourth detached Cargo workspace with its own Cargo.lock; AGENTS.md's list of detached workspaces (three fuzz crates and two Go guests) and .github/dependabot.yml do not know about it yet. Inline on its Cargo.toml.

What I verified as sound

Comparator against ore-rs 0.9.0 (read src/scheme/bit2_w6.rs, chained.rs, width.rs, primitives/hash.rs, ciphertext.rs, vitaminc.rs from the registry copy the lockfile names, and the head functions.sql side by side):

  • Layout and offsets: header 4 (02 02) or 7 (02 03 + 3-byte policy), then count symbols, count 16-byte tags, 16-byte nonce, count 8-byte right blocks; nonce_b = header + 17*cb, right block k at 1-based nonce_b + 17 + 8k, tag k of a at header + ca + 16k + 1. All match total_len, f_at, nonce_at, right_at.
  • Lewi-Wu split: a's xt/f, b's nonce and right block, as FirstDiff::latch(a[n], left_block(a_f,n), right_block(b_right,n), first).
  • Bit extraction: PostgreSQL get_bit(bytea, n) is byte n/8, bit n%8 LSB-first, which is ct_select_byte(right, xt/8) then ct_bit(byte, xt%8), and the same order lsb_mask/indicator_mask_xor write.
  • σ-MMO: GF(2^128) doubling big-endian with the lo top bit carried into hi and 0x87 folded on hi's top bit, XOR nonce, AES-128-ECB under the public ORE-rs.v2.H-pi.1, lsb(π(m)) ⊕ lsb(m); aes-ecb/pad:none on an exact 16-byte block. The 12 vectors pass here.
  • Refusal set equals ore-rs's: version/scheme id, count ≥ 1, ≤ 14 for the fixed scheme, exact length, Padding::from_wire (None only with n = 0, padded_len(0) ≤ 49151) and admits_count (k in 0..=15, bucket > min_bytes, ceil(8n/6) = (8n+5)/6), every symbol < 64, different schemes, different fixed counts, different policies. ore-rs returns None; the SQL raises, naming only the length.
  • Chained: lexicographic over min(ca, cb), shorter first on a full match (ca.cmp(&cb) = sign(ca - cb)); for the fixed scheme ca = cb is enforced so that term is 0.
  • Result selection and the latch are arithmetic; the hash runs on every path.

Security

  • Key derivation: ORE_BLOCK_KEY_DOMAIN (stack-encrypt/sem/ore-block-key/v1) is distinct from the CLLW domains; the 32-byte index key goes to Bit6BlockOre::new/ChainedBlockOre::new, which derive their 16-byte ORE keys under separate labels (block-ore-bit6/key/v2, chained-block-ore/key/v2) and bind scheme, target and context into the term salt through ore-rs's init_with_salt stages. Consistent with ADR-0002 as amended; left_halves_are_pinned and terms_are_keyed_by_their_context hold it.
  • Nonce: 16 bytes from ChaCha20Rng::from_entropy() inside ore-rs, per term.
  • Wiping: the plaintext travels as Carried<T>; ore-rs frames text/bytes in a Zeroizing copy and scrubs the stack; heap_hygiene reads the producible text targets from the table, so TextOrdOre and TextSearchOre are covered without an edit.
  • Limits: 1023 = DEFAULT_MAX_STR_BYTES − the marker byte; trailing NUL refused because NUL padding would alias "a\0" and "a". Both are enforced at derivation (ore-rs's TooLong/TrailingNul, mapped to the new TermErrors) and by the key-free ore_block_text check on the query path, both bounds included. Neither message names the value.
  • Error messages in SQL name a length or a cardinality, never bytes; the test asserts it.

Dependencies and release

  • ore-rs = "0.9.0" from crates.io (default-features = false, vitaminc), not git or path. The Cargo.lock churn in the root, EQL, Go encrypt guest and stack-encrypt fuzz workspaces is ore-rs and its dependencies (block-modes, byteorder, num*, orderable-bytes, subtle-ng) plus vitaminc-* 0.6.0→0.6.1, which ore-rs 0.9.0 requires. protect-ffi's and the auth guest's lockfiles are untouched, so no WASM-unknown build picks up getrandom.
  • cargo metadata --locked passes in the root, packages/eql, both Go guests, packages/stack-encrypt/fuzz and ctgen; packages/stack-auth/fuzz fails on the base too, as the PR says.
  • eql-bindings pins stack-encrypt = "0.3.0", which crates.io does not have yet (max 0.2.0); pre-existing on the base, and the IndexSpec::OreBlock break sits under [Unreleased] in stack-encrypt's CHANGELOG, so the ordering stays "stack-encrypt 0.3.0 before eql-bindings".

v3 bundle and v4 objects

  • mise run --force build at base and head: cipherstash-encrypt.sql and cipherstash-encrypt-uninstall.sql are byte-identical (8eccb1cb…, b1b5131b…). The v4 bundle names ore_block_256 only in @file path comments (5) and ore_block_v2 173 times; installed, eql_v4_internal has no ore_block_256 object, and ore_block_v2_operator_class is the default btree opclass for the type.
  • {{ore_block}} refusal: the_block_ore_name_follows_the_version passes (literal refused except as sem/ore_block_256/).

Breaking surface and docs: CHANGELOGs (stack-encrypt breaking entry under Unreleased, eql-bindings, Go), ADR-0002 and ADR-0009 amendments, the @cipherstash/eql minor changeset. The skills name only v3 objects, so leaving them is right. stored_rows.rs and record_lowering.rs are not in the diff and the suite passes.

Checks run

  • mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 695 passed, 1 skipped.
  • packages/eql: cargo test --locked -p eql-codegen (162), -p eql-bindings --all-features, -p eql-encryption-tests; mise run types:check.
  • go test ./stashgen/... ./cmd/stashgen/... ./internal/...: ok.
  • Scratch PostgreSQL 15.12 (mise's; Docker was down) with the head v4 bundle: the --ignored suites v4_ore_block (5), text_eq, v4_types, v4_order, v4_text_search all pass, including EXPLAIN showing the index for < and ORDER BY through ord_term_ore.
  • tests/timing/ore_block_v2/run.sh 20000: 0 wrong of 1,500; t = +0.93/+1.21 (fixed first vs last), −2.07/−3.41 (equal vs first), +2.48/+2.41 (chained first vs last), −2.14/−0.73 (prefix vs first).
  • CI: all checks green at this head. No prior reviews or review comments on the PR.

Comment thread packages/eql/src/v3/sem/ore_block_256/functions.sql Outdated
Comment thread packages/eql/src/v3/sem/ore_block_256/functions.sql Outdated
Comment thread packages/eql/src/v3/sem/ore_block_256/functions.sql
Comment thread packages/eql/tests/timing/ore_block_v2/ctgen/Cargo.toml
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 4430c0c to 9048302 Compare October 19, 2026 05:38
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 4799dd1 to 59087de Compare October 19, 2026 05:38
@coderdan

Copy link
Copy Markdown
Contributor Author

Restacked onto the new #1274 and force-pushed. Besides hashes: smallint has a Rust plaintext now (#1267), so SmallintOrdOre is producible here alongside the other *OrdOre types (the engine's block ORE target already took i16). v4_ore_block.rs covers it, including an i16 PostgreSQL agreement check, and the reason-text tests follow next's "no Rust plaintext type for this family" wording. The ctgen workspace is now in AGENTS.md, its lock is checked in test-eql.yml, and run.sh builds it --locked.

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-review of the restack to 2189d5a (onto #1274's 30d3a35). Range-diff against 4799dd1: commits 1 (the engine) and 3 (Go) are identical; commit 2 (the SQL comparator and harness) and commit 4 (the bindings) changed.

Substantive, commit 2, responses to review 5566990967.

  • Housekeeping, the ctgen detached workspace: addressed. AGENTS.md lists it, test-eql.yml's rust-crates job runs cargo metadata --locked against its manifest (reachable: packages/eql/tests/** is in both the push filter and the relevant: block, and the job's working directory is packages/eql), and run.sh builds it with --locked. One sentence of the AGENTS.md text is wrong; finding 1 below.
  • The three LOWs on the comparator (the shape check runs twice with two SPI sub-selects each; no upper bound on a chained term's block count; the residual memcmp/substr data dependence) are unchanged. The author left each to the maintainer (1, 2, 3). Still open; the comparator body in functions.sql is byte-identical to 4799dd1.

Rebase adaptations, commit 4 (smallint). SmallintOrdOre is producible, deliberately and consistently. The engine already takes i16 for a block ORE term (sem/mod.rs:1217, i16 => I16, under Bit6BlockOre for targets of at most ten bytes); ENCRYPTION_DOMAINS gains ("smallint","ord_ore") in the same position as the other families; v4/smallint.rs derives it with #[stash(plaintext = i16)]; v4/targets.rs has the TARGETS row (indexes: ["ore_block"], producible) and all three _named arms; eql_gen.go agrees (mise run types:check no drift at this head); the guest's targets.rs and native_ops.rs, eql_test.go, query_test.go (families now includes Smallint), the codegen tests, tests/targets.rs, both READMEs, both CHANGELOGs and the changeset list it. The v4_ore_block.rs changes are this adaptation, not a review response: SmallintOrdOre joins the shape table, and an i16 PostgreSQL agreement check runs over [MIN, -300, -1, 0, 1, 63, 64, 300, MAX], which crosses a 6-bit block boundary at 63/64. The refusal tests move to DateOrdOre with the base's "no Rust plaintext type" wording. The run.sh change is the --locked flag only. No fixture changed.

New findings

  1. LOW, docs (inline on AGENTS.md): "None of them has a Dependabot entry" is wrong for five of the six detached workspaces. .github/dependabot.yml (lines 254–260) lists /packages/stack-auth/fuzz, /packages/stack-kms/fuzz, /packages/stack-encrypt/fuzz, /languages/golang/encrypt/guest and /languages/golang/auth/guest under a lockfile-only cargo entry, held by scripts/__tests__/dependabot-cargo-cross-workspace.test.mjs. Only ctgen is unlisted, and that file's comment still counts "five detached workspaces". A reader of AGENTS.md would either add a duplicate entry for the fuzz crates or take ctgen's omission as the convention. Either add /packages/eql/tests/timing/ore_block_v2/ctgen to that directories: list (it shares no crate by path, so the cross-workspace rule does not bite) and update the count, or reword the sentence to say ctgen alone is unlisted and why.
  2. LOW, coverage (inline on query_test.go): no Go test seals or queries a block ORE type through the guest. TestTheCatalogListsEachTypesOperators reads the catalog only, and no testusers model carries an *OrdOre or TextSearchOre field, so there is no Go-sealed row and no Lt term for ob checked from Go, where the OPE types have Ranking and TestV4Ordered* and TextSearch has TestV4TextSearchStoresEveryTermAndQueriesCarryOne. The guest's own tests (native_ops.rs) run natively, so ore-rs is never executed on wasm32 anywhere in CI; a wasm32-only panic or a term the guest writes differently from native Rust would surface in a customer program first. A model with an IntegerOrdOre and a TextOrdOre field, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL the way Ranking is, would close it (the nonce makes ob non-deterministic, so the fixture would compare everything but c and ob and check ob through the comparator).

Nit: the PR body still says "Smallint's OrdOre is not produced here" and "the six types are producible"; the restack comment has it right.

Checks (at this head, no credentials): cargo test --locked -p eql-codegen -p eql-bindings --all-features (162 + 116 unit, all integration suites pass), -p eql-encryption-tests (v4_ore_block 1 passed, 5 ignored without a database); mise run types:check no drift; mise x --env test -- cargo nextest run -p stack-encrypt --all-features 695 passed, 1 skipped; go test ./stashgen/... ./cmd/stashgen/... ok; go vet clean. CI is green at this head.

Comment thread AGENTS.md
three fuzz crates, the two Go guests and the EQL v4 block ORE timing
harness's ciphertext generator
(`packages/eql/tests/timing/ore_block_v2/ctgen`) are detached workspaces
too, each with its own `Cargo.lock`. None of them has a Dependabot entry.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW (docs): this is wrong for five of the six. .github/dependabot.yml lines 254–260 list /packages/stack-auth/fuzz, /packages/stack-kms/fuzz, /packages/stack-encrypt/fuzz, /languages/golang/encrypt/guest and /languages/golang/auth/guest under a lockfile-only cargo entry (held by scripts/__tests__/dependabot-cargo-cross-workspace.test.mjs). Only ctgen is unlisted, and that file's own comment still says "five detached workspaces". Either add /packages/eql/tests/timing/ore_block_v2/ctgen to that directories: list (it shares no crate by path, so the cross-workspace rule does not apply) and fix the count there, or reword this to say ctgen alone has no entry and why.

for _, family := range families {
orderIndex[family+"Ord"] = record.Ope
orderIndex[family+"OrdOpe"] = record.Ope
orderIndex[family+"OrdOre"] = record.OreBlock

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW (coverage): this test reads the catalog only, and no testusers model carries an *OrdOre or TextSearchOre field, so no Go test seals or queries a block ORE type through the guest. The OPE types have Ranking and TestV4Ordered*, and TextSearch has TestV4TextSearchStoresEveryTermAndQueriesCarryOne; the block ORE types have nothing on the Go side. The guest's native_ops.rs tests run natively, so ore-rs is never executed on wasm32 in CI; a wasm32-only panic, or a term the guest writes differently from native Rust, would show up in a customer program first. A model with an IntegerOrdOre and a TextOrdOre field, sealed through the deterministic guest, opened in Rust and compared in PostgreSQL as Ranking is, would close it (the nonce makes ob non-deterministic, so the fixture would compare everything but c and ob and check ob through the comparator).

@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 2189d5a to b4acdbf Compare October 19, 2026 06:19
@coderdan

Copy link
Copy Markdown
Contributor Author

Force-pushed b4acdbf (from 2189d5a). Two content changes, folded into their owning commits: the v4 shape check no longer uses sub-selects, which makes a comparison about 1.5–2× faster, and it now refuses chained terms over 1,366 blocks (79b5689). There are two new PostgreSQL tests (b4acdbf). Details are in the review threads. v3's install and both uninstall bundles are byte-identical.

@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 30d3a35 to 61eb180 Compare October 19, 2026 06:32
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from b4acdbf to b079a84 Compare October 19, 2026 06:47
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto #1274's new head (61eb180), after #1262, #1263 and #1264 merged into next. Two fixes went into feat(stack-encrypt)!: derive ore-rs 6-bit block ORE terms as a fifth index (now 77bb17e), both needed because next moved under it:

  • next replaced the hand-written index-key parser with a serde wire form (IndexWire, OutputWire, and Serialize for IndexSpec). ore_block is now a variant of all three, and the existing round-trip test, which already lists IndexSpec::OreBlock, holds it.
  • One query test built its plan from FfiValue, which refactor(stack-encrypt)!: remove FfiValue #1264 removed. It now builds it from Tree, as its neighbours do.

The lockfile conflicts resolved to the new base plus ore-rs 0.9 and its dependencies. Every Cargo.lock passes cargo metadata --locked at every commit. At the head: fmt, nextest (1236 passed), clippy -D warnings, guest tests with and without eql, the EQL crate tests and types:check, all guests built, the Go suite, and no go generate drift.

@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 61eb180 to 422f4a1 Compare October 19, 2026 08:06
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from b079a84 to 380c49a Compare October 19, 2026 08:26
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto #1274's new head (b079a84 → 380c49a). The block ORE commit changed with it: OreBlockVisitor holds its plaintext in zeroize::Zeroizing and encrypts by reference through vitaminc's Encrypt, as the ORE and OPE visitors do after #1235's reduction of sem::Orderable to const TARGET; and, per the same decision, u8 and i8 have no block ORE term either (256 values rank every row), so integers_order_by_value_at_their_width covers 16 to 64 bits, the boolean refusal is a compile error rather than TermError::NoOrderTerm, and the CHANGELOG entry says so. Block ORE term pins did not move. At the tip: fmt, nextest (1235), clippy -D warnings, rustdoc, guest tests with and without eql, EQL crate tests and types:check, all five guests, the Go suite, go vet, no go generate drift; cargo mutants --in-diff over the PR: 92 tested, 63 caught, 29 unviable, none missed. The PostgreSQL and timing runs were not repeated: the comparator SQL is untouched.

@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 422f4a1 to bd9091a Compare October 19, 2026 08:39
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 380c49a to 0d96d0a Compare October 19, 2026 08:39
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from bd9091a to 92b0e1c Compare October 19, 2026 09:27
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 0d96d0a to 488a100 Compare October 19, 2026 09:27
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto next at 0800375 (after #1283 merged main back in). Only lockfiles changed, by exactly the crate moves that merge brought; the commits' content is otherwise identical to the previous head. Every Cargo.lock passes --locked at every commit, and the full Rust, guest, EQL and Go suites pass at the chain's tips against the merged lock.

…index

`IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block
ORE under its own key domain, over the order keys the CLLW indexes use:
`ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every
integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and
the 128-bit integers. The legacy 8-bit scheme is not used.

`sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so
it compares by the scheme's comparator, never by its bytes, and
`from_bytes` checks a stored term's structure. A query is the full
ciphertext too: EQL v4's comparator reads the right half of the operand a
btree index passes second. Text whose fold is longer than 1023 bytes or
ends in NUL is refused, on the query path before any key is touched.

The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte.

BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a
`match` over it needs an arm for it.
@coderdan
coderdan marked this pull request as ready for review October 19, 2026 12:29
@coderdan
coderdan requested a review from a team as a code owner October 19, 2026 12:29
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T12:37:02.479046Z 488a100 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-19T12:37:38.923584Z 488a100 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 92b0e1c to 213d998 Compare October 19, 2026 12:33
@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 488a100 to 4e8bd65 Compare October 19, 2026 12:33
Base automatically changed from feat/eql-v4-text-search-optional-terms to next October 19, 2026 12:55
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto #1274's new head 213d998 (488a100 → ba28e6d), itself on next at d827132 (#1272, #1243, #1265 and #1244 merged). Conflicts, each resolved in the commit where it arose:

Verified at the tip: every Cargo.lock passes --locked at every commit; fmt (root and EQL workspaces), nextest (1251 passed), clippy -D warnings, trybuild, guest tests with and without eql, the EQL crates (the encryption crate's DB-free tests included), types:check and codegen:parity, all guests built, the Go suite, no go generate drift. The PostgreSQL half of the encryption tests could not run locally (no database here) and is left to CI. Mutants over the packages/stack-encrypt diff: 94 mutants, 63 caught, 31 unviable, 0 missed; the v4_range.rs change is outside that diff.

@coderdan
coderdan force-pushed the feat/eql-v4-block-ore branch from 4e8bd65 to ba28e6d Compare October 19, 2026 13:05
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:

- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
  doubling on two bigint halves with no branch on the data, then AES under
  the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
  (header, block count, exact length, padding policy and the block counts
  it can produce, every symbol below 64). A malformed term raises, naming
  its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
  fixed-width terms of different block counts and chained terms padded
  under different policies. For terms of one shape it runs a constant
  operation count: one statement per block evaluates both comparisons and
  latches the first difference arithmetically, the hash runs once on every
  path, and the result is selected without a branch. Chained terms compare
  lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.

The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.

`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block`
(`record.OreBlock`) in `se_targets`, and the engine answers its orderings
through it. The checker accepts it in a type's indexes and as an operator's
index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a
field reaches it only by naming such a type with `encrypt_into=`, so it has
no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still
refuse it.

The static test engine produces `TextOrdOre`, `TextSearchOre` and
`IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it
does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`,
`DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their
`ob` term is Stack Encrypt's block ORE term, one hex element holding the
whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable
plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its
stored value takes the empty filter, and its query carries only the term
its operator reads.

In the target table an `ob` term is the index `ore_block`, not `ore`,
which is CLLW ORE and has no EQL domain, and the block-ORE reason in
`encryption_gap` is gone. The unproducible types are the date, timestamp
and numeric families, which have no Rust plaintext type yet, and the two
JSON types.

`tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts
in PostgreSQL, both ways round, against the plaintext order and ore-rs's
own comparator: equal values, a first difference in every block, shared
prefixes and text in three padding buckets. It checks the hash bit against
ore.rs's vectors, that malformed and mismatched terms raise, and that a
btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres`
runs it. The Go catalog, the guest's tests and the Go operator test follow.
ADR-0009 is amended.

The EQL v4 range tests (tests/encryption/tests/v4_range.rs) now derive
their order terms from the engine. They arrived from the range domains
with cipherstash-client's ORE and OPE indexers writing the `ob` and `op`
of every row and bound, which was the only source of block ORE terms
while EQL's ordered types were not producible. The comparator this
series adds refuses that 8-bit layout, and this commit is where the
`OrdOre` types become producible, so the rows are real `IntegerOrdOre`,
`TextOrdOpe`, ... values and each bound is the term the single-bound
query of that type carries. The smoke test ranges over the families
whose ordered types are producible and records that the date, timestamp
and numeric ones are not.
@coderdan

Copy link
Copy Markdown
Contributor Author

Pushed 45a79a0 (ba28e6d → 45a79a0). The only change is the EQL 3.1 upgrade note, packages/eql/docs/upgrading/v3.1.md, with U-001 (the text search query terms and raises from #1274) and U-002 (this PR's block ORE scheme change), and the two changesets now cross-link their notes. It answers the Codex P1 left on #1274 after that PR merged. In the SQL commit a6619c4 (the note and the #1274 changeset link) and the bindings commit 45a79a0 (this PR's changeset link). The Postgres half of the encryption tests passed in CI at ba28e6d; this head changes no code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant