[coderdan]

Summary

EQL (Encrypt Query Language) is the SQL bundle that stores and queries encrypted values in PostgreSQL; EQL v4 is the version for values written by Stack Encrypt, the Rust encryption engine in packages/stack-encrypt. Its json_tree type (unreleased) stores an encrypted JSON document as one entry per JSON node: a small ciphertext c holding that node alone, plus keyed tags that let PostgreSQL search and walk the document without the decryption key. Two of those tags matter here. A selector s is a keyed hash of a node's path. A token is a secret 16-byte value per node that the client hands the server in a query: from it the server can find that node's children, through each child's parent tag p, and so walk the subtree under the node.

This PR shrinks what each entry stores from nine fields to four (s, c, p, op) and changes how the server walks a subtree, so a walk no longer costs the document's widest node for every node it visits. Stacked on #1325. The stored format changes with it, so it has to land before the family ships.

• Three fields had no reader once the containment projection q was stored (perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325). q is the document's selector pairs grouped by array element, so PostgreSQL's own jsonb @> over it decides containment. Those three fields were the per-entry value selector vs and the two group tags n and g. The engine now reads q back as the projection instead of rebuilding it from the entries.
• Two more fields existed only so the server could continue the token chain: the wrapped token w and the ordinal o. A child's token is now H("t" ‖ t(parent) ‖ LE32(o)), a hash of its parent's token and its position among its siblings, which the server derives as it walks. The positions under a node run 0, 1, 2, … with no gaps, so the walk probes them in turn and stops at the first miss. A subtree of N nodes costs about 2N probes (a SHA-256 and a lookup each) plus N token hashes, where it cost every position up to the document's widest node at every node visited.
• The client can no longer derive an object member's token: its position is a rank among sibling keys the client does not know. So a query sends the deepest token it can derive (a top-level field's, then any array positions directly after it) plus one gate per further path step: that step's selector, or #n for an array position. The server descends through the gates and returns the node they reach at depth d = 0, with its descendants beneath it.

Changes

• SQL (packages/eql/src/v3/json_tree/, rendered into EQL v4 only):
  • The entry validator accepts s, c, p and op only, so a document written by the earlier build of this unreleased type is refused.
  • eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64) replaces the earlier (doc, token, selector, budget) form. It walks the gates and returns the reached node at d = 0 with its descendants. json_tree_subtree(doc, token, budget) is now that function with no gates, and still returns the token's node's descendants from d = 1.
  • json_tree_children and json_tree_child_count take the same optional gates.
  • The budget is unchanged: when the document's root is sealed (its ciphertext holds the whole document) and a walk would return more than budget entries, the walk returns the root entry alone at j = 0, d = 0, and the client opens it and selects the path itself.
  • eql_v4_internal.json_tree_child_token is added; json_tree_unwrap, json_tree_xor16 and json_tree_fanout are removed.
• stack-encrypt (src/json/):
  • JsonEntry is {s, c, p, op}, and JsonDocument carries its projection, which the wire form requires as q.
  • JsonSubtreeKey holds a token and gates().
  • JsonDocument::children() probes positions in turn.
  • Rows from a gated walk open to the reached node's value.
• eql-bindings (src/v4/json_tree.rs): JsonTreeEntry drops vs, o, w, n and g, and the typed document hands q to the engine.
• Tests in all three places are updated for the new fields and the gated walk.
• Design doc (docs/plans/2026-10-17-json-index-stack-encrypt-spike.md): the Phase 3b implementation note carries the measured outcome below. The earlier sections, which still describe the wrapped token, group tags and stored ordinal, are kept as the record, with dated notes where they would otherwise mislead.
• Doc comments that still described the old format are corrected (the header's SQL reader, the projection's source, the walk's cost).
• Changeset: @cipherstash/eql minor.

Verification

• cargo nextest run -p stack-encrypt --all-features: 733 passed.

• cargo test -p eql-bindings --all-features json_tree: passed.

• mise run test:encryption:postgres against a fresh database with the v4 bundle built from this tree: all suites passed, including the json_tree oracle, which compares containment over the encrypted column with jsonb @> over the plaintexts (postgres_containment_over_the_encrypted_column_is_jsonb_containment, postgres_extracts_walks_and_orders_by_path).

• After the docs follow-up commit: cargo clippy --all-features --all-targets -D warnings and cargo fmt --check clean for stack-encrypt and eql-bindings, mise run test:schema passed, and mise run docs:validate's coverage and required-tags checks passed. Its documented-sql step needs the Docker database on port 7432 and could not connect locally; CI runs it.

• Benchmarks, cipherstash/benches@2d9d5f2 (branch feat/json-tree-benches, report/JSON_TREE_COMPARISON.md, "Third measurement"). They ran at 10,000 and 100,000 rows on one machine (Apple M1 Max, PostgreSQL 17.10), over the same documents before and after, so every scenario returned the same rows. Query-only criterion medians in ms, before → after, on the nested corpus (about 213 entries a document):

  scenario	10,000 rows	100,000 rows
  subtree/orders, default budget	9.48 → 5.71	8.77 → 5.46
  subtree_at/items ($.orders[*].items), default budget	12.95 → 8.21	10.92 → 7.18
  subtree/orders, walked out (budget 0)	22.97 → 10.92	24.57 → 11.35
  subtree_at/items, walked out (budget 0)	23.12 → 10.93	24.83 → 11.27
  subtree/user	3.90 → 2.38	3.99 → 2.45
  path_query/wildcard	2.24 → 1.64	1.80 → 1.41
  field_order/nested	4.81 → 3.81	5.23 → 4.12
  children/orders	1.63 → 1.54	1.64 → 1.49
  child_count/orders	1.62 → 1.53	1.66 → 1.52

  children and child_count moved only 6 to 9%. The report's explanation, not yet tested, is that a per-call fixed cost (detoasting the document and building the parent-tag map) dominates them, and this change does not touch it.

  Containment and field queries are unchanged within noise. The flat corpus read +7% to +13% on byte-identical operands, plans and indexes, and the native jsonb controls over unchanged plaintext drifted +5% to +19% in the same run, so this reads as machine noise rather than the format.

• Sizes:

  • Nested document: 113,458 → 85,157 characters (−25%); sv 94,324 → 66,023 (−30%).
  • Nested pg_column_size: 81,264 → 62,143 bytes.
  • Nested heap: 810 MB → 621 MB at 10,000 rows, 8,093 MB → 6,201 MB at 100,000.
  • GIN index unchanged (40 MB / 358 MB). GIN build 5 s at 10,000 rows; 58 → 65 s at 100,000, one sample.
  • Flat document: 2,346 → 2,136 characters; pg_column_size 2,452 → 2,250 bytes.

• Correctness: a new benches task, check:v4:json_tree <N>, ran at both tiers. It decrypted every walk form for 25 random rows and compared each with the plaintext, and it compared the encrypted id sets with native jsonb for 10 needles per predicate. 462 passed, 0 failed, at both tiers.

Related

• Refs perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325, the base of this stack.
• Refs json_tree: the smaller performance findings from the first benchmarks #1324, whose item 4 (a nested document's size) this partly addresses: the stored document is 25% smaller. The encoding and nonce levers listed there are untouched, and -> (item 1) is 15 to 19% faster only because sv is shorter.

Review notes

• Start with the walk comment block in functions.sql and json_tree_subtree_at, then subtree_key_for in stack-encrypt/src/json/derive.rs, which decides what the client derives and what it gates.
• The one semantic subtlety is in DecryptInto<Value> for JsonSubtree: a d = 0 row means "the node the gates reached", so its value is returned, rather than a container assembled from the rows. A sealed-root fallback also comes back at d = 0, at j = 0, and is told apart by that.
• Trust is unchanged: only c is authenticated, through its AAD (the associated data the AEAD binds), and q was already what containment ran over. The w mask's binding to c is gone. A swapped ciphertext no longer breaks the walk on the server, but it still fails when the client opens the entry, because c is bound to s and its position j in the document.
• At rest the format leaks less: there are no stored positions to count children from.
• The changesets from feat(stack-encrypt, eql): the JSON index and the json_tree EQL v4 domain family #1288 and perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325 still describe the format this PR replaces. They will sit beside this one in the same release's changelog, and this one says what superseded them.

What a query looks like to a user

Users never see selectors, tokens or gates. The SDK derives them from a path and a plaintext needle, and the only encrypted thing in the SQL text is one typed operand.

SQL, as the v4 bundle ships it, for a column profile of type public.eql_v4_json_tree:

-- containment, served by the GIN index; $1 is the encrypted needle
SELECT id FROM users WHERE profile @> $1::eql_v4.query_json_tree;

-- range at a path, served by a btree on the functional index; $sel is the path selector
SELECT id FROM users WHERE (profile -> $sel) >= $2::eql_v4.query_json_tree_leaf;
SELECT id FROM users ORDER BY eql_v4.ord_term(profile -> $sel);

-- one node, opens on the client
SELECT profile -> $sel FROM users WHERE id = 7;

-- a subtree: the token and the gates this PR introduces; rows open on the client into the value
SELECT e FROM users, eql_v4.json_tree_subtree_at(profile, $tok, $gates::text[]) e WHERE id = 7;

Rust, as stack-encrypt has it on this branch:

let q = cipher.query(op::Contains, &json!({"orders": [{"id": 2}]})).on(&plan).await?;
let b = cipher.query(op::AtPath(op::Gte), &("$.age", 18)).on(&plan).await?;
let k = keyset.json_subtree_key(&"$.orders[*].items".parse()?, options, column).await?;
// k.token() and k.gates() are the arguments of json_tree_subtree_at
let items: Vec<Value> = cipher.decrypt_as(JsonTreeSubtree::new(rows), column).await?;

TypeScript through Drizzle, as the EQL v3 surface reads today; a v4 column keeps it and gains the walk, which has no adapter shape yet:

db.select().from(users).where(await ops.contains(users.profile, { orders: [{ id: 2 }] }))
db.select().from(users).where(await ops.selector(users.profile, "$.age").gte(18))
db.select().from(users).orderBy(ops.asc(ops.selector(users.profile, "$.age")))

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
18	0	12	0

Every mutant in the changed lines was caught by a test.

[changeset-bot]

🦋 Changeset detected

Latest commit: 6adf37f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Minor
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR