[coderdan]

Summary

EQL v4's json_tree column stores an encrypted JSON document as one entry per JSON node. To read part of a document, the client derives a "subtree key" for a path and the database walks the stored entries and returns the ones at that path. For a top-level field ($.profile), and for a path ending in an array position right after one ($.tags[2]), the walk returned the node's descendants and never the node itself. So {}, [], a scalar and a missing field all came back as no rows, and the client could not tell them apart. PostgreSQL's own jsonb answers {}, [], the scalar and NULL. Now every subtree read returns the node it names, with everything beneath it, so the client gets the same four answers jsonb does. No stored field is added.

Stacked on #1358, which adds the client-side checks these rows go through.

Changes

SQL (packages/eql/src/v3/json_tree/functions.sql, EQL v4 only)

• eql_v4.json_tree_subtree(doc, token, budget) becomes json_tree_subtree(doc, token, selector, budget DEFAULT 64). This is a deliberate signature change to an unreleased function. A top-level field is the one entry with no parent tag that carries its selector (with the root not sealed, the root is). The walk returns that entry at depth 0, then its descendants. When the row has no such entry it returns nothing, which is "no node at the path".
• Both walks now run one internal function, eql_v4_internal.json_tree_walk(doc, token, gates, budget, selector). json_tree_subtree_at calls it with no selector, so its behaviour is unchanged.
• The budget counts the node. Past the budget the walk still returns the sealed root alone.

Stack Encrypt (packages/stack-encrypt)

• A subtree key for a path ending in an array position no longer chains to the element's token, from which no walk can return the element. The last position becomes a gate from the array's token (#n:<selector>), and the gated walk already returns the element at depth 0. Positions before the last step still chain.
• JsonSubtreeKey::selector() is what json_tree_subtree takes for a key with no gates. The key's serialized form carries it as s.
• JsonSubtree opens to the node's own value ({}, [] and scalars included). As an Option<Value>, it opens to None where the row has no node at the path. That matches ->, which returns SQL NULL for a path the row does not have. As a Value, no node is still refused as OpenedType. As a list, it opens to the reached nodes for a gated key, and to the node's children for a top-level field, as before.
• The sealed-root fallback gives the same answers: {} and [] select as themselves, and a missing path selects nothing.
• The checks from fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358 now always start from the reached node, so a walk with no gates is checked from its head down.

EQL bindings

• JsonTreeSubtree gains DecryptInto<Option<Value>>, and its docs say which SQL function to call for which key.

Docs

• Leakage page: what a subtree read now sends (see below).
• Design record: the "descendants and not the node itself" bullet is marked closed, and there is a new Phase 4 note.
• CHANGELOGs, plus a minor changeset for @cipherstash/eql (the SQL signature change).

What the database learns

• json_tree_subtree now sends the top-level field's selector beside its token. It is the selector -> sends for the same path, so it is nothing new on its own. Together with the token, it tells the database which field the token is for. The stored entry order already showed that the first time the token reached one of the field's members: the entry stored right before a subtree's first member is its parent. What is new is the pairing for a field that has no members in any row the token reaches.
• A path ending in an array position now sends the array's token, not the element's. With it the database can walk every element of that array, in every row, for as long as the column lives. A read of $.tags[*], or of anything under a key below $.tags, already sends the same token.

Both are on the leakage page.

Verification

Run from the worktree root unless noted. PostgreSQL 17 was a disposable container on port 55437, stopped afterwards.

• mise x --env test -- cargo nextest run -p stack-encrypt --all-features: Summary [ 41.859s] 837 tests run: 837 passed, 1 skipped
• cargo test -p stack-encrypt --doc --all-features: test result: ok. 50 passed; 0 failed
• cargo clippy -p stack-encrypt --all-features --all-targets -- -D warnings: clean
• cargo fmt --all --check at the root and in packages/eql: clean
• RUSTDOCFLAGS=-D warnings cargo doc -p stack-encrypt --all-features --no-deps: clean
• packages/eql: cargo test -p eql-bindings --all-features: all pass. mise run types:check: passes. mise run test:encryption: exit 0.
• packages/eql: mise run test:encryption:postgres: exit 0. The json_tree suite: test result: ok. 8 passed.
  • It includes the new postgres_every_subtree_read_answers_as_jsonb_does. That test holds every subtree read to jsonb_path_query_first(plain, 'strict <path>') over the plaintexts. A strict path is used because #> reads a position as an object key. It covers {}, [], scalars (null included), missing nodes and non-empty containers. Each is read at a top-level field, at array positions after one (chained and not), and behind a gate, with the root sealed and not sealed, walked out (budget 0) and past a budget of 1, where reads of more than one entry fall back to the sealed root.
  • The existing walk tests were updated for the new signature and the extra head row.
• SQL checks in packages/eql: test:self_contained_v3, test:installer_complete, test:symbol_order_v3, test:v4_isolation, test:build_ordering_helpers, docs:validate:source and test:docs_v3_grep all pass. docs:validate:documented-sql against the container: Validated: 290, Errors: 0. test:splinter was not run: it lints the eql_v3 schemas only, and this SQL is v4-only.
• New engine tests in packages/stack-encrypt/tests/json_read_checks.rs, module given_a_node_with_nothing_beneath_it. They cover each of {}, [], "text", 0, null, false and a non-empty object at a top-level field, at array positions, behind a gate, with the root open and not, walked and past the budget. They also cover no node (None, and no key retrieved), lists of a head's children, and a wildcard over such nodes.
• Mutation testing, mise x --env test -- cargo mutants -p stack-encrypt --in-diff <(git diff fix/json-tree-extracted-entry-check...HEAD): 28 mutants, 13 caught, 15 unviable, 0 missed.

Related

Closes #1356

Refs #1358 (base of this PR), #1355, #1357.

Review notes

• Start with eql_v4_internal.json_tree_walk in functions.sql, which is the old json_tree_subtree_at body plus the start CTE and the found gate. Then read subtree_key_for, place and open_reached in packages/stack-encrypt/src/json/derive.rs.
• Callers pick the SQL by the key: json_tree_subtree(doc, key.token(), key.selector()) when key.gates() is empty, otherwise json_tree_subtree_at(doc, key.token(), key.gates()). The oracle's subtree_rows helper shows it.
• Like fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358, this touches a few lines of packages/stack-encrypt/tests/json_index.rs (keys for positional paths, and walks that now include the head), which test: hold json_tree containment to jsonb with proptest, and restructure the JSON index tests #1345 rewrites.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

[changeset-bot]

🦋 Changeset detected

Latest commit: 121e1f4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Minor
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
13	0	15	0

Every mutant in the changed lines was caught by a test.

[coderdan]

This PR's mechanism (a top-level field found by its selector, a path's last array position sent as a gate from the array's token) is superseded by #1360, which makes every path's token derivable and adds a self tag per entry, so no read has to send an array's or a field's token to reach a node below it. Its semantics carry over: the four answers, JsonSubtree opening as Option<Value> to None where the row has no node, and the oracle test postgres_every_subtree_read_answers_as_jsonb_does. Write-up in draft PR #1361. Not closing this yet; the plan is to close it once the implementation PR is open.

[coderdan]

Superseded by #1363, which lands the same semantics (the node itself at depth 0, Option<Value> reading an absent node as None, the four jsonb #> answers distinguishable) through derivable tokens and a self tag instead of a selector beside the token and a position gate, so a read of $.tags[2] no longer sends the array's token. The oracle test from this PR carries over there. Design: #1360, PR #1361.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu