feat(stack-encrypt, eql): the JSON index and the json_tree EQL v4 domain family

.cargo/mutants.toml

@@ -108,6 +108,19 @@ exclude_re = [
   # (`entry_key`): its error is discarded (`.ok()`), so its message is never
   # read at all.
   'stack-encrypt/src/dynamic/record\.rs:\d+:\d+: replace entry_key::<impl Deserialize<\x27de> for Keys>::deserialize::<impl Visitor<\x27de> for V>::expecting -> fmt::Result with Ok\(Default::default\(\)\)$',
+  # The JSON index. `unhex` joins a high nibble shifted into the top four
+  # bits with a low nibble below 16: the bits never overlap, so `^` is `|`.
+  'stack-encrypt/src/json/mod\.rs:\d+:\d+: replace \| with \^ in unhex$',
+  # `JsonPath::root` is `Self::default()`.
+  'stack-encrypt/src/json/path\.rs:\d+:\d+: replace JsonPath::root -> Self with Default::default\(\)$',
+  # The sealed root's `None` token: with that arm gone, the root falls to
+  # the arm that wraps its parent's token, and the root has no parent, so
+  # it is `None` there too. `the_sealed_root_has_no_token_and_its_fields_are_heads`
+  # pins the value; the arm stays so the match says it.
+  'stack-encrypt/src/json/derive\.rs:\d+:\d+: delete match arm \(0, true\) in Walker<\x27_>::visit$',
+  # A document always has a root, so its terms are never empty: `false` is
+  # the value `is_empty` returns. The `true` replacement is still caught.
+  'stack-encrypt/src/json/derive\.rs:\d+:\d+: replace JsonTerms::is_empty -> bool with false$',
 ]
 
 # Headroom over the measured baseline before a slow-but-correct mutant is

.changeset/eql-json-tree-domain.md

@@ -0,0 +1,7 @@
+---
+"@cipherstash/eql": minor
+---
+
+**EQL v4 gains the `json_tree` domain family: `public.eql_v4_json_tree`, an encrypted JSON document stored as one entry per node, with `@>` and `<@` that agree with PostgreSQL's own `jsonb @>` across array elements.** The family is new beside `public.eql_v4_json` and is not a port of the v3 SteVec surface: it is what Stack Encrypt's JSON index writes. A document's entries each carry a path selector `s` and a value selector `vs` (containment is a set comparison over the pair), the node's own ciphertext `c` (its label, kind and scalar value; a document's ciphertext is linear in its size, where a SteVec document repeats every subtree at every level), and a token chain (`p`, `w`, `o`) that lets core PostgreSQL walk the subtree under a node from one client-derived token: `eql_v4.json_tree_subtree(doc, token)`, `json_tree_subtree_at(doc, token, selector)`, `json_tree_children` and `json_tree_child_count`. By default the root is sealed, so no query can hand the server a token for the whole column. Containment projects a stored document into an element-nested shape from its group tags (`n`, `g`), `eql_v4.json_tree_query(doc)`, and compares it with the needle (`eql_v4.query_json_tree`) by native `jsonb @>`, so each needle array element must fit inside one stored element, which closes the cross-element false positives of selector-set containment (#1231) for this family; a GIN index on `(eql_v4.json_tree_query(col)::jsonb) jsonb_path_ops` serves it. `->` by selector returns an entry that opens on its own (`public.eql_v4_json_tree_entry`, with the header and ordinal grafted), and `eql_v4.query_json_tree_leaf` orders an extracted number or string leaf with `<`, `<=`, `>`, `>=`. Every other native jsonb operator on the column is blocked, as on the other EQL domains. EQL v4 only (`-- ONLY EQL VERSION: 4`); the v3 bundle is unchanged. The `eql-bindings` crate carries the hand-written `v4::json_tree` types with Stack Encrypt's `EncryptFrom` / `DecryptInto` for the document and the entry.
+
+Why. Stack Encrypt had no JSON index: the operators (`Contains`, `ContainedBy`, `AtPath`) were declared and answered by nothing, and the v4 JSON wire format was unconstrained by anything shipped. This is Phase 1 and 2 of the `json_tree` index's phased plan (#1249), implemented against `docs/plans/2026-10-17-json-tree-index.md`, whose "Implementation notes" section records where the code deviates from the design.

AGENTS.md

@@ -682,6 +682,18 @@ monorepo, which is where the silent failures are.
 - **Changesets for `@cipherstash/eql` go in the repo-root `.changeset/`.** The
   subtree's own `.changeset/` was deleted with the import; there is no second
   one to put them in by mistake.
+- **The JSON tree family is EQL v4 only and spans two crates.** The engine
+  half is `packages/stack-encrypt/src/json/` (the `Json` index, `JsonDocument`,
+  the query terms); the SQL is `packages/eql/src/v3/json_tree/` under
+  `-- ONLY EQL VERSION: 4`; the EQL types are hand-written in
+  `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; and the live
+  oracle, containment over the encrypted column against PostgreSQL's own
+  `jsonb @>` over the plaintexts, is `packages/eql/tests/encryption/tests/json_tree.rs`,
+  run by `mise run test:encryption:postgres`. The design and its
+  implementation notes are `docs/plans/2026-10-17-json-tree-index.md`.
+  A change to the derivations on one side (a PAE domain, the header layout,
+  the projection shape) is a change to the other: the Postgres suite is
+  what holds them together.
 
 ## Working on the Rust crates
 

docs/plans/2026-10-17-json-tree-index.md

@@ -336,6 +336,28 @@ Scope: prove the three changes end to end with real encryption under the fake ke
 
 Each phase is a PR against `next`, where #1209 landed.
 
+## Implementation notes (2026-10-19, Phases 1 and 2)
+
+The index, the SQL and the live oracle landed in the pull request stacked on this document; what follows records where the code is and where it departs from the design above, so the two can be read against each other. Phase 0 was folded into Phase 2, and Phases 3 and 4 remain open.
+
+**Where it is.** The engine half is `packages/stack-encrypt/src/json/`: `Json` is the index (`Index<serde_json::Value>`, term `JsonTerms`), `target::json_document` the sealing operation, `JsonDocument` the stored term and an `EncryptFrom<Value>` / `DecryptInto<Value>` target, `JsonQuery` the needle for `Contains`, `ContainedBy` and `AtPath<Eq>`, `JsonLeafQuery` the operand for the orderings at a path, and `JsonSelector` / `JsonSubtreeKey` the extraction keys, derived through `KeysetCipher::json_selector` and `json_subtree_key`. `IndexSpec::Json(JsonOptions)` is the data form, with the key `"json"` and `{"json": {"sealed_root": …}}` as its wire form; a data plan reads it and refuses to run it (`Reason::JsonIndexPending`). The SQL is `packages/eql/src/v3/json_tree/`, confined to EQL v4; the EQL types are `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; the oracle is `packages/eql/tests/encryption/tests/json_tree.rs`, which stores the design's corpus and random documents encrypted beside their plaintexts and checks that `@>` and `<@` over the encrypted column select the rows PostgreSQL's own `jsonb` operators select over the plaintext column, with the GIN index shown in `EXPLAIN`. The pure-Rust half of that oracle is `packages/stack-encrypt/tests/json_index.rs`, against a model of `jsonb @>` pinned to the answers in `jsonb-semantics.sql`.
+
+**Decisions taken by the code.** Of the eleven listed below: 1, hex for every sixteen-byte field and base64 for `c` and `h`, pending Phase 3; 2, leaf ordering terms bound to the column only, numbers as the `f64` they round to and strings as text, under the scalar `Ope` index's derivation; 3, wildcard arrays only; 4, group tags on, and only on container array elements; 5, the child count is in every container node's ciphertext, and a truncated subtree is `OpenedType`, not a shorter array; 7, plain SHA-256; 8, the option is the boolean `sealed_root`, on by default, recorded in the header's flags; 9, the projection is built from the tags server-side (`eql_v4.json_tree_query`) and `groups_ok` was not needed; 10, position-keyed parent tags on, the ordinal stored as `o`, object members ranked under the parent's token; 11, keyed: a child's token is `PRF(json/token-step ‖ column, PAE(t(parent), 0x01, key))` (`0x02, LE64(index)` for an element) and an object member's rank is `PRF(json/rank ‖ column, PAE(t(parent), key))`, each under its own PAE domain rather than the heads' `json/token/v1`, so a token's holder can neither test key names against the child tokens the walk unwraps nor rank guessed names against the ordinals. Decision 6 is out of scope, as proposed. The first implementation hashed both steps unkeyed, as the token chain section above writes them; it was corrected before any release, so no stored document carries the unkeyed form.
+
+**Deviations.**
+
+- *Nonces are random, not derived from the selector and ordinal.* Every entry seals through the engine's one AEAD path (`vitaminc-encrypt`'s AES-256-GCM with its own random 96-bit nonce, carried in `c`), so there is one leaf cipher in the crate and no second AES-GCM call site. Uniqueness under one document key is then probabilistic (`N² / 2^97` at `N` entries) rather than structural. The entry's AAD still binds its ordinal `j`, so a moved or reordered entry fails to open as the design requires; the cost is twelve bytes per entry, which Phase 3 should count.
+- *The ordinal is `o`, not `i`*, because `i` is the envelope's identifier, which `->` grafts onto an extracted entry. An extracted entry also carries `j`, its ordinal in the document, which its AAD is bound to, and a walk adds `d`, its depth below the start.
+- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4.
+- *Containment is not `STRICT` in SQL.* PostgreSQL inlines a `STRICT` SQL function only when it can prove the body is NULL for a NULL argument, and the `OR` that adds the needle's `alt` probe is not provably NULL; un-inlined, the call was an opaque filter and the GIN index never engaged. `eql_v4.json_tree_contains` and the `@>` / `<@` wrappers over it are declared without `STRICT` and return NULL for a NULL side on their own.
+- *`<@` cannot match a stored root scalar inside a needle root array*, since a stored document carries no `alt` pair. The oracle records the case; `@>` is exact.
+- *Value selectors compare strings byte for byte*, with no Unicode normalization, because `jsonb @>` does, and the oracle is agreement with `jsonb @>`. The scalar equality index normalizes to NFC; a JSON value does not. Numbers canonicalize by value, so `2` and `2.0` are one value, as `jsonb` compares them.
+- *`AtPath<Eq>` is the containment of the needle document that holds the value at the path*, with a position in the path read as the wildcard, rather than a single `{vs}` entry: a leaf inside an array element sits in that element's group of the projection, and the needle has to say which group. Containment of that needle is equality only for a scalar, so a container value is refused (`TermError::JsonValueNotScalar`; `op::Contains` asks the containment question), and the needle carries no `alt`, so `("$", 1)` does not match a stored `[1]`.
+- *The data-plan form and the catalog are not done.* The Go guest, `stashgen`, the target table and `v4::all()` do not know the family; `IndexSpec::Json` crosses the wire and is refused with a reason. The family is hand-written in `eql-bindings` as the SteVec structs are. Both are the next bindings issue.
+- *A document is at most 127 levels deep*, under either root option. The sealed root's plaintext is the document as JSON, and `serde_json` parses JSON back only to 128 levels of nesting, so the 512 levels the walk first admitted sealed documents that then never opened. The bound is the reader's, so a document too deep fails when it is sealed, while the caller still holds it, and not when it is read.
+- *A stored document is read as untrusted input.* Its chain fields are not authenticated, and whoever holds a token can write tags that form a cycle or a chain deeper than any sealed document. `JsonDocument::children` checks each entry at its own ordinal instead of looping over a fan-out the document chooses; `JsonDocument::subtree` and the SQL walks stop below 127 levels, and the Rust walk also stops once it has returned as many entries as the document holds; the rebuild of a subtree's rows refuses a repeated row and a nesting deeper than 127 levels as `OpenedType`. Each would otherwise have overflowed the stack, looped, or allocated what the document asked for.
+- *Nothing wipes the plaintext copies the walk makes* (the per-node scalars it clones before sealing), where the scalar paths hold theirs in `Zeroizing`. A `serde_json::Value` does not zeroize; a follow-up should walk the caller's value without copying scalars out of it.
+
 ## Decisions to take before Phase 1
 
 1. **Token encoding on the wire:** hex, matching v3's `s`, or base64url, saving about 40 characters per entry. Proposal: measure in Phase 3, default to base64url.

packages/eql/AGENTS.md

@@ -60,7 +60,7 @@ Where the two versions' SQL must differ, the source holds both forms in **versio
 
 #### Files confined to one EQL version
 
-A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Today only the generated range surface does: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7), and are `-- ONLY EQL VERSION: 4`. EQL v3 is what cipherstash-client writes, nothing there produces a range, and its SQL must not change. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the v4-only files; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
+A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Two surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `blockers.sql`): `public.eql_v4_json_tree`, `public.eql_v4_json_tree_entry`, `eql_v4.query_json_tree` and `eql_v4.query_json_tree_leaf`, the document Stack Encrypt's JSON index writes, a new family beside `public.eql_v4_json` and not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; the family is not in the catalog yet). Both are `-- ONLY EQL VERSION: 4`. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the v4-only files; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
 
 Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) reads each file's rendered copy from `build/render/eql_v3/`. A new tool that parses `src/v3` must do the same, and depend on `build`; do not substitute the placeholders a second way.