From 01c193b38b90c94555dd50c1ceed2737480cc858 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 21 Oct 2026 13:19:37 +1100 Subject: [PATCH 1/7] perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree The first measurements of the json_tree family (cipherstash/benches, branch feat/json-tree-benches) put it on par with the v3 SteVec on a flat corpus and one to four orders of magnitude behind native jsonb on a nested one: containment 70-80 ms a query, a 244-entry subtree 10 s, the GIN index 44 s to build on 10,000 rows. Three mechanisms accounted for all of it. The containment projection was rebuilt from the group tags by a recursive plpgsql function on every GIN recheck and again in the Filter, about 3 ms per 220-entry document. The client already derives that shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` reads `doc -> 'q'`. It returns jsonb rather than the needle domain, since a domain cast runs the needle's recursive CHECK and the recheck would pay it per row. `q` is the selectors already in `sv`, regrouped; nothing new is revealed, and the field is required (the family is unreleased). The walk evaluated the parent tag inside a LATERAL scan of `sv`, once per (node, ordinal, entry): ~670k SHA-256 calls for the 244-entry subtree. It now builds one jsonb map of the chained entries keyed by `p` and looks each computed tag up, so a level costs its fan-out in hashes, which is the design's W x F as written. 10 s becomes ~50 ms; children and child_count 7 ms become under 3. And with the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where the walk returns ~440 characters per entry and pays the fan-out per node. So `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, past it, return the root entry alone with `d = 0`; the client opens it whole and takes the walked path over it (`JsonSubtree::at`, `JsonTreeSubtree::new(rows).at`, over the new `JsonPath::select`). Budget 0 never falls back, a token that names nothing still yields nothing, an unsealed root is always walked out. Also `json_tree_path_query_first` is costed at 1000: under a prepared statement's generic plan the default cost chose a sequential scan with the extraction as a Filter over the functional btree (17 ms against 0.12 ms). Measured in the benches repository; the Postgres oracle (`mise run test:encryption:postgres`) covers the budget, the fallback's opening at the path, and containment against `jsonb @>` with the stored projection. The design document's implementation notes record Phase 3. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql-json-tree-walk-and-projection.md | 13 + ...26-10-17-json-index-stack-encrypt-spike.md | 14 +- .../crates/eql-bindings/src/v4/json_tree.rs | 82 ++++- packages/eql/src/v3/json_tree/functions.sql | 316 +++++++++++------- packages/eql/src/v3/json_tree/types.sql | 86 ++--- .../eql/tests/encryption/tests/json_tree.rs | 70 +++- packages/stack-encrypt/src/json/derive.rs | 53 ++- packages/stack-encrypt/src/json/mod.rs | 50 ++- packages/stack-encrypt/src/json/path.rs | 55 +++ packages/stack-encrypt/src/json/query.rs | 15 +- packages/stack-encrypt/tests/json_index.rs | 108 +++++- 11 files changed, 685 insertions(+), 177 deletions(-) create mode 100644 .changeset/eql-json-tree-walk-and-projection.md diff --git a/.changeset/eql-json-tree-walk-and-projection.md b/.changeset/eql-json-tree-walk-and-projection.md new file mode 100644 index 000000000..819c573db --- /dev/null +++ b/.changeset/eql-json-tree-walk-and-projection.md @@ -0,0 +1,13 @@ +--- +"@cipherstash/eql": minor +--- + +**The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: + +- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. +- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same 244 entries return in about 50 ms; `children` and `child_count` in under 3 ms. +- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it (`JsonSubtree::at(path)` in `stack-encrypt`, `JsonTreeSubtree::new(rows).at(path)` in `eql-bindings`). A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. + +Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (17 ms down to 0.1 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed. + +The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows` and an optional `path`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. diff --git a/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md b/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md index 37b85832b..adfd3f4b2 100644 --- a/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md +++ b/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md @@ -204,7 +204,8 @@ A document in the new domain, `public.eql_v4_json_tree` (placeholder name; a sib { "s": "…", "vs": "…", "c": "…" }, // $.list (array) a chain head { "s": "…", "vs": "…", "p": "…", "w": "…", "n": "…", "c": "…" }, // $.list[*] element 0 (object) { "s": "…", "vs": "…", "p": "…", "w": "…", "g": "…", "c": "…", "op": "…" } // $.list[*].x = 1, inside element 0 - ] + ], + "q": { "sv": [ {"s": "…", "vs": "…"}, … ], "el": [ { "sv": [ … ], "el": [ … ] } ] } // the containment projection, written by the client (Phase 3) } ``` @@ -350,6 +351,17 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi - *The data-plan form and the catalog are not done.* The Go guest, `stashgen`, the target table and `v4::all()` do not know the family; `IndexSpec::Json` crosses the wire and is refused with a reason. The family is hand-written in `eql-bindings` as the SteVec structs are. Both are the next bindings issue. - *Nothing wipes the plaintext copies the walk makes* (the per-node scalars it clones before sealing), where the scalar paths hold theirs in `Zeroizing`. A `serde_json::Value` does not zeroize; a follow-up should walk the caller's value without copying scalars out of it. +## Implementation notes (2026-10-21, Phase 3: first measurements and the fixes they forced) + +The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: + +- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. Nothing new is revealed: `q` is the selectors already in `sv`, regrouped, at about seven per cent of the document. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. +- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The 244 entries return in about 50 ms (3.7 ms for a six-entry subtree); `children` and `child_count` in under 3 ms, from 7. +- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it: `JsonSubtree::at(path)` in the engine, `JsonTreeSubtree::new(rows).at(path)` in the bindings, over the new `JsonPath::select`. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. The 244-entry subtree returns the root in about 8 ms. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. +- **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 17 ms against 0.12 ms with the index. + +Still open from the measurements, filed as a follow-up: `->` scans `sv` on every extraction (0.8 ms on a nested document; an index of `sv` by `s` would make it a lookup); `json_tree_contains` plans a second GIN probe for `alt` even when the needle carries none; `ORDER BY ord_term(col -> sel)` measured 0.92 ms against v3's 0.25 ms on an index scan both ways, unexplained; and the nested document is 24 times its plaintext (hex tags, a random nonce per entry, the sealed-root copy), which decisions 1 and 6 above and the opaque-subtree policy are the levers on. + ## Decisions to take before Phase 1 1. **Token encoding on the wire:** hex, matching v3's `s`, or base64url, saving about 40 characters per entry. Proposal: measure in Phase 3, default to base64url. diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index 371b01844..0262b5f51 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -22,7 +22,9 @@ use schemars::{schema_for, JsonSchema, Schema}; use serde::{Deserialize, Serialize}; use serde_json::Value; -use stack_encrypt::json::{JsonDocument, JsonLeafQuery, JsonNode, JsonQuery, JsonSubtree}; +use stack_encrypt::json::{ + JsonDocument, JsonLeafQuery, JsonNode, JsonPath, JsonQuery, JsonSubtree, +}; use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec}; use stack_encrypt::{ DecryptField, DecryptInto, Decryptable, Decryption, EncryptFrom, Encryption, Error, NonEmpty, @@ -40,9 +42,12 @@ use crate::Identifier; pub struct JsonTreeTag(pub String); /// `public.eql_v4_json_tree`: a JSON tree document, `{v, i, h, sv: -/// [entry]}`. Strict. `h` is the document's key header, stored once (every -/// entry is sealed under the document's one data key); `sv` holds one -/// entry per node in depth-first order, the root first. +/// [entry], q: group}`. Strict. `h` is the document's key header, stored +/// once (every entry is sealed under the document's one data key); `sv` +/// holds one entry per node in depth-first order, the root first; `q` is +/// the document's containment projection, its `{s, vs}` pairs nested by +/// array element, which the client writes beside the entries so the +/// server's `eql_v4.json_tree_query` reads it rather than rebuilding it. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeDocument { @@ -50,6 +55,7 @@ pub struct JsonTreeDocument { pub i: Identifier, pub h: KeyHeader, pub sv: Vec, + pub q: JsonTreeQueryGroup, } /// `public.eql_v4_json_tree_entry`: one node. `s` and `vs` are the path @@ -144,9 +150,34 @@ pub struct JsonTreeLeafQuery { /// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or /// `json_tree_subtree_at`), collected: opens to the value under the /// walked node, or to the list of complete subtrees the rows hold. +/// +/// Past its budget a walk returns the sealed root alone, at depth `0`; +/// those rows open to the whole document and the path the walk was keyed +/// from, attached with [`at`](Self::at), selects the value under it. The +/// path is not part of the wire. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(transparent)] -pub struct JsonTreeSubtree(pub Vec); +pub struct JsonTreeSubtree { + pub rows: Vec, + #[serde(skip)] + pub path: Option, +} + +impl JsonTreeSubtree { + /// The rows, as returned. + pub fn new(rows: Vec) -> Self { + Self { rows, path: None } + } + + /// The rows with the walked path, for the sealed root a walk may fall + /// back to. + pub fn at(self, path: JsonPath) -> Self { + Self { + path: Some(path), + ..self + } + } +} macro_rules! json_tree_domain_type { ($ty:ident, $sql:literal) => { @@ -335,12 +366,16 @@ impl DecryptInto> for JsonTreeSubtree { impl JsonTreeSubtree { fn subtree(&self) -> Result { - Ok(JsonSubtree::new( - self.0 + let rows = JsonSubtree::new( + self.rows .iter() .map(JsonTreeEntry::node) .collect::>()?, - )) + ); + Ok(match &self.path { + Some(path) => rows.at(path.clone()), + None => rows, + }) } } @@ -382,6 +417,37 @@ mod tests { .is_err()); } + #[test] + fn a_document_carries_its_projection() { + let document = serde_json::json!({ + "v": 4, "i": {"t": "users", "c": "profile"}, "h": "AA==", + "sv": [{"s": "aa", "vs": "bb", "c": "AA=="}], + "q": {"sv": [{"s": "aa", "vs": "bb"}], "el": []}, + }); + let stored: JsonTreeDocument = serde_json::from_value(document.clone()).unwrap(); + assert_eq!(stored.q.sv.len(), 1); + assert_eq!(serde_json::to_value(&stored).unwrap(), document); + let mut without = document.clone(); + let _ = without.as_object_mut().unwrap().remove("q"); + assert!( + serde_json::from_value::(without).is_err(), + "a document without `q` is not a stored document" + ); + } + + #[test] + fn a_subtree_carries_its_path_beside_the_wire() { + let rows: JsonTreeSubtree = serde_json::from_value(serde_json::json!([])).unwrap(); + assert!(rows.path.is_none()); + let at = rows.at(JsonPath::parse("$.a").unwrap()); + assert_eq!( + at.path.as_ref().map(ToString::to_string).as_deref(), + Some("$.a") + ); + assert_eq!(serde_json::to_value(&at).unwrap(), serde_json::json!([])); + assert!(at.subtree().unwrap().path().is_some()); + } + #[test] fn an_entry_without_the_grafted_fields_does_not_open() { let entry: JsonTreeEntry = serde_json::from_value(serde_json::json!({ diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index 042e9b498..0a96d8516 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -123,6 +123,40 @@ AS $$ FROM jsonb_array_elements(val -> 'sv') AS elem $$; +--! @brief Whether the document's root is sealed: its header's sealed-root +--! flag, bit 0 of the flags byte at offset 1 of the frozen layout. +--! A sealed root's `c` is the whole document, so a walk may return +--! the root in place of a subtree it would be cheaper to open whole. +--! @internal +--! @param h text The document's header, base64. +--! @return boolean True when the root is sealed. +CREATE FUNCTION {{prefix}}_internal.json_tree_sealed_root(h text) + RETURNS boolean + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT (get_byte(decode(h, 'base64'), 1) & 1) = 1 +$$; + +--! @brief The chained entries of a document keyed by parent tag: one object +--! whose keys are each entry's `p` and whose values are the entry +--! with its ordinal `j` in `sv` added. A walk looks a child up by +--! the tag it computes, `map -> tag`, instead of scanning `sv` for +--! it, so the document is read once per walk and a level costs its +--! fan-out in hashes, not its fan-out times the document's size. +--! @internal +--! @param val jsonb The document. +--! @return jsonb The map; `{}` for a document with no chained entry. +CREATE FUNCTION {{prefix}}_internal.json_tree_by_parent(val jsonb) + RETURNS jsonb + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT COALESCE( + jsonb_object_agg(elem ->> 'p', elem || jsonb_build_object('j', ord - 1)), + '{}'::jsonb) + FROM jsonb_array_elements(val -> 'sv') WITH ORDINALITY AS t(elem, ord) + WHERE elem ? 'p' +$$; + ------------------------------------------------------------------------------ -- Envelope helpers ------------------------------------------------------------------------------ @@ -183,7 +217,15 @@ $$; COMMENT ON FUNCTION {{prefix}}.json_tree_path_query(jsonb, text) IS 'eql-inline-critical: raw-jsonb path query helper; must stay inlinable (unpinned search_path)'; ---! @brief The first entry whose path selector is `selector`, or NULL. +--! @brief The first entry whose path selector is `selector`, or NULL: what +--! `->` returns, and what the ordering comparisons extract. +--! +--! Costed at 1000 (ten times the default): the body scans `sv`, and its +--! cost is what decides whether a comparison on it, `(col -> sel) > $1`, +--! takes the functional btree on `{{prefix}}.ord_term(col -> sel)` or filters +--! a sequential scan. Under a prepared statement's generic plan, where the +--! operand's selectivity is a guess, the default cost chose the scan. +--! --! @param val jsonb The document. --! @param selector text The path selector, hex. --! @return public.{{prefix}}_json_tree_entry The entry, or NULL when the document @@ -191,6 +233,7 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_query(jsonb, text) IS CREATE FUNCTION {{prefix}}.json_tree_path_query_first(val jsonb, selector text) RETURNS public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + COST 1000 AS $$ SELECT {{prefix}}_internal.json_tree_graft(val, elem, (ord - 1)::integer) FROM jsonb_array_elements(val -> 'sv') WITH ORDINALITY AS t(elem, ord) @@ -223,67 +266,46 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_exists(jsonb, text) IS -- Containment: the element-nested projection ------------------------------------------------------------------------------ ---! @brief One group of a stored document's projection, from its group tags. ---! @internal +--! @brief The stored document's containment projection: the needle shape +--! `{{prefix}}.query_json_tree` holds, as the client wrote it in `q`. --! ---! An entry belongs to the group of its own tag `n` when it is an array ---! element that is a container, else to the group of its enclosing element's ---! tag `g`, else to the top level. A group's `el` holds one nested group per ---! container element whose `g` is the group's tag. The needle is projected ---! the same way by the client, from its structure, so `jsonb @>` over the two ---! is PostgreSQL's own containment rule: each needle element has to fit ---! inside one stored element. ---! ---! @param sv jsonb The document's `sv` array. ---! @param parent text The group's tag, hex, or NULL for the top level. ---! @return jsonb `{"sv": [{"s", "vs"}, …], "el": [group, …]}`. ---! @note plpgsql: recursive. IMMUTABLE and deterministic (document order), so ---! `{{prefix}}.json_tree_query` over it serves an expression index. -CREATE FUNCTION {{prefix}}_internal.json_tree_group(sv jsonb, parent text) - RETURNS jsonb - LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE -AS $$ -DECLARE - pairs jsonb; - groups jsonb; -BEGIN - SELECT COALESCE(jsonb_agg(jsonb_build_object('s', elem -> 's', 'vs', elem -> 'vs') ORDER BY ord), '[]'::jsonb) - INTO pairs - FROM jsonb_array_elements(sv) WITH ORDINALITY AS t(elem, ord) - WHERE COALESCE(elem ->> 'n', elem ->> 'g') IS NOT DISTINCT FROM parent; - SELECT COALESCE(jsonb_agg({{prefix}}_internal.json_tree_group(sv, elem ->> 'n') ORDER BY ord), '[]'::jsonb) - INTO groups - FROM jsonb_array_elements(sv) WITH ORDINALITY AS t(elem, ord) - WHERE elem ? 'n' AND (elem ->> 'g') IS NOT DISTINCT FROM parent; - RETURN jsonb_build_object('sv', pairs, 'el', groups); -END; -$$; - ---! @brief Project a stored document into its containment shape: the needle ---! shape `{{prefix}}.query_json_tree` holds, from the document's group ---! tags. +--! The projection is the document's `{s, vs}` pairs nested by array +--! element, `{"sv": [{"s", "vs"}, …], "el": [group, …]}`: the pairs outside +--! any array element at the top, and one nested group per array element +--! that is a container, so that `jsonb @>` over a stored projection and a +--! needle is PostgreSQL's own containment rule, each needle element inside +--! one stored element. The client derives it from the document it seals +--! and stores it beside `sv`; the server reads it and never rebuilds it, +--! so a containment recheck costs a field read, not a pass over `sv`. --! --! This is the GIN index expression for containment: ---! `CREATE INDEX … USING gin (({{prefix}}.json_tree_query(col)::jsonb) jsonb_path_ops);` +--! `CREATE INDEX … USING gin (({{prefix}}.json_tree_query(col)) jsonb_path_ops);` --! The typed `@>` inlines to native `jsonb @>` over the same expression, so --! the index engages; GIN is lossy and the executor rechecks the real --! operator on the heap tuple, which is the exact containment. --! --! @param e public.{{prefix}}_json_tree The stored document. ---! @return {{prefix}}.query_json_tree Its projection. +--! @return jsonb Its projection, the shape of `{{prefix}}.query_json_tree`. +--! @note Returns raw jsonb rather than the needle domain: a domain cast +--! runs the needle's recursive CHECK, and the recheck would pay it +--! once per candidate row. The document's own CHECK validated `q` +--! when the row was written. CREATE FUNCTION {{prefix}}.json_tree_query(e public.{{prefix}}_json_tree) - RETURNS {{prefix}}.query_json_tree + RETURNS jsonb LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}_internal.json_tree_group(e::jsonb -> 'sv', NULL)::{{prefix}}.query_json_tree + SELECT e::jsonb -> 'q' $$; +COMMENT ON FUNCTION {{prefix}}.json_tree_query(public.{{prefix}}_json_tree) IS + 'eql-inline-critical: the containment projection the GIN index is built over; must stay inlinable (unpinned search_path)'; + --! @brief Does the stored document `a` contain the needle `b`? --! --! Native `jsonb @>` over the projections, plus the needle's `alt` projection --! when it carries one (a root scalar against a stored root array). Both ---! probes are over `{{prefix}}.json_tree_query(a)::jsonb`, so a GIN index on ---! that expression serves them as one bitmap OR. +--! probes are over `{{prefix}}.json_tree_query(a)`, so a GIN index on that +--! expression serves them as one bitmap OR. --! --! @param a public.{{prefix}}_json_tree The stored document. --! @param b {{prefix}}.query_json_tree The needle. @@ -299,8 +321,8 @@ CREATE FUNCTION {{prefix}}.json_tree_contains(a public.{{prefix}}_json_tree, b { RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ - SELECT {{prefix}}.json_tree_query(a)::jsonb @> (b::jsonb - 'alt') - OR ((b::jsonb ? 'alt') AND {{prefix}}.json_tree_query(a)::jsonb @> (b::jsonb -> 'alt')) + SELECT {{prefix}}.json_tree_query(a) @> (b::jsonb - 'alt') + OR ((b::jsonb ? 'alt') AND {{prefix}}.json_tree_query(a) @> (b::jsonb -> 'alt')) $$; --! @brief Is the stored document `a` contained by the needle `b`? @@ -316,7 +338,7 @@ CREATE FUNCTION {{prefix}}.json_tree_contained_by(a public.{{prefix}}_json_tree, RETURNS boolean LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT (b::jsonb - 'alt') @> {{prefix}}.json_tree_query(a)::jsonb + SELECT (b::jsonb - 'alt') @> {{prefix}}.json_tree_query(a) $$; --! @brief Does the stored document `a` contain the stored document `b`? @@ -327,20 +349,27 @@ CREATE FUNCTION {{prefix}}.json_tree_document_contains(a public.{{prefix}}_json_ RETURNS boolean LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}.json_tree_query(a)::jsonb @> {{prefix}}.json_tree_query(b)::jsonb + SELECT {{prefix}}.json_tree_query(a) @> {{prefix}}.json_tree_query(b) $$; ------------------------------------------------------------------------------ -- The token chain: children and subtrees by token ------------------------------------------------------------------------------ +--! The walk, in every function below: for the node in hand, compute the +--! parent tag a child at each ordinal under the document's fan-out would +--! carry, and look each tag up in the document's parent map +--! ({{prefix}}_internal.json_tree_by_parent). A hit is a child; its token +--! unwraps under the parent's and the walk recurses. The map and the +--! document's header fields are computed once per call, so a level costs +--! its fan-out in SHA-256 evaluations and one lookup each, and the +--! document is never scanned per node. + --! @brief The children of the node whose token is `token`, as rows. --! ---! For every ordinal under the document's fan-out, the parent tag the child ---! at that ordinal would carry; the entries carrying one are the children, ---! and each row's token is unwrapped on the way, though not returned. What ---! `jsonb_array_elements` is for an encrypted array: one row per element, in ---! ordinal order, each opening on its own. +--! What `jsonb_array_elements` is for an encrypted array: one row per +--! element, in ordinal order, each opening on its own. Each child's token +--! is unwrapped on the way, though not returned. --! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The parent's token, hex: derived by the client from the @@ -350,14 +379,20 @@ CREATE FUNCTION {{prefix}}.json_tree_children(e public.{{prefix}}_json_tree, tok RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT ({{prefix}}_internal.json_tree_graft(e::jsonb, c.elem, (c.ord - 1)::integer)::jsonb || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry - FROM generate_series(0, {{prefix}}_internal.json_tree_fanout(e::jsonb) - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(e::jsonb -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag( - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h'), decode(token, 'hex'), o) - ) AS c ON true + WITH + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta + ) + SELECT (doc.meta || c.entry || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry + FROM doc + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, decode(token, 'hex'), o) AS entry + ) AS c + WHERE c.entry IS NOT NULL ORDER BY o $$; @@ -384,44 +419,77 @@ $$; --! and the client reads the node's kind from its children's labels, or --! fetches the node by its selector. --! +--! **The budget.** A walk costs the fan-out in hashes per node it visits, +--! and returns about four hundred characters per entry, where a sealed root +--! holds the whole document in one ciphertext of about the plaintext's +--! size. So when the root is sealed and the subtree has more than `budget` +--! entries, the walk stops and returns the root entry alone, with `d = 0`: +--! the client opens it to the whole document and takes the path itself, +--! which is cheaper on both sides than the walk past that point. The +--! default of 64 is past the crossover by bytes (about a dozen entries) by +--! a margin that keeps a small subtree arriving as the entries it is. A +--! `budget` of `0` never falls back. A token that names no node yields no +--! row under any budget, and a document whose root is not sealed is always +--! walked in full, since its root holds no document to return. +--! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The subtree's token, hex. +--! @param budget integer The most entries to return before falling back to +--! the sealed root; `0` to always walk the subtree out. Default 64. --! @return SETOF public.{{prefix}}_json_tree_entry The descendants, each with its ---! depth `d` below the node, in depth-first order. -CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text) +--! depth `d` below the node, in depth-first order; or the root +--! entry alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ WITH RECURSIVE - doc AS ( - SELECT e::jsonb AS val, - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta, + e::jsonb -> 'sv' -> 0 AS root, + budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), - walk(depth, t, j, entry, path) AS ( - SELECT 0, decode(token, 'hex'), NULL::integer, NULL::jsonb, ARRAY[]::integer[] + walk(depth, t, entry) AS ( + SELECT 0, decode(token, 'hex'), NULL::jsonb UNION ALL SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.elem), - (c.ord - 1)::integer, - c.elem, - w.path || (c.ord - 1)::integer + {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), + c.entry FROM walk AS w CROSS JOIN doc - JOIN LATERAL ( - SELECT elem, ord - FROM generate_series(0, doc.fanout - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(doc.val -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) - ) AS hit ON true - ) AS c ON true + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry + ) AS c + WHERE c.entry IS NOT NULL + ), + -- One row past the budget is enough to know the walk went over; a walk + -- under it is the subtree complete. Uncapped, the limit is NULL: all. + found AS MATERIALIZED ( + SELECT w.depth, w.entry + FROM walk AS w + WHERE w.entry IS NOT NULL + LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) + ), + over AS MATERIALIZED ( + SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc ) - SELECT ({{prefix}}_internal.json_tree_graft(doc.val, w.entry, w.j)::jsonb || jsonb_build_object('d', w.depth))::public.{{prefix}}_json_tree_entry - FROM walk AS w CROSS JOIN doc - WHERE w.entry IS NOT NULL - ORDER BY w.path + SELECT rows.entry + FROM ( + SELECT 0 AS j, + (doc.meta || doc.root || '{"j": 0, "d": 0}'::jsonb)::public.{{prefix}}_json_tree_entry AS entry + FROM doc, over + WHERE over.fallback + UNION ALL + SELECT (f.entry ->> 'j')::integer, + (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry + FROM found AS f, doc, over + WHERE NOT over.fallback + ) AS rows + ORDER BY rows.j $$; --! @brief The subtrees under the node whose token is `token` whose path @@ -431,48 +499,68 @@ $$; --! The walk starts at the token and emits an entry once it has passed --! through, or reached, an entry carrying the selector: for --! `$.orders[*].items` it starts at `$.orders` and returns every element's ---! `items` subtree, each complete. +--! `items` subtree, each complete. The budget is `json_tree_subtree`'s, +--! counted over the entries emitted: past it, the sealed root alone, with +--! `d = 0`, and the client takes the wildcard path over the whole document. --! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The anchor's token, hex. --! @param selector text The whole path's selector, hex. +--! @param budget integer The most entries to return before falling back to +--! the sealed root; `0` to always walk out. Default 64. --! @return SETOF public.{{prefix}}_json_tree_entry The matching subtrees' entries, ---! each with its depth `d` below the anchor, in depth-first order. -CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, selector text) +--! each with its depth `d` below the anchor, in depth-first order; +--! or the root entry alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, selector text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ WITH RECURSIVE - doc AS ( - SELECT e::jsonb AS val, - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta, + e::jsonb -> 'sv' -> 0 AS root, + budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), - walk(depth, t, j, entry, path, hit) AS ( - SELECT 0, decode(token, 'hex'), NULL::integer, NULL::jsonb, ARRAY[]::integer[], false + walk(depth, t, entry, hit) AS ( + SELECT 0, decode(token, 'hex'), NULL::jsonb, false UNION ALL SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.elem), - (c.ord - 1)::integer, - c.elem, - w.path || (c.ord - 1)::integer, - w.hit OR c.elem ->> 's' = selector + {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), + c.entry, + w.hit OR c.entry ->> 's' = selector FROM walk AS w CROSS JOIN doc - JOIN LATERAL ( - SELECT elem, ord - FROM generate_series(0, doc.fanout - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(doc.val -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) - ) AS hit ON true - ) AS c ON true + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry + ) AS c + WHERE c.entry IS NOT NULL + ), + found AS MATERIALIZED ( + SELECT w.depth, w.entry + FROM walk AS w + WHERE w.entry IS NOT NULL AND w.hit + LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) + ), + over AS MATERIALIZED ( + SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc ) - SELECT ({{prefix}}_internal.json_tree_graft(doc.val, w.entry, w.j)::jsonb || jsonb_build_object('d', w.depth))::public.{{prefix}}_json_tree_entry - FROM walk AS w CROSS JOIN doc - WHERE w.entry IS NOT NULL AND w.hit - ORDER BY w.path + SELECT rows.entry + FROM ( + SELECT 0 AS j, + (doc.meta || doc.root || '{"j": 0, "d": 0}'::jsonb)::public.{{prefix}}_json_tree_entry AS entry + FROM doc, over + WHERE over.fallback + UNION ALL + SELECT (f.entry ->> 'j')::integer, + (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry + FROM found AS f, doc, over + WHERE NOT over.fallback + ) AS rows + ORDER BY rows.j $$; ------------------------------------------------------------------------------ diff --git a/packages/eql/src/v3/json_tree/types.sql b/packages/eql/src/v3/json_tree/types.sql index da4b59076..d6647551b 100644 --- a/packages/eql/src/v3/json_tree/types.sql +++ b/packages/eql/src/v3/json_tree/types.sql @@ -10,7 +10,7 @@ --! Stack Encrypt's JSON index writes. It shares no wire shape and no SQL with --! the SteVec surface. Four jsonb-backed domains (none over another domain, --! so the native-jsonb firewall in blockers.sql attaches): ---! - public.{{prefix}}_json_tree — storage: an EQL envelope `{v, i, h, sv}`. +--! - public.{{prefix}}_json_tree — storage: an EQL envelope `{v, i, h, sv, q}`. --! - public.{{prefix}}_json_tree_entry — one `sv` element (returned by `->` --! and the subtree walk). --! - {{prefix}}.query_json_tree — a containment needle: the element-nested @@ -34,8 +34,13 @@ --! `g` the group tag of the nearest enclosing such element. --! `op` the leaf ordering term (CLLW OPE, hex) on number and string leaves. --! The header `h` (base64) carries the document's data key material once; the ---! SQL reads only the ZeroKMS `iv` at byte offset 22 of it, the per-document ---! salt of the chain tags, and otherwise carries it opaquely. +--! SQL reads only the sealed-root flag at byte offset 1 and the ZeroKMS `iv` +--! at byte offset 22 of it, the per-document salt of the chain tags, and +--! otherwise carries it opaquely. `q` is the document's containment +--! projection, `{"sv": [{s, vs}, …], "el": [group, …]}`: its `{s, vs}` +--! pairs nested by array element, written by the client beside `sv` so the +--! server compares it with a needle by native `jsonb @>` and never +--! rebuilds it from the entries. --! @brief Validate one JSON tree entry. --! @internal @@ -68,42 +73,15 @@ AS $$ ) $$; ---! @brief Validate a JSON tree document. ---! @internal ---! @param val jsonb Candidate document. ---! @return boolean True when `val` is an envelope with `v = {{eql_version}}`, `i`, a ---! string header `h` and a non-empty `sv` array of valid entries, and ---! no other key. A document always has a root, so `sv` is never empty. -CREATE OR REPLACE FUNCTION public.{{prefix}}_is_valid_json_tree_document_payload(val jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT COALESCE( - jsonb_typeof(val) = 'object' - AND val ? 'v' - AND val ->> 'v' = '{{eql_version}}' - AND val ? 'i' - AND jsonb_typeof(val -> 'h') = 'string' - AND jsonb_typeof(val -> 'sv') = 'array' - AND jsonb_array_length(val -> 'sv') >= 1 - AND val - ARRAY['v', 'i', 'h', 'sv']::text[] = '{}'::jsonb - AND NOT EXISTS ( - SELECT 1 - FROM jsonb_array_elements(val -> 'sv') AS elem - WHERE NOT public.{{prefix}}_is_valid_json_tree_entry_payload(elem) - ), - false - ) -$$; - --! @brief Validate one group of a containment needle, recursively. --! @internal --! --! A group is `{"sv": [{"s": "…", "vs": "…"}, …], "el": [group, …]}`: the --! selector pairs that sit directly in it, and one nested group per array ---! element that is a container. The same shape `{{prefix}}.json_tree_query` ---! projects a stored document into, so native `jsonb @>` over the two is ---! exact containment. +--! element that is a container. The same shape a stored document carries +--! in `q`, its own projection, so native `jsonb @>` over the two is exact +--! containment. Validates both: a needle at its cast, a document's `q` at +--! its write. --! --! @param val jsonb Candidate group. --! @return boolean True when `val` is a group. @@ -138,6 +116,36 @@ BEGIN END; $$; +--! @brief Validate a JSON tree document. +--! @internal +--! @param val jsonb Candidate document. +--! @return boolean True when `val` is an envelope with `v = {{eql_version}}`, `i`, a +--! string header `h`, a non-empty `sv` array of valid entries and +--! the document's containment projection `q`, a group, and no +--! other key. A document always has a root, so `sv` is never empty. +CREATE OR REPLACE FUNCTION public.{{prefix}}_is_valid_json_tree_document_payload(val jsonb) + RETURNS boolean + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT COALESCE( + jsonb_typeof(val) = 'object' + AND val ? 'v' + AND val ->> 'v' = '{{eql_version}}' + AND val ? 'i' + AND jsonb_typeof(val -> 'h') = 'string' + AND jsonb_typeof(val -> 'sv') = 'array' + AND jsonb_array_length(val -> 'sv') >= 1 + AND val - ARRAY['v', 'i', 'h', 'sv', 'q']::text[] = '{}'::jsonb + AND NOT EXISTS ( + SELECT 1 + FROM jsonb_array_elements(val -> 'sv') AS elem + WHERE NOT public.{{prefix}}_is_valid_json_tree_entry_payload(elem) + ) + AND {{prefix}}_internal.json_tree_is_valid_query_group(val -> 'q'), + false + ) +$$; + --! @brief Validate a containment needle. --! @internal --! @param val jsonb Candidate needle. @@ -162,11 +170,12 @@ $$; --! @brief Storage domain for an encrypted JSON tree column. --! --! CHECK: the EQL envelope (`v = {{eql_version}}`, `i`, `h`) around a non-empty `sv` ---! array of valid entries. The firewall in blockers.sql attaches to this ---! domain to stop native jsonb operators from reaching a column value. +--! array of valid entries and the projection `q`. The firewall in +--! blockers.sql attaches to this domain to stop native jsonb operators from +--! reaching a column value. --! --! @note Constructing from inline JSON uses the standard DOMAIN cast: ---! `'{"v":{{eql_version}},"i":{…},"h":"…","sv":[…]}'::public.{{prefix}}_json_tree`. +--! `'{"v":{{eql_version}},"i":{…},"h":"…","sv":[…],"q":{…}}'::public.{{prefix}}_json_tree`. DO $$ BEGIN IF NOT EXISTS ( @@ -212,7 +221,8 @@ $$; --! --! The element-nested projection of the needle's selectors, as Stack --! Encrypt's `JsonQuery` serializes it: `{"sv": [{"s", "vs"}, …], "el": ---! [group, …], "alt": group?}`. Typing the needle this way stops raw jsonb +--! [group, …], "alt": group?}`: the shape a stored document's `q` has, with +--! the optional `alt` beside it. Typing the needle this way stops raw jsonb --! from casting and matching every row via bare `jsonb @>`. --! --! @note Construct from inline JSON via the DOMAIN cast: diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index 8efd1016b..07aa9e75d 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -306,7 +306,7 @@ fn postgres_containment_over_the_encrypted_column_is_jsonb_containment() { let mut db = connect(); db.batch_execute( "CREATE TEMP TABLE json_tree_docs (id int PRIMARY KEY, plain jsonb NOT NULL, doc public.eql_v4_json_tree NOT NULL); - CREATE INDEX json_tree_docs_doc ON json_tree_docs USING gin ((eql_v4.json_tree_query(doc)::jsonb) jsonb_path_ops);", + CREATE INDEX json_tree_docs_doc ON json_tree_docs USING gin ((eql_v4.json_tree_query(doc)) jsonb_path_ops);", ) .unwrap(); @@ -541,7 +541,7 @@ fn postgres_extracts_walks_and_orders_by_path() { .all(|e| e.d.is_some() && e.j.is_some() && e.h.is_some())); let value: Value = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), ) .unwrap(); assert_eq!(value, expected, "row {id}"); @@ -555,6 +555,48 @@ fn postgres_extracts_walks_and_orders_by_path() { .get(0); assert_eq!(garbage, 0, "a wrong token unlocks nothing in any row"); + // The budget: past it the walk returns the sealed root alone, at depth + // 0, and the rows open to the value at the walked path; at it, or + // with the budget off, the walk is complete; a wrong token still + // yields nothing under any budget. + let walked = |db: &mut postgres::Client, budget: i32| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", + &[&nested.token(), &budget], + ) + .unwrap() + .iter() + .map(entry) + .collect() + }; + assert_eq!(walked(&mut db, 6).len(), 6, "at the budget, the subtree"); + assert_eq!(walked(&mut db, 0).len(), 6, "budget 0 never falls back"); + let root = walked(&mut db, 5); + assert_eq!(root.len(), 1, "past the budget, the root alone"); + assert_eq!((root[0].j, root[0].d), (Some(0), Some(0))); + let refused = wait( + &runtime, + cipher.decrypt_as::(JsonTreeSubtree::new(root.clone()), column()), + ); + assert!(refused.is_err(), "the fallback needs the walked path"); + let value: Value = wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(root).at(JsonPath::parse("$.nested").unwrap()), + column(), + ), + ) + .unwrap(); + assert_eq!(value, fixture()["nested"]); + let garbage: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, 1) e", + &[&"00".repeat(16)], + ) + .unwrap() + .get(0); + assert_eq!(garbage, 0); + // A wildcard below the anchor: every element's `x`, each a complete // subtree, from the list's token gated on the whole path's selector. let xs = subtree_key("$.nested.list[*].x"); @@ -569,7 +611,29 @@ fn postgres_extracts_walks_and_orders_by_path() { .collect(); let values: Vec = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), + ) + .unwrap(); + assert_eq!(values, vec![json!(1), json!(2)]); + // Past the budget, the gated walk falls back the same way, and the + // wildcard path selects every subtree over the whole document. + let root: Vec = db + .query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text, 1) e WHERE id = 1", + &[&xs.token(), &xs.selector().unwrap().to_string()], + ) + .unwrap() + .iter() + .map(entry) + .collect(); + assert_eq!(root.len(), 1); + assert_eq!(root[0].d, Some(0)); + let values: Vec = wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(root).at(JsonPath::parse("$.nested.list[*].x").unwrap()), + column(), + ), ) .unwrap(); assert_eq!(values, vec![json!(1), json!(2)]); diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 6564cd84f..5f928ff0b 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -943,11 +943,17 @@ impl DecryptInto for JsonNode { /// value, in order. What `json_tree_subtree_at` returns is one subtree per /// node whose selector matched; what `json_tree_subtree` returns is the /// children of one node, which [`DecryptInto`] assembles into that -/// node's value. +/// node's value. The sealed root a walk fell back to opens whole, and the +/// path attached with [`JsonSubtree::at`] selects over it: one value per +/// node the path reaches. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { + if self.is_sealed_root() { + return open_fallback(self, cipher, context) + .map(|(path, whole)| path.select(&whole).into_iter().cloned().collect()); + } open_subtree(self, cipher, context) .map(|forest| forest.into_iter().map(|(_, v)| v).collect()) }) @@ -958,11 +964,24 @@ impl DecryptInto> for JsonSubtree { /// that token: an array when the rows are elements, an object when they are /// members. No rows is no value ([`Error::OpenedType`]): an empty container /// and a scalar both have no descendants, and `->` on the node's selector -/// returns the node itself. +/// returns the node itself. The sealed root a walk fell back to opens +/// whole, and the path attached with [`JsonSubtree::at`] selects the one +/// value under it; a path that reaches no node, or more than one, is +/// [`Error::OpenedType`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { Decryption::from_open(move |cipher| { + if self.is_sealed_root() { + return open_fallback(self, cipher, context).try_map(|(path, whole)| { + let mut found = path.select(&whole); + match found.len() { + 1 => Ok(found.remove(0).clone()), + 0 => Err(shape("a node at the walked path")), + _ => Err(shape("one node at the walked path")), + } + }); + } open_subtree(self, cipher, context).try_map(|forest| { let mut members = Map::new(); let mut elements: Vec<(u64, Value)> = Vec::new(); @@ -1025,6 +1044,36 @@ fn open_subtree<'a, K>( }) } +/// Open the sealed root a walk fell back to: the whole document, with the +/// path the rows were keyed from, which the caller selects over. Refused +/// without the path, since nothing in the rows records it. +fn open_fallback<'a, K>( + subtree: JsonSubtree, + cipher: &'a StackCipher, + context: AeadContext, +) -> Pending<'a, (JsonPath, Value), K> { + let Some(path) = subtree.path().cloned() else { + return Pending::failed( + cipher, + shape("the walked path, attached with JsonSubtree::at, to select over the sealed root the walk returned"), + ); + }; + let Some(root) = subtree.nodes().first().cloned() else { + return Pending::failed(cipher, shape("the sealed root")); + }; + let (request, aad) = match retrieval(&root.header, context) { + Ok(parts) => parts, + Err(error) => return Pending::failed(cipher, error), + }; + Pending::request(cipher, vec![request], move |responses| { + let key = responses.next_retrieved_key()?; + let aes = leaf_cipher(&key)?; + let opened = open_entry(&aes, &root.header, &aad, &root.entry, root.j)?; + let whole = opened.whole.ok_or_else(|| shape("a sealed root"))?; + Ok((path, whole)) + }) +} + // ============================================================================= // Extraction keys, on the keyset cipher // ============================================================================= diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index 3b90241ef..d47c7a3d5 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -333,8 +333,10 @@ impl JsonEntry { /// [`DecryptInto`](crate::DecryptInto) opens. /// /// Serializes to the wire form the EQL domain stores: `{"h": "…", "sv": -/// [{"s": "…", "vs": "…", "c": "…", …}, …]}`; see the -/// [module docs](self#byte-encodings). +/// [{"s": "…", "vs": "…", "c": "…", …}, …], "q": {"sv": [{"s", "vs"}, …], +/// "el": […]}}`, where `q` is the document's own containment +/// [projection](Self::projection), written once so the server never +/// rebuilds it; see the [module docs](self#byte-encodings). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(try_from = "DocumentWire", into = "DocumentWire")] pub struct JsonDocument { @@ -494,22 +496,57 @@ where /// members an object. A node with no descendants, an empty container or a /// scalar, yields no rows and so no value; `->` on its selector returns the /// node itself. +/// +/// Past its budget, a walk over a document with a sealed root returns the +/// root entry alone, at depth `0`, instead of a subtree it would be +/// cheaper to open whole: the client opens the document and takes the +/// walked path over it. Opening such rows needs that path, attached with +/// [`at`](Self::at); without it the opening fails with +/// [`Error::OpenedType`](crate::Error::OpenedType). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(transparent)] pub struct JsonSubtree { nodes: Vec, + #[serde(skip)] + path: Option, } impl JsonSubtree { /// The rows, as returned. pub fn new(nodes: Vec) -> Self { - Self { nodes } + Self { nodes, path: None } + } + + /// The rows with the path the walk was keyed from: what the opening + /// takes over the whole document when the rows are the sealed root + /// the server fell back to. A positional path opens to one value; a + /// path with a wildcard, as `json_tree_subtree_at` walks, to one value + /// per node it reaches. + pub fn at(self, path: JsonPath) -> Self { + Self { + path: Some(path), + ..self + } } /// The rows. pub fn nodes(&self) -> &[JsonNode] { &self.nodes } + + /// The path attached with [`at`](Self::at), if any. + pub fn path(&self) -> Option<&JsonPath> { + self.path.as_ref() + } + + /// Whether the rows are the sealed root the server returned in place + /// of a subtree past its budget: one row, the root, at depth `0`. + pub fn is_sealed_root(&self) -> bool { + matches!( + self.nodes.as_slice(), + [node] if node.j == 0 && node.depth == Some(0) && node.header.sealed_root() + ) + } } impl Decryptable for JsonSubtree { @@ -580,12 +617,16 @@ fn unbase64(text: &str) -> Result, WireError> { #[error("a JSON index wire field does not decode: {0}")] pub struct WireError(&'static str); -/// `h` and `sv`, as stored. +/// `h`, `sv` and `q`, as stored. `q` is written from the entries and not +/// read back: a document deserialized from the wire derives its projection +/// from `sv` again, so the field is accepted and ignored on the way in. #[derive(Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct DocumentWire { h: String, sv: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + q: Option, } /// One entry, as stored: tags in hex, the ciphertext in base64. @@ -648,6 +689,7 @@ impl From for DocumentWire { Self { h: base64(&doc.header.to_bytes()), sv: doc.entries.iter().map(EntryWire::from).collect(), + q: Some(query::GroupWire::from(&doc.projection())), } } } diff --git a/packages/stack-encrypt/src/json/path.rs b/packages/stack-encrypt/src/json/path.rs index 968747b71..87ce2c960 100644 --- a/packages/stack-encrypt/src/json/path.rs +++ b/packages/stack-encrypt/src/json/path.rs @@ -152,6 +152,32 @@ impl JsonPath { !self.segments.iter().any(|s| matches!(s, Segment::Wildcard)) } + /// The values at this path in `value`, in document order: at most one + /// for a positional path, one per element a wildcard reaches, none when + /// the path names nothing. A key step into an array, or a position + /// into an object, reaches nothing, as the token chain reaches nothing + /// there. What a client takes over a whole document when a walk + /// returned the sealed root in place of the subtree. + pub fn select<'a>(&self, value: &'a serde_json::Value) -> Vec<&'a serde_json::Value> { + let mut current = vec![value]; + for segment in &self.segments { + let mut next = Vec::new(); + for node in current { + match segment { + Segment::Key(key) => next.extend(node.as_object().and_then(|o| o.get(key))), + Segment::Index(index) => next.extend( + usize::try_from(*index) + .ok() + .and_then(|i| node.as_array().and_then(|a| a.get(i))), + ), + Segment::Wildcard => next.extend(node.as_array().into_iter().flatten()), + } + } + current = next; + } + current + } + /// This path with one more step. pub(crate) fn then(&self, segment: Segment) -> Self { let mut segments = Vec::with_capacity(self.segments.len() + 1); @@ -311,6 +337,35 @@ mod tests { assert!(JsonPath::parse("$.a[2]").unwrap().is_positional()); } + #[test] + fn selects_what_the_chain_would_reach() { + let doc = serde_json::json!({ + "a": [{"x": 1}, {"x": 2, "y": [3, 4]}, 5], + "b": {"c": "d"}, + }); + let at = |text: &str| JsonPath::parse(text).unwrap().select(&doc); + assert_eq!(at("$"), vec![&doc]); + assert_eq!(at("$.b.c"), vec![&serde_json::json!("d")]); + assert_eq!(at("$.a[1].x"), vec![&serde_json::json!(2)]); + assert_eq!( + at("$.a[*].x"), + vec![&serde_json::json!(1), &serde_json::json!(2)] + ); + assert_eq!( + at("$.a[*].y[*]"), + vec![&serde_json::json!(3), &serde_json::json!(4)] + ); + assert!(at("$.a.x").is_empty(), "a key step into an array"); + assert!(at("$.b[0]").is_empty(), "a position into an object"); + assert!(at("$.a[3]").is_empty(), "past the end"); + assert!(at("$.b[*]").is_empty(), "the wildcard over an object"); + assert!(at("$.z").is_empty()); + assert!(JsonPath::parse("$[18446744073709551615]") + .unwrap() + .select(&doc) + .is_empty()); + } + #[test] fn refuses_what_it_does_not_address() { for (text, at) in [ diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index 77fae1910..98b83de27 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -121,9 +121,11 @@ fn group_of(nodes: &[TermNode], parent: Option) -> JsonProjection { } impl JsonDocument { - /// The stored document's projection, from its group tags, as the SQL's - /// `json_tree_query` computes it: what a [`JsonQuery`] is matched - /// against. + /// The stored document's projection, from its group tags: what a + /// [`JsonQuery`] is matched against. The document writes it to the wire + /// as `q`, beside `sv`, and the SQL's `json_tree_query` reads it from + /// there; a document deserialized from the wire derives it again from + /// the entries, so a stored `q` is never trusted over them. pub fn projection(&self) -> JsonProjection { fn belongs(e: &JsonEntry) -> Option<&Tag> { e.n.as_ref().or(e.g.as_ref()) @@ -146,14 +148,15 @@ impl JsonDocument { } } -#[derive(Serialize, Deserialize)] +#[derive(Clone, Serialize, Deserialize)] struct PairWire { s: String, vs: String, } -#[derive(Serialize, Deserialize)] -struct GroupWire { +/// One group, as a needle or a stored document's `q` spells it. +#[derive(Clone, Serialize, Deserialize)] +pub(crate) struct GroupWire { sv: Vec, el: Vec, } diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index fc640e84c..f432b2314 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -116,7 +116,26 @@ async fn the_wire_form_is_what_the_sql_domain_stores() { .keys() .map(String::as_str) .collect(); - assert_eq!(keys, ["h", "sv"]); + assert_eq!(keys, ["h", "q", "sv"]); + // `q` is the document's own projection, written once: the needle a + // `Contains` of the whole document derives has the same pairs in the + // same groups, and nothing else. + let needle = cipher + .query(op::Contains, &fixture()) + .on(&plan()) + .await + .unwrap(); + assert_eq!(wire["q"], serde_json::to_value(&needle).unwrap()); + assert!( + serde_json::from_value::(json!({"h": wire["h"], "sv": wire["sv"]})).is_ok(), + "a document without `q` still deserializes: the projection is derived from the entries" + ); + let back: JsonDocument = serde_json::from_value(wire.clone()).unwrap(); + assert_eq!( + serde_json::to_value(&back).unwrap(), + wire, + "`q` round-trips through the entries" + ); for (j, entry) in wire["sv"].as_array().unwrap().iter().enumerate() { let entry = entry.as_object().unwrap(); for key in entry.keys() { @@ -997,6 +1016,93 @@ async fn each_stored_form_opens_as_a_record_field() { assert_eq!(opened, json!([{"x": 1}, {"x": 2}])); } +/// Past its budget the server returns the sealed root alone, at depth 0, +/// in place of the subtree: the rows open to the whole document and the +/// path attached with `at` selects over it. Without the path, or with one +/// that reaches nothing, the opening is refused; a root at depth 0 under +/// an unsealed document is not the fallback. +#[tokio::test] +async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { + let cipher = stack_cipher().await; + let doc = seal(&cipher, &fixture()).await; + let fallback = |doc: &JsonDocument| -> JsonNode { + let mut wire = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + wire["d"] = json!(0); + serde_json::from_value(wire).unwrap() + }; + let root = fallback(&doc); + let rows = JsonSubtree::new(vec![root.clone()]); + assert!(rows.is_sealed_root()); + assert!(rows.path().is_none()); + + let nested = cipher + .decrypt_as::( + rows.clone().at(JsonPath::parse("$.nested").unwrap()), + aead(), + ) + .await + .unwrap(); + assert_eq!(nested, fixture()["nested"]); + let xs = cipher + .decrypt_as::, JsonSubtree>( + rows.clone() + .at(JsonPath::parse("$.nested.list[*].x").unwrap()), + aead(), + ) + .await + .unwrap(); + assert_eq!(xs, vec![json!(1), json!(2)]); + let none = cipher + .decrypt_as::, JsonSubtree>( + rows.clone().at(JsonPath::parse("$.missing[*]").unwrap()), + aead(), + ) + .await + .unwrap(); + assert!(none.is_empty(), "a path that reaches nothing is no subtree"); + + for (path, why) in [ + ("$.missing", "a path that reaches nothing"), + ("$.nested.list[*].x", "a wildcard path reaching two nodes"), + ] { + let refused = cipher + .decrypt_as::( + rows.clone().at(JsonPath::parse(path).unwrap()), + aead(), + ) + .await; + assert!( + matches!(refused, Err(Error::OpenedType { .. })), + "{why}: {refused:?}" + ); + } + let unattached = cipher + .decrypt_as::(rows.clone(), aead()) + .await; + assert!( + matches!(unattached, Err(Error::OpenedType { .. })), + "the fallback needs the path: {unattached:?}" + ); + + // The same root at depth 1, or at depth 0 under an unsealed document, + // is a walk's row and not the fallback; neither assembles. + let mut as_walked = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + as_walked["d"] = json!(1); + let as_walked: JsonNode = serde_json::from_value(as_walked).unwrap(); + assert!(!JsonSubtree::new(vec![as_walked.clone()]).is_sealed_root()); + let refused = cipher + .decrypt_as::( + JsonSubtree::new(vec![as_walked]).at(JsonPath::parse("$.nested").unwrap()), + aead(), + ) + .await; + assert!(matches!(refused, Err(Error::OpenedType { .. }))); + let open_doc = seal_with(&cipher, &fixture(), JsonOptions { sealed_root: false }).await; + assert!(!JsonSubtree::new(vec![fallback(&open_doc)]).is_sealed_root()); + assert!(!JsonSubtree::new(vec![]).is_sealed_root()); + assert!(!JsonSubtree::new(vec![root.clone(), root]).is_sealed_root()); +} + /// The walk reports each row's depth below where it started, in `d`; an /// extracted node carries none. #[tokio::test] From c98162fdae2db87eb19440c3bd95d46c919d9b2b Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 21 Oct 2026 14:05:05 +1100 Subject: [PATCH 2/7] fix(eql): keep json_tree_query out of the planner's inlining so the GIN index wins over a TOASTed seq scan Re-measured after the projection moved to the wire, the nested corpus's containment queries ran as sequential scans: 1.5 s for a one-row match on 10,000 rows, 17 s on 100,000, with the GIN index present and usable (`enable_seqscan = off` took it at 0.8 ms). Inlined to `value -> 'q'`, the filter is costed as one operator per row, and a column of 97 KB documents is TOASTed out of the heap, so the main relation is a few hundred pages and the sequential scan is costed below the bitmap scan. The planner does not cost the decompression that reading `q` from each row actually pays. So `eql_v4.json_tree_query` is now plpgsql, which the planner cannot inline, with `COST 1000`. The index expression still matches the call, the call keeps its cost, and the planner takes the index on every nested containment scenario at both tiers. The per-row call is a few microseconds. The oracle suite passes against a fresh install. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql-json-tree-walk-and-projection.md | 2 +- ...026-10-17-json-index-stack-encrypt-spike.md | 2 +- packages/eql/src/v3/json_tree/functions.sql | 18 +++++++++++++----- 3 files changed, 15 insertions(+), 7 deletions(-) diff --git a/.changeset/eql-json-tree-walk-and-projection.md b/.changeset/eql-json-tree-walk-and-projection.md index 819c573db..0c7190f91 100644 --- a/.changeset/eql-json-tree-walk-and-projection.md +++ b/.changeset/eql-json-tree-walk-and-projection.md @@ -4,7 +4,7 @@ **The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: -- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. +- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.8 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. - **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same 244 entries return in about 50 ms; `children` and `child_count` in under 3 ms. - **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it (`JsonSubtree::at(path)` in `stack-encrypt`, `JsonTreeSubtree::new(rows).at(path)` in `eql-bindings`). A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. diff --git a/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md b/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md index adfd3f4b2..007f2faf2 100644 --- a/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md +++ b/docs/plans/2026-10-17-json-index-stack-encrypt-spike.md @@ -355,7 +355,7 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: -- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. Nothing new is revealed: `q` is the selectors already in `sv`, regrouped, at about seven per cent of the document. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. +- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. And it is plpgsql with `COST 1000`, not an inlinable SQL function, for a reason the first re-measurement found: inlined to `value -> 'q'` the filter is costed as one operator, and the nested table's rows are TOASTed, so its main heap is a few hundred pages and a sequential scan over it was costed below the GIN bitmap scan and chosen, at 1.5 s for a one-row match on 10,000 rows (17 s on 100,000), because every row is decompressed to read `q`. Un-inlined, the call keeps its cost, the index expression `eql_v4.json_tree_query(col)` still matches it, and the planner takes the index: 0.8 ms. Nothing new is revealed: `q` is the selectors already in `sv`, regrouped, at about seven per cent of the document. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. - **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The 244 entries return in about 50 ms (3.7 ms for a six-entry subtree); `children` and `child_count` in under 3 ms, from 7. - **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it: `JsonSubtree::at(path)` in the engine, `JsonTreeSubtree::new(rows).at(path)` in the bindings, over the new `JsonPath::select`. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. The 244-entry subtree returns the root in about 8 ms. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. - **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 17 ms against 0.12 ms with the index. diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index 0a96d8516..c23ea403e 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -290,16 +290,24 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_exists(jsonb, text) IS --! runs the needle's recursive CHECK, and the recheck would pay it --! once per candidate row. The document's own CHECK validated `q` --! when the row was written. +--! @note plpgsql, deliberately, and costed at 1000: a SQL body this simple +--! would be inlined to `value -> 'q'`, which the planner costs as one +--! operator, and a column of large documents is TOASTed out of the +--! heap, so a sequential scan over it looks almost free on paper and +--! is chosen over the GIN index. It is not free: every row is +--! decompressed to read `q`. Un-inlined, the call keeps its cost, the +--! index expression still matches it, and the planner takes the +--! index. The call itself is a few microseconds per recheck row. CREATE FUNCTION {{prefix}}.json_tree_query(e public.{{prefix}}_json_tree) RETURNS jsonb - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE + COST 1000 AS $$ - SELECT e::jsonb -> 'q' +BEGIN + RETURN e::jsonb -> 'q'; +END; $$; -COMMENT ON FUNCTION {{prefix}}.json_tree_query(public.{{prefix}}_json_tree) IS - 'eql-inline-critical: the containment projection the GIN index is built over; must stay inlinable (unpinned search_path)'; - --! @brief Does the stored document `a` contain the needle `b`? --! --! Native `jsonb @>` over the projections, plus the needle's `alt` projection From 319395d885d256cf1adb85a70d35ca26059a8912 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Wed, 21 Oct 2026 14:35:40 +1100 Subject: [PATCH 3/7] test(eql, stack-encrypt): pin the stored projection's wire key and the fallback's refusal messages CI found two gaps. The DB-free round-trip test still asserted the stored document's keys as `h, i, sv, v`; `q` is one of them now. And the in-diff mutants gate deleted the fallback opening's `0 =>` arm and survived: both refusals are `Error::OpenedType`, and the test checked only the variant. It now asserts each refusal's `expected` text, so a path that reaches nothing and a wildcard that reaches two nodes are told apart. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/eql/tests/encryption/tests/json_tree.rs | 2 +- packages/stack-encrypt/tests/json_index.rs | 16 +++++++++------- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index 07aa9e75d..0370c8139 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -76,7 +76,7 @@ async fn a_document_round_trips_through_the_typed_target() { .keys() .map(String::as_str) .collect(); - assert_eq!(keys, ["h", "i", "sv", "v"]); + assert_eq!(keys, ["h", "i", "q", "sv", "v"]); let opened: Value = cipher .decrypt_as(stored.clone(), Default::default()) diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index f432b2314..28a6b5b02 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -1061,9 +1061,11 @@ async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { .unwrap(); assert!(none.is_empty(), "a path that reaches nothing is no subtree"); - for (path, why) in [ - ("$.missing", "a path that reaches nothing"), - ("$.nested.list[*].x", "a wildcard path reaching two nodes"), + // Each refusal names what it expected: nothing at the path, or one node + // where a wildcard reached two. + for (path, expected) in [ + ("$.missing", "a node at the walked path"), + ("$.nested.list[*].x", "one node at the walked path"), ] { let refused = cipher .decrypt_as::( @@ -1071,10 +1073,10 @@ async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { aead(), ) .await; - assert!( - matches!(refused, Err(Error::OpenedType { .. })), - "{why}: {refused:?}" - ); + match refused { + Err(Error::OpenedType { expected: got }) => assert_eq!(got, expected, "{path}"), + other => panic!("{path}: {other:?}"), + } } let unattached = cipher .decrypt_as::(rows.clone(), aead()) From 923364d26182fad72ff727d644bfd6742f62f008 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 23 Oct 2026 17:55:32 +1100 Subject: [PATCH 4/7] fix(stack-encrypt, eql-bindings): open a fallen-back walk to what the walked rows open to Past its budget a walk returns the sealed root, and the client selects the attached path over the whole document. For json_tree_subtree rows that gave the wrong shape as a list: the walked rows open to the children of the token's node, the fallback opened to the node itself. So a caller reading an encrypted array's elements got the whole array as one element once the subtree passed 64 entries, silently. The rows cannot say which walk produced them, and neither can the path's shape (the ordinal chain walks positional paths through gates), so the caller says it: JsonSubtree::below(path) for json_tree_subtree rows, where a list opens to the node's children, and at(path), as before, for json_tree_subtree_at rows, where it opens to the nodes the path reaches. The same pair is on JsonTreeSubtree. A test opens each shape both walked and fallen back and requires the two to agree. Also records the fallback and the stored projection in the stack-encrypt CHANGELOG, and gives eql-bindings' CHANGELOG the json_tree entry it never had. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/eql/crates/eql-bindings/CHANGELOG.md | 14 +++ .../crates/eql-bindings/src/v4/json_tree.rs | 45 ++++++- packages/stack-encrypt/CHANGELOG.md | 9 +- packages/stack-encrypt/src/json/derive.rs | 43 +++++-- packages/stack-encrypt/src/json/mod.rs | 43 +++++-- packages/stack-encrypt/tests/json_index.rs | 114 ++++++++++++++++++ 6 files changed, 242 insertions(+), 26 deletions(-) diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index e1a655afe..f110c3e99 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -9,6 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **The EQL v4 `json_tree` family's types, `v4::json_tree`** (`stack-encrypt` + feature). `JsonTreeDocument` is `public.eql_v4_json_tree`, `{v, i, h, sv, + q}`: the document's entries, one per JSON node, and `q`, its containment + projection, which the server compares with a needle by native `jsonb @>`. + `JsonTreeEntry` is one entry as `->` and the walks return it, + `JsonTreeQuery` the containment needle and `JsonTreeLeafQuery` the + ordering operand at a path. `JsonTreeSubtree` collects a walk's rows; past + the walk's budget the server returns the sealed root instead, and + `JsonTreeSubtree::below(path)` (for `json_tree_subtree`) or `at(path)` + (for `json_tree_subtree_at`) attaches the path the rows then open at. + The document is an `EncryptFrom` and `DecryptInto` target, + the entry and the subtree open through `DecryptInto`, and the needle and + the ordering operand convert from Stack Encrypt's `JsonQuery` and + `JsonLeafQuery`. - **Range query operands for `eql_v4.between`** (`stack-encrypt` feature). Every ordered v4 family has two: `IntegerRangeOreQuery` and `IntegerRangeOpeQuery`, `TextRangeOreQuery` and `TextRangeOpeQuery`, and so diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index d6ab9727e..0629678bf 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -152,28 +152,52 @@ pub struct JsonTreeLeafQuery { /// walked node, or to the list of complete subtrees the rows hold. /// /// Past its budget a walk returns the sealed root alone, at depth `0`; -/// those rows open to the whole document and the path the walk was keyed -/// from, attached with [`at`](Self::at), selects the value under it. The -/// path is not part of the wire. +/// those rows open to the whole document, and the path the walk was keyed +/// from selects the same value(s) the walked rows open to. Attach it with +/// [`below`](Self::below) for `json_tree_subtree` rows and +/// [`at`](Self::at) for `json_tree_subtree_at` rows. The path is not part +/// of the wire. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(transparent)] pub struct JsonTreeSubtree { pub rows: Vec, #[serde(skip)] pub path: Option, + /// Whether `path` names the node `json_tree_subtree` walked below + /// ([`below`](Self::below)), rather than the nodes a + /// `json_tree_subtree_at` path reaches ([`at`](Self::at)). + #[serde(skip)] + pub below: bool, } impl JsonTreeSubtree { /// The rows, as returned. pub fn new(rows: Vec) -> Self { - Self { rows, path: None } + Self { + rows, + path: None, + below: false, + } } - /// The rows with the walked path, for the sealed root a walk may fall - /// back to. + /// The rows of `json_tree_subtree_at`, with the whole path its key was + /// derived from: past the budget they open to one value per node the + /// path reaches, as the walked rows do. pub fn at(self, path: JsonPath) -> Self { Self { path: Some(path), + below: false, + ..self + } + } + + /// The rows of `json_tree_subtree`, with the path of the node whose + /// token keyed the walk: past the budget they open to that node, or as + /// a list to its children, as the walked rows do. + pub fn below(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: true, ..self } } @@ -373,6 +397,7 @@ impl JsonTreeSubtree { .collect::>()?, ); Ok(match &self.path { + Some(path) if self.below => rows.below(path.clone()), Some(path) => rows.at(path.clone()), None => rows, }) @@ -446,6 +471,14 @@ mod tests { ); assert_eq!(serde_json::to_value(&at).unwrap(), serde_json::json!([])); assert!(at.subtree().unwrap().path().is_some()); + assert!(!at.below); + let below = at.below(JsonPath::parse("$.b").unwrap()); + assert!(below.below); + assert_eq!( + below.subtree().unwrap(), + JsonSubtree::new(Vec::new()).below(JsonPath::parse("$.b").unwrap()) + ); + assert_eq!(serde_json::to_value(&below).unwrap(), serde_json::json!([])); } #[test] diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 0be913577..173780f49 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -28,7 +28,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, `JsonLeafNotOrdered` and `JsonValueNotScalar`. A data plan reads and writes the index (`"json"`, or `{"json": {"sealed_root": bool}}`) and refuses to run it with - `Reason::JsonIndexPending`. + `Reason::JsonIndexPending`. A `JsonDocument` writes its own containment + projection to the wire as `q`, beside `sv`, for the server to compare + with a needle, and ignores a stored one, deriving it from the entries + again. A walk past its budget returns the sealed root in place of the + subtree: `JsonSubtree::below(path)` (for `json_tree_subtree`) or + `JsonSubtree::at(path)` (for `json_tree_subtree_at`) attaches the walked + path, and the rows then open to the same value either way, over + `JsonPath::select`, which evaluates a path over a value. - **A block ORE index, `OreBlock`** (ADR-0002 and ADR-0009, amended 2026-10-19). It derives ore-rs 0.9.0's 6-bit block ORE under its own key diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 3b74f1bcd..c1f82297d 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -1014,16 +1014,29 @@ impl DecryptInto for JsonNode { /// value, in order. What `json_tree_subtree_at` returns is one subtree per /// node whose selector matched; what `json_tree_subtree` returns is the /// children of one node, which [`DecryptInto`] assembles into that -/// node's value. The sealed root a walk fell back to opens whole, and the -/// path attached with [`JsonSubtree::at`] selects over it: one value per -/// node the path reaches. +/// node's value, or this opens into its children's values. The sealed root +/// a walk fell back to opens whole, and the attached path selects over it +/// to the same values the walked rows open to: one per node the path +/// reaches, for a path attached with [`JsonSubtree::at`]; the children of +/// the one node at the path, for [`JsonSubtree::below`]. An object's +/// members come in an order neither form fixes, and without their keys; +/// open it as a [`Value`] to keep them. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { if self.is_sealed_root() { - return open_fallback(self, cipher, context) - .map(|(path, whole)| path.select(&whole).into_iter().cloned().collect()); + let below = self.below; + return open_fallback(self, cipher, context).try_map(move |(path, whole)| { + let found = path.select(&whole); + if !below { + return Ok(found.into_iter().cloned().collect()); + } + match found.as_slice() { + [node] => Ok(children(node)), + _ => Err(shape("one node at the walked path")), + } + }); } open_subtree(self, cipher, context) .map(|forest| forest.into_iter().map(|(_, v)| v).collect()) @@ -1036,9 +1049,9 @@ impl DecryptInto> for JsonSubtree { /// members. No rows is no value ([`Error::OpenedType`]): an empty container /// and a scalar both have no descendants, and `->` on the node's selector /// returns the node itself. The sealed root a walk fell back to opens -/// whole, and the path attached with [`JsonSubtree::at`] selects the one -/// value under it; a path that reaches no node, or more than one, is -/// [`Error::OpenedType`]. +/// whole, and the path attached with [`JsonSubtree::below`] (or +/// [`JsonSubtree::at`]) selects the one value under it; a path that +/// reaches no node, or more than one, is [`Error::OpenedType`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { @@ -1121,6 +1134,18 @@ fn open_subtree<'a, K>( }) } +/// The values of a node's children, in the node's order: an array's +/// elements, an object's member values, and none for a scalar. What a +/// `json_tree_subtree` walk's rows open to as a [`Vec`], taken over the +/// whole document when the walk fell back to the sealed root. +fn children(node: &Value) -> Vec { + match node { + Value::Array(items) => items.clone(), + Value::Object(members) => members.values().cloned().collect(), + _ => Vec::new(), + } +} + /// Open the sealed root a walk fell back to: the whole document, with the /// path the rows were keyed from, which the caller selects over. Refused /// without the path, since nothing in the rows records it. @@ -1132,7 +1157,7 @@ fn open_fallback<'a, K>( let Some(path) = subtree.path().cloned() else { return Pending::failed( cipher, - shape("the walked path, attached with JsonSubtree::at, to select over the sealed root the walk returned"), + shape("the walked path, attached with JsonSubtree::below or JsonSubtree::at, to select over the sealed root the walk returned"), ); }; let Some(root) = subtree.nodes().first().cloned() else { diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index fab84e69a..33c47da7f 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -500,31 +500,53 @@ where /// Past its budget, a walk over a document with a sealed root returns the /// root entry alone, at depth `0`, instead of a subtree it would be /// cheaper to open whole: the client opens the document and takes the -/// walked path over it. Opening such rows needs that path, attached with -/// [`at`](Self::at); without it the opening fails with -/// [`Error::OpenedType`](crate::Error::OpenedType). +/// walked path over it. Opening such rows needs that path and which walk +/// it keyed, attached with [`below`](Self::below) for `json_tree_subtree` +/// or [`at`](Self::at) for `json_tree_subtree_at`, so that the rows open +/// to the same value whichever form the server returned; without a path +/// the opening fails with [`Error::OpenedType`](crate::Error::OpenedType). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(transparent)] pub struct JsonSubtree { nodes: Vec, #[serde(skip)] path: Option, + #[serde(skip)] + below: bool, } impl JsonSubtree { /// The rows, as returned. pub fn new(nodes: Vec) -> Self { - Self { nodes, path: None } + Self { + nodes, + path: None, + below: false, + } } - /// The rows with the path the walk was keyed from: what the opening - /// takes over the whole document when the rows are the sealed root - /// the server fell back to. A positional path opens to one value; a - /// path with a wildcard, as `json_tree_subtree_at` walks, to one value - /// per node it reaches. + /// The rows of `json_tree_subtree_at`, with the whole path its key was + /// derived from: what the opening takes over the whole document when + /// the rows are the sealed root the server fell back to. They open as + /// the walked rows do, to one value per node the path reaches. pub fn at(self, path: JsonPath) -> Self { Self { path: Some(path), + below: false, + ..self + } + } + + /// The rows of `json_tree_subtree`, with the path of the node whose + /// token keyed the walk. They open as the walked rows do, which are + /// that node's descendants: to the node's value, or, as a + /// [`Vec`], to its children's values. Attach the path this way and not + /// with [`at`](Self::at), which would open the fallback to the node + /// itself where the walk opens to its children. + pub fn below(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: true, ..self } } @@ -534,7 +556,8 @@ impl JsonSubtree { &self.nodes } - /// The path attached with [`at`](Self::at), if any. + /// The path attached with [`at`](Self::at) or [`below`](Self::below), + /// if any. pub fn path(&self) -> Option<&JsonPath> { self.path.as_ref() } diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index 6c0ed7ec0..be6c71de4 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -1239,6 +1239,120 @@ async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { assert!(!JsonSubtree::new(vec![root.clone(), root]).is_sealed_root()); } +/// The sealed root a walk fell back to opens to exactly what the walked +/// rows open to, so a caller's result does not change shape when a +/// document grows past the budget. `json_tree_subtree`'s rows are the +/// descendants of the token's node, so the path is attached with `below` +/// and a `Vec` opens to that node's children; `at` is for +/// `json_tree_subtree_at`, whose rows are the nodes the path reaches. +#[tokio::test] +async fn the_fallback_opens_to_what_the_walk_opens_to() { + let cipher = stack_cipher().await; + let doc = seal(&cipher, &fixture()).await; + let mut root = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + root["d"] = json!(0); + let fallback = JsonSubtree::new(vec![serde_json::from_value(root).unwrap()]); + assert!(fallback.is_sealed_root()); + let path = |text: &str| JsonPath::parse(text).unwrap(); + let walked = |text: &'static str| { + let cipher = &cipher; + let doc = &doc; + async move { + let key = subtree_key(cipher, text, JsonOptions::default()) + .await + .unwrap(); + let rows = doc + .subtree(key.token_bytes()) + .iter() + .map(|(j, _, _)| doc.node(*j).unwrap()) + .collect(); + JsonSubtree::new(rows) + } + }; + let sorted = |mut values: Vec| { + values.sort_by_key(ToString::to_string); + values + }; + + // An array: its elements, in order, either way. + let list = cipher + .decrypt_as::, JsonSubtree>(walked("$.nested.list").await, aead()) + .await + .unwrap(); + assert_eq!(list, vec![json!({"x": 1}), json!({"x": 2})]); + let below = |text: &str| fallback.clone().below(path(text)); + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>(below("$.nested.list"), aead()) + .await + .unwrap(), + list + ); + // An object: its member values, in an order neither form fixes. + let members = cipher + .decrypt_as::, JsonSubtree>(walked("$.nested").await, aead()) + .await + .unwrap(); + assert_eq!(members.len(), 2); + assert_eq!( + sorted( + cipher + .decrypt_as::, JsonSubtree>(below("$.nested"), aead()) + .await + .unwrap() + ), + sorted(members) + ); + // As one value, the node itself, either way. + assert_eq!( + cipher + .decrypt_as::(below("$.nested"), aead()) + .await + .unwrap(), + fixture()["nested"] + ); + // A scalar has no children, as its walk has no rows. + assert!(cipher + .decrypt_as::, JsonSubtree>(below("$.hello"), aead()) + .await + .unwrap() + .is_empty()); + // `below` names one node: nothing at the path, or a wildcard reaching + // two, is refused. + for text in ["$.missing", "$.nested.list[*]"] { + match cipher + .decrypt_as::, JsonSubtree>(below(text), aead()) + .await + { + Err(Error::OpenedType { expected }) => { + assert_eq!(expected, "one node at the walked path", "{text}") + } + other => panic!("{text}: {other:?}"), + } + } + // `at` is the other walk's: the nodes the path reaches, so on this path + // the list itself. Attaching the wrong one is what changes the shape. + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>( + fallback.clone().at(path("$.nested.list")), + aead() + ) + .await + .unwrap(), + vec![fixture()["nested"]["list"].clone()] + ); + let attached = fallback.below(path("$.nested")); + assert_eq!( + attached.path().map(ToString::to_string).as_deref(), + Some("$.nested") + ); + assert_eq!( + attached.clone().at(path("$.nested")).path(), + attached.path() + ); +} + /// The walk reports each row's depth below where it started, in `d`; an /// extracted node carries none. #[tokio::test] From c1059d775965d1686eeb1cf1fbd4191dc0591dd2 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 23 Oct 2026 17:55:33 +1100 Subject: [PATCH 5/7] fix(eql): bound the json_tree walks by the document's size, not only its depth A stored document's chain fields are not authenticated, and two of them let a row stop a query. `w` wraps a child's token under its parent's with a public unwrap, so whoever holds a token can write children whose tokens are their parent's: two per node double the walk's rows at each of the 127 levels the depth bound allows. And the fan-out every walk level probes was the document's largest `o` plus one, so one entry claiming o = 2147483646 made json_tree_children hash two billion tags. Both ran until statement_timeout on a three-entry document. A sealed document's walk visits each entry at most once, and a node's ordinals sit below the entry count, so both caps cost an honest walk nothing: the walks read their recursion through LIMIT entries (which also stops the recursion, since PostgreSQL computes a recursive query only as far as its reader pulls), and json_tree_fanout takes the least of max(o) + 1 and the entry count. The Postgres suite now builds both hostile documents, sealed and not, under a statement_timeout so a regression fails instead of hanging; and pins the two fallback shapes the suite did not cover: a list opened walked and fallen back, and the budget gated on the header's sealed-root flag. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/eql/src/v3/json_tree/functions.sql | 53 +++++- .../eql/tests/encryption/tests/json_tree.rs | 155 +++++++++++++++++- 2 files changed, 198 insertions(+), 10 deletions(-) diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index 799e253ad..3b35559fb 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -110,7 +110,15 @@ AS $$ $$; --! @brief How many ordinals a walk tries under each node: one more than the ---! largest ordinal any entry of the document carries. +--! largest ordinal any entry of the document carries, and never more +--! than the document has entries. +--! +--! A node's children sit at ordinals below its child count, which is below +--! the document's entry count, so the cap changes nothing for a sealed +--! document. It is there for a stored one: `o` is not authenticated, and +--! without the cap one entry carrying `o = 2147483646` would make every +--! level of every walk hash two billion candidate tags. +--! --! @internal --! @param val jsonb The document. --! @return integer The fan-out bound; `0` for a document with no chained @@ -119,7 +127,7 @@ CREATE FUNCTION {{prefix}}_internal.json_tree_fanout(val jsonb) RETURNS integer LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT COALESCE(MAX((elem ->> 'o')::integer), -1) + 1 + SELECT LEAST(COALESCE(MAX((elem ->> 'o')::integer), -1) + 1, count(*)::integer) FROM jsonb_array_elements(val -> 'sv') AS elem $$; @@ -427,7 +435,10 @@ $$; --! and the client reads the node's kind from its children's labels, or --! fetches the node by its selector. --! The walk descends at most 127 levels, the deepest a sealed document ---! nests, so a stored document whose tags do not form a tree still ends it. +--! nests, and returns at most as many rows as the document has entries, +--! which a tree never exceeds. A stored document whose tags form a longer +--! chain or a loop therefore still ends it, after at most its entry count +--! times its fan-out in hashes. --! --! **The budget.** A walk costs the fan-out in hashes per node it visits, --! and returns about four hundred characters per entry, where a sealed root @@ -460,6 +471,7 @@ AS $$ {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, {{prefix}}.json_tree_meta(e::jsonb) AS meta, e::jsonb -> 'sv' -> 0 AS root, + jsonb_array_length(e::jsonb -> 'sv') AS entries, budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), walk(depth, t, entry) AS ( @@ -475,15 +487,26 @@ AS $$ SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry ) AS c -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a cycle or a longer chain, which whoever holds a - -- token can write, stops here instead of recursing without end. + -- whose tags form a longer chain, which whoever holds a token can + -- write, stops here. A cycle that branches is stopped by `visited`. WHERE c.entry IS NOT NULL AND w.depth < 127 ), + -- A walk of a sealed document reaches each entry at most once, so it + -- has at most as many rows as the document has entries, the seed row + -- standing in for the root. A stored document can loop instead: `w` + -- is not authenticated, so whoever holds a token can wrap a child's + -- token to its parent's, and two such children double the rows at + -- every level the depth bound allows. PostgreSQL computes a recursive + -- query only as far as its reader pulls rows, so this limit stops the + -- recursion, and not only the output. + visited AS MATERIALIZED ( + SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) + ), -- One row past the budget is enough to know the walk went over; a walk -- under it is the subtree complete. Uncapped, the limit is NULL: all. found AS MATERIALIZED ( SELECT w.depth, w.entry - FROM walk AS w + FROM visited AS w WHERE w.entry IS NOT NULL LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) ), @@ -535,6 +558,7 @@ AS $$ {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, {{prefix}}.json_tree_meta(e::jsonb) AS meta, e::jsonb -> 'sv' -> 0 AS root, + jsonb_array_length(e::jsonb -> 'sv') AS entries, budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), walk(depth, t, entry, hit) AS ( @@ -551,13 +575,24 @@ AS $$ SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry ) AS c -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a cycle or a longer chain, which whoever holds a - -- token can write, stops here instead of recursing without end. + -- whose tags form a longer chain, which whoever holds a token can + -- write, stops here. A cycle that branches is stopped by `visited`. WHERE c.entry IS NOT NULL AND w.depth < 127 ), + -- A walk of a sealed document reaches each entry at most once, so it + -- has at most as many rows as the document has entries, the seed row + -- standing in for the root. A stored document can loop instead: `w` + -- is not authenticated, so whoever holds a token can wrap a child's + -- token to its parent's, and two such children double the rows at + -- every level the depth bound allows. PostgreSQL computes a recursive + -- query only as far as its reader pulls rows, so this limit stops the + -- recursion, and not only the output. + visited AS MATERIALIZED ( + SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) + ), found AS MATERIALIZED ( SELECT w.depth, w.entry - FROM walk AS w + FROM visited AS w WHERE w.entry IS NOT NULL AND w.hit LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) ), diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index 827fa6816..0250b449d 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -582,7 +582,7 @@ fn postgres_extracts_walks_and_orders_by_path() { let value: Value = wait( &runtime, cipher.decrypt_as( - JsonTreeSubtree::new(root).at(JsonPath::parse("$.nested").unwrap()), + JsonTreeSubtree::new(root).below(JsonPath::parse("$.nested").unwrap()), column(), ), ) @@ -596,6 +596,58 @@ fn postgres_extracts_walks_and_orders_by_path() { .unwrap() .get(0); assert_eq!(garbage, 0); + // The fallback opens to what the walk opens to, as a list too: the + // list's elements, whether the server walked its four entries or + // returned the sealed root, so a caller's result does not change shape + // when a document grows past the budget. + let list = subtree_key("$.nested.list"); + let list_rows = |db: &mut postgres::Client, budget: i32| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", + &[&list.token(), &budget], + ) + .unwrap() + .iter() + .map(entry) + .collect() + }; + let elements = |rows: Vec| -> Vec { + wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(rows).below(JsonPath::parse("$.nested.list").unwrap()), + column(), + ), + ) + .unwrap() + }; + let walked_list = list_rows(&mut db, 0); + assert_eq!(walked_list.len(), 4); + let fallen_back = list_rows(&mut db, 3); + assert_eq!(fallen_back.len(), 1); + assert_eq!( + elements(walked_list), + vec![json!({"x": 1}), json!({"x": 2})] + ); + assert_eq!( + elements(fallen_back), + vec![json!({"x": 1}), json!({"x": 2})] + ); + // The budget is gated on the header's sealed-root flag, the bit + // `JsonHeader` writes: with the flags byte cleared, the same document + // has no root to return and is walked out under any budget. + let unflagged: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_rows, + eql_v4.json_tree_subtree( + jsonb_set(doc::jsonb, '{h}', to_jsonb(encode(set_byte(decode(doc::jsonb ->> 'h', 'base64'), 1, 0), 'base64')))::public.eql_v4_json_tree, + $1::text, 1) e + WHERE id = 1", + &[&nested.token()], + ) + .unwrap() + .get(0); + assert_eq!(unflagged, 6, "an unsealed root is walked out"); // A wildcard below the anchor: every element's `x`, each a complete // subtree, from the list's token gated on the whole path's selector. @@ -765,3 +817,104 @@ fn postgres_extracts_walks_and_orders_by_path() { } } } + +/// Needs an EQL v4 installation. A stored document is untrusted input: its +/// chain fields are not authenticated, so whoever holds a token can write +/// children whose tokens loop back to their parent's, or an ordinal far +/// past the document's size. Every walk still ends, in time bounded by the +/// document's size, under any budget; `statement_timeout` turns a walk +/// that would not end into a failure instead of a hung suite. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_walks_end_on_hostile_documents() { + let mut db = connect(); + db.batch_execute("SET statement_timeout = '10s'").unwrap(); + // A root and two children at ordinals 0 and 1 under the all-zero + // token, each wrapping that same token as its own: unbounded, the walk + // doubles at every one of its 127 levels. `flags` is the header's flags + // byte, so the same document is built with its root sealed and not. + let looping = |flags: i32| { + format!( + "WITH k AS ( + SELECT set_byte(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 1, {flags}) AS h, + decode(repeat('00', 16), 'hex') AS iv, + decode(repeat('00', 16), 'hex') AS t + ), e AS ( + SELECT o, c, + encode(eql_v4_internal.json_tree_h16('p'::bytea || k.iv || k.t || eql_v4_internal.json_tree_le32(o)), 'hex') AS p, + encode(eql_v4_internal.json_tree_xor16(k.t, eql_v4_internal.json_tree_h16('w'::bytea || k.iv || k.t || decode(c, 'base64'))), 'hex') AS w + FROM k, (VALUES (0, 'AA=='), (1, 'AQ==')) AS v(o, c) + ) + SELECT (jsonb_build_object( + 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), + 'h', (SELECT encode(h, 'base64') FROM k), + 'sv', jsonb_build_array(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==')) + || (SELECT jsonb_agg(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', c, 'o', o, 'p', p, 'w', w) ORDER BY o) FROM e), + 'q', '{{\"sv\": [], \"el\": []}}'::jsonb))::public.eql_v4_json_tree" + ) + }; + let token = "00".repeat(16); + for flags in [0, 1] { + let doc = looping(flags); + let count = |db: &mut postgres::Client, walk: &str| -> i64 { + db.query_one( + &format!("SELECT count(*) FROM ({doc}) AS d(doc), {walk} e"), + &[&token], + ) + .unwrap_or_else(|error| panic!("{walk} (flags {flags}): {error}")) + .get(0) + }; + assert_eq!( + count(&mut db, "eql_v4.json_tree_children(d.doc, $1::text)"), + 2 + ); + for budget in [0, 64] { + let walked = count( + &mut db, + &format!("eql_v4.json_tree_subtree(d.doc, $1::text, {budget})"), + ); + assert!( + walked <= 3, + "a walk returns at most as many rows as the document has entries: {walked} (flags {flags}, budget {budget})" + ); + // No entry carries the selector, so nothing is emitted and the + // budget never trips: only the walk's own bound ends it. + assert_eq!( + count( + &mut db, + &format!("eql_v4.json_tree_subtree_at(d.doc, $1::text, 'ff', {budget})"), + ), + 0 + ); + } + } + + // One entry claiming the largest ordinal an integer holds: the fan-out + // is capped at the document's entry count, so the probe is two tags, + // not two billion. + let wide = "SELECT (jsonb_build_object( + 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), + 'h', encode(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 'base64'), + 'sv', jsonb_build_array( + jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA=='), + jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==', 'o', 2147483646, 'p', '00', 'w', '00')), + 'q', '{\"sv\": [], \"el\": []}'::jsonb))::public.eql_v4_json_tree"; + let children: i32 = db + .query_one( + &format!( + "SELECT eql_v4.json_tree_child_count(d.doc, $1::text) FROM ({wide}) AS d(doc)" + ), + &[&token], + ) + .unwrap() + .get(0); + assert_eq!(children, 0); + let walked: i64 = db + .query_one( + &format!("SELECT count(*) FROM ({wide}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, 0) e"), + &[&token], + ) + .unwrap() + .get(0); + assert_eq!(walked, 0); +} From 014d25d08acd1393649af82cb6c2e423ec4f5ca0 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 23 Oct 2026 17:55:33 +1100 Subject: [PATCH 6/7] docs(eql): align the json_tree Phase 3 notes with the report they cite, and state what q is trusted for The design doc and the changeset quoted an interim run (50 ms for the walked subtree, 0.8 ms for a one-row match, q at seven per cent of the document, the document at 24 times its plaintext) where the report the PR cites, cipherstash/benches 57abcf9, says 23 ms, 0.5 to 0.7 ms, about 17 per cent and 29 times. They now quote the report and name its commits, and flag the one count the report disagrees with itself on. They also say what the numbers leave out: the fallback's cost on the client, which the benchmarks do not decrypt, and that it puts every field of the document in the client's memory for a subtree read; and that the server trusts q as written, so a row not written by the index can answer @> differently from its entries. Whether to verify q at write time, and whether the budget should be on by default, are left open for the owner. operators.sql still described the index expression with the ::jsonb cast the projection no longer needs. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .changeset/eql-json-tree-walk-and-projection.md | 12 ++++++------ docs/plans/2026-10-17-json-tree-index.md | 12 ++++++------ packages/eql/src/v3/json_tree/operators.sql | 2 +- packages/eql/src/v3/json_tree/types.sql | 5 ++++- 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/.changeset/eql-json-tree-walk-and-projection.md b/.changeset/eql-json-tree-walk-and-projection.md index 0c7190f91..d44a6560b 100644 --- a/.changeset/eql-json-tree-walk-and-projection.md +++ b/.changeset/eql-json-tree-walk-and-projection.md @@ -2,12 +2,12 @@ "@cipherstash/eql": minor --- -**The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: +**The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured (cipherstash/benches, `report/JSON_TREE_COMPARISON.md` at 57abcf9) on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: -- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.8 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. -- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same 244 entries return in about 50 ms; `children` and `child_count` in under 3 ms. -- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it (`JsonSubtree::at(path)` in `stack-encrypt`, `JsonTreeSubtree::new(rows).at(path)` in `eql-bindings`). A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. +- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.5 to 0.7 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. +- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size: it probes at most as many ordinals as the document has entries and returns at most as many rows, so a stored document whose unauthenticated chain fields claim a huge ordinal, or wrap a child's token back to its parent's, can no longer keep a query running until `statement_timeout`. +- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to: attach the path with `below(path)` for `json_tree_subtree` rows and `at(path)` for `json_tree_subtree_at` rows (`JsonSubtree` in `stack-encrypt`, `JsonTreeSubtree::new(rows)` in `eql-bindings`). Rows that fell back and carry no path refuse to open. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. -Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (17 ms down to 0.1 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed. +Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (16.9 ms down to 0.11 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed. -The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows` and an optional `path`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. +The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows`, an optional `path` and `below`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::below`, `at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 27879284a..d46293c6e 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -357,14 +357,14 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi ## Implementation notes (2026-10-21, Phase 3: first measurements and the fixes they forced) -The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: +The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`; the first run at commit 076ebfc, the re-measurement of this phase's fixes at 57abcf9, both at 10,000 and 100,000 rows) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The figures below are that report's at 10,000 rows unless they say otherwise. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: -- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. And it is plpgsql with `COST 1000`, not an inlinable SQL function, for a reason the first re-measurement found: inlined to `value -> 'q'` the filter is costed as one operator, and the nested table's rows are TOASTed, so its main heap is a few hundred pages and a sequential scan over it was costed below the GIN bitmap scan and chosen, at 1.5 s for a one-row match on 10,000 rows (17 s on 100,000), because every row is decompressed to read `q`. Un-inlined, the call keeps its cost, the index expression `eql_v4.json_tree_query(col)` still matches it, and the planner takes the index: 0.8 ms. Nothing new is revealed: `q` is the selectors already in `sv`, regrouped, at about seven per cent of the document. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. -- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The 244 entries return in about 50 ms (3.7 ms for a six-entry subtree); `children` and `child_count` in under 3 ms, from 7. -- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it: `JsonSubtree::at(path)` in the engine, `JsonTreeSubtree::new(rows).at(path)` in the bindings, over the new `JsonPath::select`. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. The 244-entry subtree returns the root in about 8 ms. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. -- **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 17 ms against 0.12 ms with the index. +- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. And it is plpgsql with `COST 1000`, not an inlinable SQL function, for a reason the first re-measurement found: inlined to `value -> 'q'` the filter is costed as one operator, and the nested table's rows are TOASTed, so its main heap is a few hundred pages and a sequential scan over it was costed below the GIN bitmap scan and chosen, at 1.5 s for a one-row match on 10,000 rows (17 s on 100,000), because every row is decompressed to read `q`. Un-inlined, the call keeps its cost, the index expression `eql_v4.json_tree_query(col)` still matches it, and the planner takes the index: 0.5 to 0.7 ms for a one-row match, 3.2 ms for ten deep-object rows (0.36 ms native), and 5 s to build the index. Nothing new is revealed: `q` is the selectors already in `sv`, grouped as `sv`'s own `n` and `g` tags already group them. It is not small: 19,100 of a nested document's 114,309 characters, about 17 per cent, though nothing measurable on disk, since its hex pairs compress against `sv`. And it is trusted as written: the CHECK validates its shape, not that it is `sv`'s, so a row written by anything but this index can answer `@>` differently from `->`, the walks and decryption, which answer from `sv`. The server-built projection could not disagree with the entries; verifying `q` against `sv` at write time would restore that at roughly the old per-row cost, paid once per write instead of once per recheck, and is left open. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. +- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The `$.orders` subtree walks out in 23 ms (the six-entry `$.company` subtree in 2.8 ms); `children` and `child_count` in 1.6 ms, from 7.1. The report's prose calls `$.orders` the same 244 entries as the first run's 10.2 s, while its table counts 122 rows for the walked scenario; which count the 23 ms is for is still to be confirmed. The walk is bounded by the document as well as by depth: the fan-out it probes is at most the document's entry count, where a stored `o` could otherwise ask for two billion, and it returns at most as many rows as the document has entries, where a stored `w` that wraps a child's token to its parent's could otherwise double the rows at every level. Neither changes a sealed document's walk; the Postgres suite holds both against hostile documents (`postgres_walks_end_on_hostile_documents`). +- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it, over the new `JsonPath::select`, and the result is what the walked rows open to: the path is attached with `JsonSubtree::below(path)` for `json_tree_subtree`, whose rows are the descendants of the path's node, and `JsonSubtree::at(path)` for `json_tree_subtree_at`, whose rows are the nodes the path reaches (the same two on `JsonTreeSubtree` in the bindings). The two differ when the rows are opened as a list, to the node's children or to the node, which is why the rows cannot be opened without saying which walk they came from. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. `$.orders` returns the root in 9.5 ms, against 23 ms walked out; `json_tree_subtree_at` over its items in 13 ms against 23. Two costs are not in those numbers: the client's, since the benchmarks do not decrypt, which for the fallback is one AES-GCM open and a JSON parse of a document about 1.3 times the plaintext's size; and the plaintext the client then holds, which is every field of the document and not the subtree asked for. The data key is the same, so nothing is readable that was not, but the note above that a `serde_json::Value` is never zeroized now covers the whole document on every subtree read past the budget. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. +- **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 16.9 ms against 0.11 ms with the index, at both tiers. -Still open from the measurements, filed as a follow-up: `->` scans `sv` on every extraction (0.8 ms on a nested document; an index of `sv` by `s` would make it a lookup); `json_tree_contains` plans a second GIN probe for `alt` even when the needle carries none; `ORDER BY ord_term(col -> sel)` measured 0.92 ms against v3's 0.25 ms on an index scan both ways, unexplained; and the nested document is 24 times its plaintext (hex tags, a random nonce per entry, the sealed-root copy), which decisions 1 and 6 above and the opaque-subtree policy are the levers on. +Still open from the measurements, filed as a follow-up: `->` scans `sv` on every extraction (0.6 ms on a nested document, 0.2 on a flat one, against 0.1 native; an index of `sv` by `s` would make it a lookup); `json_tree_contains` plans a second GIN probe for `alt` even when the needle carries none; `ORDER BY ord_term(col -> sel)` measured 0.96 ms against v3's 0.25 ms on an index scan both ways, unexplained; and the nested document is 29 times its plaintext in characters and 34 times on disk (hex tags, a random nonce per entry, the sealed-root copy, and now `q`), which decisions 1 and 6 above and the opaque-subtree policy are the levers on. Also open, for the owner: whether the budget should be on by default (a caller who opens a walk's rows without attaching a path works until a document passes the budget, then fails), and whether to verify `q` at write time. ## Decisions to take before Phase 1 diff --git a/packages/eql/src/v3/json_tree/operators.sql b/packages/eql/src/v3/json_tree/operators.sql index e4c75d78c..0d81b48b0 100644 --- a/packages/eql/src/v3/json_tree/operators.sql +++ b/packages/eql/src/v3/json_tree/operators.sql @@ -63,7 +63,7 @@ CREATE OPERATOR ->> ( --! @brief @> contains operator with a query_json_tree needle. --! ---! Inlines to native `jsonb @>` over `{{prefix}}.json_tree_query(a)::jsonb`, so +--! Inlines to native `jsonb @>` over `{{prefix}}.json_tree_query(a)`, so --! a functional GIN index on that expression engages. Not STRICT, for the --! reason `{{prefix}}.json_tree_contains` gives: a STRICT wrapper around a --! non-STRICT body is not inlined. diff --git a/packages/eql/src/v3/json_tree/types.sql b/packages/eql/src/v3/json_tree/types.sql index 86e4ef6b5..108477f4d 100644 --- a/packages/eql/src/v3/json_tree/types.sql +++ b/packages/eql/src/v3/json_tree/types.sql @@ -40,7 +40,10 @@ --! projection, `{"sv": [{s, vs}, …], "el": [group, …]}`: its `{s, vs}` --! pairs nested by array element, written by the client beside `sv` so the --! server compares it with a needle by native `jsonb @>` and never ---! rebuilds it from the entries. +--! rebuilds it from the entries. The CHECK validates its shape, not that it +--! is `sv`'s: containment answers from `q` as written, so a row written by +--! anything but Stack Encrypt's JSON index can answer `@>` differently +--! from `->`, the walks and decryption, which answer from `sv`. --! @brief Validate one JSON tree entry. --! @internal From 31d5da455d995527483e3f29ddf07071a6ea32ea Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Fri, 23 Oct 2026 17:56:25 +1100 Subject: [PATCH 7/7] test(stack-encrypt): pin that the last attachment decides the fallback's walk `at` and `below` each set both the path and which walk the rows came from; a test that only compared paths would let either keep the other's walk. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/stack-encrypt/tests/json_index.rs | 27 ++++++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index be6c71de4..ef89f3e69 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -1342,14 +1342,31 @@ async fn the_fallback_opens_to_what_the_walk_opens_to() { .unwrap(), vec![fixture()["nested"]["list"].clone()] ); - let attached = fallback.below(path("$.nested")); + // The last attachment decides, both the path and the walk. + let reattached = fallback + .clone() + .below(path("$.nested")) + .at(path("$.nested.list")); assert_eq!( - attached.path().map(ToString::to_string).as_deref(), - Some("$.nested") + reattached.path().map(ToString::to_string).as_deref(), + Some("$.nested.list") ); assert_eq!( - attached.clone().at(path("$.nested")).path(), - attached.path() + cipher + .decrypt_as::, JsonSubtree>(reattached, aead()) + .await + .unwrap(), + vec![fixture()["nested"]["list"].clone()] + ); + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>( + fallback.at(path("$.nested")).below(path("$.nested.list")), + aead() + ) + .await + .unwrap(), + list ); }