diff --git a/.changeset/eql-json-tree-walk-and-projection.md b/.changeset/eql-json-tree-walk-and-projection.md new file mode 100644 index 000000000..d44a6560b --- /dev/null +++ b/.changeset/eql-json-tree-walk-and-projection.md @@ -0,0 +1,13 @@ +--- +"@cipherstash/eql": minor +--- + +**The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured (cipherstash/benches, `report/JSON_TREE_COMPARISON.md` at 57abcf9) on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: + +- **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.5 to 0.7 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. +- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size: it probes at most as many ordinals as the document has entries and returns at most as many rows, so a stored document whose unauthenticated chain fields claim a huge ordinal, or wrap a child's token back to its parent's, can no longer keep a query running until `statement_timeout`. +- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to: attach the path with `below(path)` for `json_tree_subtree` rows and `at(path)` for `json_tree_subtree_at` rows (`JsonSubtree` in `stack-encrypt`, `JsonTreeSubtree::new(rows)` in `eql-bindings`). Rows that fell back and carry no path refuse to open. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. + +Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (16.9 ms down to 0.11 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed. + +The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows`, an optional `path` and `below`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::below`, `at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 3ccbd3ff7..d46293c6e 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -206,7 +206,8 @@ A document in the new domain, `public.eql_v4_json_tree` (placeholder name; a sib { "s": "…", "vs": "…", "c": "…" }, // $.list (array) a chain head { "s": "…", "vs": "…", "p": "…", "w": "…", "n": "…", "c": "…" }, // $.list[*] element 0 (object) { "s": "…", "vs": "…", "p": "…", "w": "…", "g": "…", "c": "…", "op": "…" } // $.list[*].x = 1, inside element 0 - ] + ], + "q": { "sv": [ {"s": "…", "vs": "…"}, … ], "el": [ { "sv": [ … ], "el": [ … ] } ] } // the containment projection, written by the client (Phase 3) } ``` @@ -354,6 +355,17 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi - *A stored document is read as untrusted input.* Its chain fields are not authenticated, and whoever holds a token can write tags that form a cycle or a chain deeper than any sealed document. `JsonDocument::children` checks each entry at its own ordinal instead of looping over a fan-out the document chooses; `JsonDocument::subtree` and the SQL walks stop below 127 levels, and the Rust walk also stops once it has returned as many entries as the document holds; the rebuild of a subtree's rows refuses a repeated row and a nesting deeper than 127 levels as `OpenedType`. Each would otherwise have overflowed the stack, looped, or allocated what the document asked for. - *Nothing wipes the plaintext copies the walk makes* (the per-node scalars it clones before sealing), where the scalar paths hold theirs in `Zeroizing`. A `serde_json::Value` does not zeroize; a follow-up should walk the caller's value without copying scalars out of it. +## Implementation notes (2026-10-21, Phase 3: first measurements and the fixes they forced) + +The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`; the first run at commit 076ebfc, the re-measurement of this phase's fixes at 57abcf9, both at 10,000 and 100,000 rows) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The figures below are that report's at 10,000 rows unless they say otherwise. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: + +- **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. And it is plpgsql with `COST 1000`, not an inlinable SQL function, for a reason the first re-measurement found: inlined to `value -> 'q'` the filter is costed as one operator, and the nested table's rows are TOASTed, so its main heap is a few hundred pages and a sequential scan over it was costed below the GIN bitmap scan and chosen, at 1.5 s for a one-row match on 10,000 rows (17 s on 100,000), because every row is decompressed to read `q`. Un-inlined, the call keeps its cost, the index expression `eql_v4.json_tree_query(col)` still matches it, and the planner takes the index: 0.5 to 0.7 ms for a one-row match, 3.2 ms for ten deep-object rows (0.36 ms native), and 5 s to build the index. Nothing new is revealed: `q` is the selectors already in `sv`, grouped as `sv`'s own `n` and `g` tags already group them. It is not small: 19,100 of a nested document's 114,309 characters, about 17 per cent, though nothing measurable on disk, since its hex pairs compress against `sv`. And it is trusted as written: the CHECK validates its shape, not that it is `sv`'s, so a row written by anything but this index can answer `@>` differently from `->`, the walks and decryption, which answer from `sv`. The server-built projection could not disagree with the entries; verifying `q` against `sv` at write time would restore that at roughly the old per-row cost, paid once per write instead of once per recheck, and is left open. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. +- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The `$.orders` subtree walks out in 23 ms (the six-entry `$.company` subtree in 2.8 ms); `children` and `child_count` in 1.6 ms, from 7.1. The report's prose calls `$.orders` the same 244 entries as the first run's 10.2 s, while its table counts 122 rows for the walked scenario; which count the 23 ms is for is still to be confirmed. The walk is bounded by the document as well as by depth: the fan-out it probes is at most the document's entry count, where a stored `o` could otherwise ask for two billion, and it returns at most as many rows as the document has entries, where a stored `w` that wraps a child's token to its parent's could otherwise double the rows at every level. Neither changes a sealed document's walk; the Postgres suite holds both against hostile documents (`postgres_walks_end_on_hostile_documents`). +- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it, over the new `JsonPath::select`, and the result is what the walked rows open to: the path is attached with `JsonSubtree::below(path)` for `json_tree_subtree`, whose rows are the descendants of the path's node, and `JsonSubtree::at(path)` for `json_tree_subtree_at`, whose rows are the nodes the path reaches (the same two on `JsonTreeSubtree` in the bindings). The two differ when the rows are opened as a list, to the node's children or to the node, which is why the rows cannot be opened without saying which walk they came from. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. `$.orders` returns the root in 9.5 ms, against 23 ms walked out; `json_tree_subtree_at` over its items in 13 ms against 23. Two costs are not in those numbers: the client's, since the benchmarks do not decrypt, which for the fallback is one AES-GCM open and a JSON parse of a document about 1.3 times the plaintext's size; and the plaintext the client then holds, which is every field of the document and not the subtree asked for. The data key is the same, so nothing is readable that was not, but the note above that a `serde_json::Value` is never zeroized now covers the whole document on every subtree read past the budget. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. +- **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 16.9 ms against 0.11 ms with the index, at both tiers. + +Still open from the measurements, filed as a follow-up: `->` scans `sv` on every extraction (0.6 ms on a nested document, 0.2 on a flat one, against 0.1 native; an index of `sv` by `s` would make it a lookup); `json_tree_contains` plans a second GIN probe for `alt` even when the needle carries none; `ORDER BY ord_term(col -> sel)` measured 0.96 ms against v3's 0.25 ms on an index scan both ways, unexplained; and the nested document is 29 times its plaintext in characters and 34 times on disk (hex tags, a random nonce per entry, the sealed-root copy, and now `q`), which decisions 1 and 6 above and the opaque-subtree policy are the levers on. Also open, for the owner: whether the budget should be on by default (a caller who opens a walk's rows without attaching a path works until a document passes the budget, then fails), and whether to verify `q` at write time. + ## Decisions to take before Phase 1 1. **Token encoding on the wire:** hex, matching v3's `s`, or base64url, saving about 40 characters per entry. Proposal: measure in Phase 3, default to base64url. diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index e1a655afe..f110c3e99 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -9,6 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **The EQL v4 `json_tree` family's types, `v4::json_tree`** (`stack-encrypt` + feature). `JsonTreeDocument` is `public.eql_v4_json_tree`, `{v, i, h, sv, + q}`: the document's entries, one per JSON node, and `q`, its containment + projection, which the server compares with a needle by native `jsonb @>`. + `JsonTreeEntry` is one entry as `->` and the walks return it, + `JsonTreeQuery` the containment needle and `JsonTreeLeafQuery` the + ordering operand at a path. `JsonTreeSubtree` collects a walk's rows; past + the walk's budget the server returns the sealed root instead, and + `JsonTreeSubtree::below(path)` (for `json_tree_subtree`) or `at(path)` + (for `json_tree_subtree_at`) attaches the path the rows then open at. + The document is an `EncryptFrom` and `DecryptInto` target, + the entry and the subtree open through `DecryptInto`, and the needle and + the ordering operand convert from Stack Encrypt's `JsonQuery` and + `JsonLeafQuery`. - **Range query operands for `eql_v4.between`** (`stack-encrypt` feature). Every ordered v4 family has two: `IntegerRangeOreQuery` and `IntegerRangeOpeQuery`, `TextRangeOreQuery` and `TextRangeOpeQuery`, and so diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index f0d7576d5..0629678bf 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -22,7 +22,9 @@ use schemars::{schema_for, JsonSchema, Schema}; use serde::{Deserialize, Serialize}; use serde_json::Value; -use stack_encrypt::json::{JsonDocument, JsonLeafQuery, JsonNode, JsonQuery, JsonSubtree}; +use stack_encrypt::json::{ + JsonDocument, JsonLeafQuery, JsonNode, JsonPath, JsonQuery, JsonSubtree, +}; use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec}; use stack_encrypt::{ DecryptField, DecryptInto, Decryptable, Decryption, EncryptFrom, Encryption, Error, NonEmpty, @@ -40,9 +42,12 @@ use crate::Identifier; pub struct JsonTreeTag(pub String); /// `public.eql_v4_json_tree`: a JSON tree document, `{v, i, h, sv: -/// [entry]}`. Strict. `h` is the document's key header, stored once (every -/// entry is sealed under the document's one data key); `sv` holds one -/// entry per node in depth-first order, the root first. +/// [entry], q: group}`. Strict. `h` is the document's key header, stored +/// once (every entry is sealed under the document's one data key); `sv` +/// holds one entry per node in depth-first order, the root first; `q` is +/// the document's containment projection, its `{s, vs}` pairs nested by +/// array element, which the client writes beside the entries so the +/// server's `eql_v4.json_tree_query` reads it rather than rebuilding it. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeDocument { @@ -50,6 +55,7 @@ pub struct JsonTreeDocument { pub i: Identifier, pub h: KeyHeader, pub sv: Vec, + pub q: JsonTreeQueryGroup, } /// `public.eql_v4_json_tree_entry`: one node. `s` and `vs` are the path @@ -144,9 +150,58 @@ pub struct JsonTreeLeafQuery { /// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or /// `json_tree_subtree_at`), collected: opens to the value under the /// walked node, or to the list of complete subtrees the rows hold. +/// +/// Past its budget a walk returns the sealed root alone, at depth `0`; +/// those rows open to the whole document, and the path the walk was keyed +/// from selects the same value(s) the walked rows open to. Attach it with +/// [`below`](Self::below) for `json_tree_subtree` rows and +/// [`at`](Self::at) for `json_tree_subtree_at` rows. The path is not part +/// of the wire. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(transparent)] -pub struct JsonTreeSubtree(pub Vec); +pub struct JsonTreeSubtree { + pub rows: Vec, + #[serde(skip)] + pub path: Option, + /// Whether `path` names the node `json_tree_subtree` walked below + /// ([`below`](Self::below)), rather than the nodes a + /// `json_tree_subtree_at` path reaches ([`at`](Self::at)). + #[serde(skip)] + pub below: bool, +} + +impl JsonTreeSubtree { + /// The rows, as returned. + pub fn new(rows: Vec) -> Self { + Self { + rows, + path: None, + below: false, + } + } + + /// The rows of `json_tree_subtree_at`, with the whole path its key was + /// derived from: past the budget they open to one value per node the + /// path reaches, as the walked rows do. + pub fn at(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: false, + ..self + } + } + + /// The rows of `json_tree_subtree`, with the path of the node whose + /// token keyed the walk: past the budget they open to that node, or as + /// a list to its children, as the walked rows do. + pub fn below(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: true, + ..self + } + } +} macro_rules! json_tree_domain_type { ($ty:ident, $sql:literal) => { @@ -335,12 +390,17 @@ impl DecryptInto> for JsonTreeSubtree { impl JsonTreeSubtree { fn subtree(&self) -> Result { - Ok(JsonSubtree::new( - self.0 + let rows = JsonSubtree::new( + self.rows .iter() .map(JsonTreeEntry::node) .collect::>()?, - )) + ); + Ok(match &self.path { + Some(path) if self.below => rows.below(path.clone()), + Some(path) => rows.at(path.clone()), + None => rows, + }) } } @@ -382,6 +442,45 @@ mod tests { .is_err()); } + #[test] + fn a_document_carries_its_projection() { + let document = serde_json::json!({ + "v": 4, "i": {"t": "users", "c": "profile"}, "h": "AA==", + "sv": [{"s": "aa", "vs": "bb", "c": "AA=="}], + "q": {"sv": [{"s": "aa", "vs": "bb"}], "el": []}, + }); + let stored: JsonTreeDocument = serde_json::from_value(document.clone()).unwrap(); + assert_eq!(stored.q.sv.len(), 1); + assert_eq!(serde_json::to_value(&stored).unwrap(), document); + let mut without = document.clone(); + let _ = without.as_object_mut().unwrap().remove("q"); + assert!( + serde_json::from_value::(without).is_err(), + "a document without `q` is not a stored document" + ); + } + + #[test] + fn a_subtree_carries_its_path_beside_the_wire() { + let rows: JsonTreeSubtree = serde_json::from_value(serde_json::json!([])).unwrap(); + assert!(rows.path.is_none()); + let at = rows.at(JsonPath::parse("$.a").unwrap()); + assert_eq!( + at.path.as_ref().map(ToString::to_string).as_deref(), + Some("$.a") + ); + assert_eq!(serde_json::to_value(&at).unwrap(), serde_json::json!([])); + assert!(at.subtree().unwrap().path().is_some()); + assert!(!at.below); + let below = at.below(JsonPath::parse("$.b").unwrap()); + assert!(below.below); + assert_eq!( + below.subtree().unwrap(), + JsonSubtree::new(Vec::new()).below(JsonPath::parse("$.b").unwrap()) + ); + assert_eq!(serde_json::to_value(&below).unwrap(), serde_json::json!([])); + } + #[test] fn an_entry_without_the_grafted_fields_does_not_open() { let entry: JsonTreeEntry = serde_json::from_value(serde_json::json!({ diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index 7c7dca98e..3b35559fb 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -110,7 +110,15 @@ AS $$ $$; --! @brief How many ordinals a walk tries under each node: one more than the ---! largest ordinal any entry of the document carries. +--! largest ordinal any entry of the document carries, and never more +--! than the document has entries. +--! +--! A node's children sit at ordinals below its child count, which is below +--! the document's entry count, so the cap changes nothing for a sealed +--! document. It is there for a stored one: `o` is not authenticated, and +--! without the cap one entry carrying `o = 2147483646` would make every +--! level of every walk hash two billion candidate tags. +--! --! @internal --! @param val jsonb The document. --! @return integer The fan-out bound; `0` for a document with no chained @@ -119,10 +127,44 @@ CREATE FUNCTION {{prefix}}_internal.json_tree_fanout(val jsonb) RETURNS integer LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT COALESCE(MAX((elem ->> 'o')::integer), -1) + 1 + SELECT LEAST(COALESCE(MAX((elem ->> 'o')::integer), -1) + 1, count(*)::integer) FROM jsonb_array_elements(val -> 'sv') AS elem $$; +--! @brief Whether the document's root is sealed: its header's sealed-root +--! flag, bit 0 of the flags byte at offset 1 of the frozen layout. +--! A sealed root's `c` is the whole document, so a walk may return +--! the root in place of a subtree it would be cheaper to open whole. +--! @internal +--! @param h text The document's header, base64. +--! @return boolean True when the root is sealed. +CREATE FUNCTION {{prefix}}_internal.json_tree_sealed_root(h text) + RETURNS boolean + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT (get_byte(decode(h, 'base64'), 1) & 1) = 1 +$$; + +--! @brief The chained entries of a document keyed by parent tag: one object +--! whose keys are each entry's `p` and whose values are the entry +--! with its ordinal `j` in `sv` added. A walk looks a child up by +--! the tag it computes, `map -> tag`, instead of scanning `sv` for +--! it, so the document is read once per walk and a level costs its +--! fan-out in hashes, not its fan-out times the document's size. +--! @internal +--! @param val jsonb The document. +--! @return jsonb The map; `{}` for a document with no chained entry. +CREATE FUNCTION {{prefix}}_internal.json_tree_by_parent(val jsonb) + RETURNS jsonb + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT COALESCE( + jsonb_object_agg(elem ->> 'p', elem || jsonb_build_object('j', ord - 1)), + '{}'::jsonb) + FROM jsonb_array_elements(val -> 'sv') WITH ORDINALITY AS t(elem, ord) + WHERE elem ? 'p' +$$; + ------------------------------------------------------------------------------ -- Envelope helpers ------------------------------------------------------------------------------ @@ -183,7 +225,15 @@ $$; COMMENT ON FUNCTION {{prefix}}.json_tree_path_query(jsonb, text) IS 'eql-inline-critical: raw-jsonb path query helper; must stay inlinable (unpinned search_path)'; ---! @brief The first entry whose path selector is `selector`, or NULL. +--! @brief The first entry whose path selector is `selector`, or NULL: what +--! `->` returns, and what the ordering comparisons extract. +--! +--! Costed at 1000 (ten times the default): the body scans `sv`, and its +--! cost is what decides whether a comparison on it, `(col -> sel) > $1`, +--! takes the functional btree on `{{prefix}}.ord_term(col -> sel)` or filters +--! a sequential scan. Under a prepared statement's generic plan, where the +--! operand's selectivity is a guess, the default cost chose the scan. +--! --! @param val jsonb The document. --! @param selector text The path selector, hex. --! @return public.{{prefix}}_json_tree_entry The entry, or NULL when the document @@ -191,6 +241,7 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_query(jsonb, text) IS CREATE FUNCTION {{prefix}}.json_tree_path_query_first(val jsonb, selector text) RETURNS public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + COST 1000 AS $$ SELECT {{prefix}}_internal.json_tree_graft(val, elem, (ord - 1)::integer) FROM jsonb_array_elements(val -> 'sv') WITH ORDINALITY AS t(elem, ord) @@ -223,67 +274,54 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_exists(jsonb, text) IS -- Containment: the element-nested projection ------------------------------------------------------------------------------ ---! @brief One group of a stored document's projection, from its group tags. ---! @internal ---! ---! An entry belongs to the group of its own tag `n` when it is an array ---! element that is a container, else to the group of its enclosing element's ---! tag `g`, else to the top level. A group's `el` holds one nested group per ---! container element whose `g` is the group's tag. The needle is projected ---! the same way by the client, from its structure, so `jsonb @>` over the two ---! is PostgreSQL's own containment rule: each needle element has to fit ---! inside one stored element. +--! @brief The stored document's containment projection: the needle shape +--! `{{prefix}}.query_json_tree` holds, as the client wrote it in `q`. --! ---! @param sv jsonb The document's `sv` array. ---! @param parent text The group's tag, hex, or NULL for the top level. ---! @return jsonb `{"sv": [{"s", "vs"}, …], "el": [group, …]}`. ---! @note plpgsql: recursive. IMMUTABLE and deterministic (document order), so ---! `{{prefix}}.json_tree_query` over it serves an expression index. -CREATE FUNCTION {{prefix}}_internal.json_tree_group(sv jsonb, parent text) - RETURNS jsonb - LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE -AS $$ -DECLARE - pairs jsonb; - groups jsonb; -BEGIN - SELECT COALESCE(jsonb_agg(jsonb_build_object('s', elem -> 's', 'vs', elem -> 'vs') ORDER BY ord), '[]'::jsonb) - INTO pairs - FROM jsonb_array_elements(sv) WITH ORDINALITY AS t(elem, ord) - WHERE COALESCE(elem ->> 'n', elem ->> 'g') IS NOT DISTINCT FROM parent; - SELECT COALESCE(jsonb_agg({{prefix}}_internal.json_tree_group(sv, elem ->> 'n') ORDER BY ord), '[]'::jsonb) - INTO groups - FROM jsonb_array_elements(sv) WITH ORDINALITY AS t(elem, ord) - WHERE elem ? 'n' AND (elem ->> 'g') IS NOT DISTINCT FROM parent; - RETURN jsonb_build_object('sv', pairs, 'el', groups); -END; -$$; - ---! @brief Project a stored document into its containment shape: the needle ---! shape `{{prefix}}.query_json_tree` holds, from the document's group ---! tags. +--! The projection is the document's `{s, vs}` pairs nested by array +--! element, `{"sv": [{"s", "vs"}, …], "el": [group, …]}`: the pairs outside +--! any array element at the top, and one nested group per array element +--! that is a container, so that `jsonb @>` over a stored projection and a +--! needle is PostgreSQL's own containment rule, each needle element inside +--! one stored element. The client derives it from the document it seals +--! and stores it beside `sv`; the server reads it and never rebuilds it, +--! so a containment recheck costs a field read, not a pass over `sv`. --! --! This is the GIN index expression for containment: ---! `CREATE INDEX … USING gin (({{prefix}}.json_tree_query(col)::jsonb) jsonb_path_ops);` +--! `CREATE INDEX … USING gin (({{prefix}}.json_tree_query(col)) jsonb_path_ops);` --! The typed `@>` inlines to native `jsonb @>` over the same expression, so --! the index engages; GIN is lossy and the executor rechecks the real --! operator on the heap tuple, which is the exact containment. --! --! @param e public.{{prefix}}_json_tree The stored document. ---! @return {{prefix}}.query_json_tree Its projection. +--! @return jsonb Its projection, the shape of `{{prefix}}.query_json_tree`. +--! @note Returns raw jsonb rather than the needle domain: a domain cast +--! runs the needle's recursive CHECK, and the recheck would pay it +--! once per candidate row. The document's own CHECK validated `q` +--! when the row was written. +--! @note plpgsql, deliberately, and costed at 1000: a SQL body this simple +--! would be inlined to `value -> 'q'`, which the planner costs as one +--! operator, and a column of large documents is TOASTed out of the +--! heap, so a sequential scan over it looks almost free on paper and +--! is chosen over the GIN index. It is not free: every row is +--! decompressed to read `q`. Un-inlined, the call keeps its cost, the +--! index expression still matches it, and the planner takes the +--! index. The call itself is a few microseconds per recheck row. CREATE FUNCTION {{prefix}}.json_tree_query(e public.{{prefix}}_json_tree) - RETURNS {{prefix}}.query_json_tree - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + RETURNS jsonb + LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE + COST 1000 AS $$ - SELECT {{prefix}}_internal.json_tree_group(e::jsonb -> 'sv', NULL)::{{prefix}}.query_json_tree +BEGIN + RETURN e::jsonb -> 'q'; +END; $$; --! @brief Does the stored document `a` contain the needle `b`? --! --! Native `jsonb @>` over the projections, plus the needle's `alt` projection --! when it carries one (a root scalar against a stored root array). Both ---! probes are over `{{prefix}}.json_tree_query(a)::jsonb`, so a GIN index on ---! that expression serves them as one bitmap OR. +--! probes are over `{{prefix}}.json_tree_query(a)`, so a GIN index on that +--! expression serves them as one bitmap OR. --! --! @param a public.{{prefix}}_json_tree The stored document. --! @param b {{prefix}}.query_json_tree The needle. @@ -299,8 +337,8 @@ CREATE FUNCTION {{prefix}}.json_tree_contains(a public.{{prefix}}_json_tree, b { RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ - SELECT {{prefix}}.json_tree_query(a)::jsonb @> (b::jsonb - 'alt') - OR ((b::jsonb ? 'alt') AND {{prefix}}.json_tree_query(a)::jsonb @> (b::jsonb -> 'alt')) + SELECT {{prefix}}.json_tree_query(a) @> (b::jsonb - 'alt') + OR ((b::jsonb ? 'alt') AND {{prefix}}.json_tree_query(a) @> (b::jsonb -> 'alt')) $$; --! @brief Is the stored document `a` contained by the needle `b`? @@ -316,7 +354,7 @@ CREATE FUNCTION {{prefix}}.json_tree_contained_by(a public.{{prefix}}_json_tree, RETURNS boolean LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT (b::jsonb - 'alt') @> {{prefix}}.json_tree_query(a)::jsonb + SELECT (b::jsonb - 'alt') @> {{prefix}}.json_tree_query(a) $$; --! @brief Does the stored document `a` contain the stored document `b`? @@ -327,20 +365,27 @@ CREATE FUNCTION {{prefix}}.json_tree_document_contains(a public.{{prefix}}_json_ RETURNS boolean LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}.json_tree_query(a)::jsonb @> {{prefix}}.json_tree_query(b)::jsonb + SELECT {{prefix}}.json_tree_query(a) @> {{prefix}}.json_tree_query(b) $$; ------------------------------------------------------------------------------ -- The token chain: children and subtrees by token ------------------------------------------------------------------------------ +--! The walk, in every function below: for the node in hand, compute the +--! parent tag a child at each ordinal under the document's fan-out would +--! carry, and look each tag up in the document's parent map +--! ({{prefix}}_internal.json_tree_by_parent). A hit is a child; its token +--! unwraps under the parent's and the walk recurses. The map and the +--! document's header fields are computed once per call, so a level costs +--! its fan-out in SHA-256 evaluations and one lookup each, and the +--! document is never scanned per node. + --! @brief The children of the node whose token is `token`, as rows. --! ---! For every ordinal under the document's fan-out, the parent tag the child ---! at that ordinal would carry; the entries carrying one are the children, ---! and each row's token is unwrapped on the way, though not returned. What ---! `jsonb_array_elements` is for an encrypted array: one row per element, in ---! ordinal order, each opening on its own. +--! What `jsonb_array_elements` is for an encrypted array: one row per +--! element, in ordinal order, each opening on its own. Each child's token +--! is unwrapped on the way, though not returned. --! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The parent's token, hex: derived by the client from the @@ -350,14 +395,20 @@ CREATE FUNCTION {{prefix}}.json_tree_children(e public.{{prefix}}_json_tree, tok RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT ({{prefix}}_internal.json_tree_graft(e::jsonb, c.elem, (c.ord - 1)::integer)::jsonb || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry - FROM generate_series(0, {{prefix}}_internal.json_tree_fanout(e::jsonb) - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(e::jsonb -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag( - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h'), decode(token, 'hex'), o) - ) AS c ON true + WITH + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta + ) + SELECT (doc.meta || c.entry || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry + FROM doc + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, decode(token, 'hex'), o) AS entry + ) AS c + WHERE c.entry IS NOT NULL ORDER BY o $$; @@ -384,50 +435,97 @@ $$; --! and the client reads the node's kind from its children's labels, or --! fetches the node by its selector. --! The walk descends at most 127 levels, the deepest a sealed document ---! nests, so a stored document whose tags do not form a tree still ends it. +--! nests, and returns at most as many rows as the document has entries, +--! which a tree never exceeds. A stored document whose tags form a longer +--! chain or a loop therefore still ends it, after at most its entry count +--! times its fan-out in hashes. +--! +--! **The budget.** A walk costs the fan-out in hashes per node it visits, +--! and returns about four hundred characters per entry, where a sealed root +--! holds the whole document in one ciphertext of about the plaintext's +--! size. So when the root is sealed and the subtree has more than `budget` +--! entries, the walk stops and returns the root entry alone, with `d = 0`: +--! the client opens it to the whole document and takes the path itself, +--! which is cheaper on both sides than the walk past that point. The +--! default of 64 is past the crossover by bytes (about a dozen entries) by +--! a margin that keeps a small subtree arriving as the entries it is. A +--! `budget` of `0` never falls back. A token that names no node yields no +--! row under any budget, and a document whose root is not sealed is always +--! walked in full, since its root holds no document to return. --! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The subtree's token, hex. +--! @param budget integer The most entries to return before falling back to +--! the sealed root; `0` to always walk the subtree out. Default 64. --! @return SETOF public.{{prefix}}_json_tree_entry The descendants, each with its ---! depth `d` below the node, in depth-first order. -CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text) +--! depth `d` below the node, in depth-first order; or the root +--! entry alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ WITH RECURSIVE - doc AS ( - SELECT e::jsonb AS val, - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta, + e::jsonb -> 'sv' -> 0 AS root, + jsonb_array_length(e::jsonb -> 'sv') AS entries, + budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), - walk(depth, t, j, entry, path) AS ( - SELECT 0, decode(token, 'hex'), NULL::integer, NULL::jsonb, ARRAY[]::integer[] + walk(depth, t, entry) AS ( + SELECT 0, decode(token, 'hex'), NULL::jsonb UNION ALL SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.elem), - (c.ord - 1)::integer, - c.elem, - w.path || (c.ord - 1)::integer + {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), + c.entry FROM walk AS w CROSS JOIN doc - JOIN LATERAL ( - SELECT elem, ord - FROM generate_series(0, doc.fanout - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(doc.val -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) - ) AS hit ON true - ) AS c ON true + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry + ) AS c -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a cycle or a longer chain, which whoever holds a - -- token can write, stops here instead of recursing without end. - WHERE w.depth < 127 + -- whose tags form a longer chain, which whoever holds a token can + -- write, stops here. A cycle that branches is stopped by `visited`. + WHERE c.entry IS NOT NULL AND w.depth < 127 + ), + -- A walk of a sealed document reaches each entry at most once, so it + -- has at most as many rows as the document has entries, the seed row + -- standing in for the root. A stored document can loop instead: `w` + -- is not authenticated, so whoever holds a token can wrap a child's + -- token to its parent's, and two such children double the rows at + -- every level the depth bound allows. PostgreSQL computes a recursive + -- query only as far as its reader pulls rows, so this limit stops the + -- recursion, and not only the output. + visited AS MATERIALIZED ( + SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) + ), + -- One row past the budget is enough to know the walk went over; a walk + -- under it is the subtree complete. Uncapped, the limit is NULL: all. + found AS MATERIALIZED ( + SELECT w.depth, w.entry + FROM visited AS w + WHERE w.entry IS NOT NULL + LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) + ), + over AS MATERIALIZED ( + SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc ) - SELECT ({{prefix}}_internal.json_tree_graft(doc.val, w.entry, w.j)::jsonb || jsonb_build_object('d', w.depth))::public.{{prefix}}_json_tree_entry - FROM walk AS w CROSS JOIN doc - WHERE w.entry IS NOT NULL - ORDER BY w.path + SELECT rows.entry + FROM ( + SELECT 0 AS j, + (doc.meta || doc.root || '{"j": 0, "d": 0}'::jsonb)::public.{{prefix}}_json_tree_entry AS entry + FROM doc, over + WHERE over.fallback + UNION ALL + SELECT (f.entry ->> 'j')::integer, + (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry + FROM found AS f, doc, over + WHERE NOT over.fallback + ) AS rows + ORDER BY rows.j $$; --! @brief The subtrees under the node whose token is `token` whose path @@ -437,52 +535,83 @@ $$; --! The walk starts at the token and emits an entry once it has passed --! through, or reached, an entry carrying the selector: for --! `$.orders[*].items` it starts at `$.orders` and returns every element's ---! `items` subtree, each complete. +--! `items` subtree, each complete. The budget is `json_tree_subtree`'s, +--! counted over the entries emitted: past it, the sealed root alone, with +--! `d = 0`, and the client takes the wildcard path over the whole document. --! --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The anchor's token, hex. --! @param selector text The whole path's selector, hex. +--! @param budget integer The most entries to return before falling back to +--! the sealed root; `0` to always walk out. Default 64. --! @return SETOF public.{{prefix}}_json_tree_entry The matching subtrees' entries, ---! each with its depth `d` below the anchor, in depth-first order. -CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, selector text) +--! each with its depth `d` below the anchor, in depth-first order; +--! or the root entry alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, selector text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ WITH RECURSIVE - doc AS ( - SELECT e::jsonb AS val, - {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout + doc AS MATERIALIZED ( + SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, + {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, + {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, + {{prefix}}.json_tree_meta(e::jsonb) AS meta, + e::jsonb -> 'sv' -> 0 AS root, + jsonb_array_length(e::jsonb -> 'sv') AS entries, + budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped ), - walk(depth, t, j, entry, path, hit) AS ( - SELECT 0, decode(token, 'hex'), NULL::integer, NULL::jsonb, ARRAY[]::integer[], false + walk(depth, t, entry, hit) AS ( + SELECT 0, decode(token, 'hex'), NULL::jsonb, false UNION ALL SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.elem), - (c.ord - 1)::integer, - c.elem, - w.path || (c.ord - 1)::integer, - w.hit OR c.elem ->> 's' = selector + {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), + c.entry, + w.hit OR c.entry ->> 's' = selector FROM walk AS w CROSS JOIN doc - JOIN LATERAL ( - SELECT elem, ord - FROM generate_series(0, doc.fanout - 1) AS o - JOIN LATERAL ( - SELECT elem, ord - FROM jsonb_array_elements(doc.val -> 'sv') WITH ORDINALITY AS t(elem, ord) - WHERE elem ->> 'p' = {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) - ) AS hit ON true - ) AS c ON true + CROSS JOIN generate_series(0, doc.fanout - 1) AS o + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry + ) AS c -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a cycle or a longer chain, which whoever holds a - -- token can write, stops here instead of recursing without end. - WHERE w.depth < 127 + -- whose tags form a longer chain, which whoever holds a token can + -- write, stops here. A cycle that branches is stopped by `visited`. + WHERE c.entry IS NOT NULL AND w.depth < 127 + ), + -- A walk of a sealed document reaches each entry at most once, so it + -- has at most as many rows as the document has entries, the seed row + -- standing in for the root. A stored document can loop instead: `w` + -- is not authenticated, so whoever holds a token can wrap a child's + -- token to its parent's, and two such children double the rows at + -- every level the depth bound allows. PostgreSQL computes a recursive + -- query only as far as its reader pulls rows, so this limit stops the + -- recursion, and not only the output. + visited AS MATERIALIZED ( + SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) + ), + found AS MATERIALIZED ( + SELECT w.depth, w.entry + FROM visited AS w + WHERE w.entry IS NOT NULL AND w.hit + LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) + ), + over AS MATERIALIZED ( + SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc ) - SELECT ({{prefix}}_internal.json_tree_graft(doc.val, w.entry, w.j)::jsonb || jsonb_build_object('d', w.depth))::public.{{prefix}}_json_tree_entry - FROM walk AS w CROSS JOIN doc - WHERE w.entry IS NOT NULL AND w.hit - ORDER BY w.path + SELECT rows.entry + FROM ( + SELECT 0 AS j, + (doc.meta || doc.root || '{"j": 0, "d": 0}'::jsonb)::public.{{prefix}}_json_tree_entry AS entry + FROM doc, over + WHERE over.fallback + UNION ALL + SELECT (f.entry ->> 'j')::integer, + (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry + FROM found AS f, doc, over + WHERE NOT over.fallback + ) AS rows + ORDER BY rows.j $$; ------------------------------------------------------------------------------ diff --git a/packages/eql/src/v3/json_tree/operators.sql b/packages/eql/src/v3/json_tree/operators.sql index e4c75d78c..0d81b48b0 100644 --- a/packages/eql/src/v3/json_tree/operators.sql +++ b/packages/eql/src/v3/json_tree/operators.sql @@ -63,7 +63,7 @@ CREATE OPERATOR ->> ( --! @brief @> contains operator with a query_json_tree needle. --! ---! Inlines to native `jsonb @>` over `{{prefix}}.json_tree_query(a)::jsonb`, so +--! Inlines to native `jsonb @>` over `{{prefix}}.json_tree_query(a)`, so --! a functional GIN index on that expression engages. Not STRICT, for the --! reason `{{prefix}}.json_tree_contains` gives: a STRICT wrapper around a --! non-STRICT body is not inlined. diff --git a/packages/eql/src/v3/json_tree/types.sql b/packages/eql/src/v3/json_tree/types.sql index abedea647..108477f4d 100644 --- a/packages/eql/src/v3/json_tree/types.sql +++ b/packages/eql/src/v3/json_tree/types.sql @@ -10,7 +10,7 @@ --! Stack Encrypt's JSON index writes. It shares no wire shape and no SQL with --! the SteVec surface. Four jsonb-backed domains (none over another domain, --! so the native-jsonb firewall in blockers.sql attaches): ---! - public.{{prefix}}_json_tree — storage: an EQL envelope `{v, i, h, sv}`. +--! - public.{{prefix}}_json_tree — storage: an EQL envelope `{v, i, h, sv, q}`. --! - public.{{prefix}}_json_tree_entry — one `sv` element (returned by `->` --! and the subtree walk). --! - {{prefix}}.query_json_tree — a containment needle: the element-nested @@ -34,8 +34,16 @@ --! `g` the group tag of the nearest enclosing such element. --! `op` the leaf ordering term (CLLW OPE, hex) on number and string leaves. --! The header `h` (base64) carries the document's data key material once; the ---! SQL reads only the ZeroKMS `iv` at byte offset 22 of it, the per-document ---! salt of the chain tags, and otherwise carries it opaquely. +--! SQL reads only the sealed-root flag at byte offset 1 and the ZeroKMS `iv` +--! at byte offset 22 of it, the per-document salt of the chain tags, and +--! otherwise carries it opaquely. `q` is the document's containment +--! projection, `{"sv": [{s, vs}, …], "el": [group, …]}`: its `{s, vs}` +--! pairs nested by array element, written by the client beside `sv` so the +--! server compares it with a needle by native `jsonb @>` and never +--! rebuilds it from the entries. The CHECK validates its shape, not that it +--! is `sv`'s: containment answers from `q` as written, so a row written by +--! anything but Stack Encrypt's JSON index can answer `@>` differently +--! from `->`, the walks and decryption, which answer from `sv`. --! @brief Validate one JSON tree entry. --! @internal @@ -68,42 +76,15 @@ AS $$ ) $$; ---! @brief Validate a JSON tree document. ---! @internal ---! @param val jsonb Candidate document. ---! @return boolean True when `val` is an envelope with `v = {{eql_version}}`, `i`, a ---! string header `h` and a non-empty `sv` array of valid entries, and ---! no other key. A document always has a root, so `sv` is never empty. -CREATE OR REPLACE FUNCTION public.{{prefix}}_is_valid_json_tree_document_payload(val jsonb) - RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT COALESCE( - jsonb_typeof(val) = 'object' - AND val ? 'v' - AND val ->> 'v' = '{{eql_version}}' - AND val ? 'i' - AND jsonb_typeof(val -> 'h') = 'string' - AND jsonb_typeof(val -> 'sv') = 'array' - AND jsonb_array_length(val -> 'sv') >= 1 - AND val - ARRAY['v', 'i', 'h', 'sv']::text[] = '{}'::jsonb - AND NOT EXISTS ( - SELECT 1 - FROM jsonb_array_elements(val -> 'sv') AS elem - WHERE NOT public.{{prefix}}_is_valid_json_tree_entry_payload(elem) - ), - false - ) -$$; - --! @brief Validate one group of a containment needle, recursively. --! @internal --! --! A group is `{"sv": [{"s": "…", "vs": "…"}, …], "el": [group, …]}`: the --! selector pairs that sit directly in it, and one nested group per array ---! element that is a container. The same shape `{{prefix}}.json_tree_query` ---! projects a stored document into, so native `jsonb @>` over the two is ---! exact containment. +--! element that is a container. The same shape a stored document carries +--! in `q`, its own projection, so native `jsonb @>` over the two is exact +--! containment. Validates both: a needle at its cast, a document's `q` at +--! its write. --! --! @param val jsonb Candidate group. --! @return boolean True when `val` is a group. @@ -138,6 +119,36 @@ BEGIN END; $$; +--! @brief Validate a JSON tree document. +--! @internal +--! @param val jsonb Candidate document. +--! @return boolean True when `val` is an envelope with `v = {{eql_version}}`, `i`, a +--! string header `h`, a non-empty `sv` array of valid entries and +--! the document's containment projection `q`, a group, and no +--! other key. A document always has a root, so `sv` is never empty. +CREATE OR REPLACE FUNCTION public.{{prefix}}_is_valid_json_tree_document_payload(val jsonb) + RETURNS boolean + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT COALESCE( + jsonb_typeof(val) = 'object' + AND val ? 'v' + AND val ->> 'v' = '{{eql_version}}' + AND val ? 'i' + AND jsonb_typeof(val -> 'h') = 'string' + AND jsonb_typeof(val -> 'sv') = 'array' + AND jsonb_array_length(val -> 'sv') >= 1 + AND val - ARRAY['v', 'i', 'h', 'sv', 'q']::text[] = '{}'::jsonb + AND NOT EXISTS ( + SELECT 1 + FROM jsonb_array_elements(val -> 'sv') AS elem + WHERE NOT public.{{prefix}}_is_valid_json_tree_entry_payload(elem) + ) + AND {{prefix}}_internal.json_tree_is_valid_query_group(val -> 'q'), + false + ) +$$; + --! @brief Validate a containment needle. --! @internal --! @param val jsonb Candidate needle. @@ -162,11 +173,12 @@ $$; --! @brief Storage domain for an encrypted JSON tree column. --! --! CHECK: the EQL envelope (`v = {{eql_version}}`, `i`, `h`) around a non-empty `sv` ---! array of valid entries. The firewall in blockers.sql attaches to this ---! domain to stop native jsonb operators from reaching a column value. +--! array of valid entries and the projection `q`. The firewall in +--! blockers.sql attaches to this domain to stop native jsonb operators from +--! reaching a column value. --! --! @note Constructing from inline JSON uses the standard DOMAIN cast: ---! `'{"v":{{eql_version}},"i":{…},"h":"…","sv":[…]}'::public.{{prefix}}_json_tree`. +--! `'{"v":{{eql_version}},"i":{…},"h":"…","sv":[…],"q":{…}}'::public.{{prefix}}_json_tree`. DO $$ BEGIN IF NOT EXISTS ( @@ -212,7 +224,8 @@ $$; --! --! The element-nested projection of the needle's selectors, as Stack --! Encrypt's `JsonQuery` serializes it: `{"sv": [{"s", "vs"}, …], "el": ---! [group, …], "alt": group?}`. Typing the needle this way stops raw jsonb +--! [group, …], "alt": group?}`: the shape a stored document's `q` has, with +--! the optional `alt` beside it. Typing the needle this way stops raw jsonb --! from casting and matching every row via bare `jsonb @>`. --! --! @note Construct from inline JSON via the DOMAIN cast: diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index dd7484643..0250b449d 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -76,7 +76,7 @@ async fn a_document_round_trips_through_the_typed_target() { .keys() .map(String::as_str) .collect(); - assert_eq!(keys, ["h", "i", "sv", "v"]); + assert_eq!(keys, ["h", "i", "q", "sv", "v"]); let opened: Value = cipher .decrypt_as(stored.clone(), Default::default()) @@ -306,7 +306,7 @@ fn postgres_containment_over_the_encrypted_column_is_jsonb_containment() { let mut db = connect(); db.batch_execute( "CREATE TEMP TABLE json_tree_docs (id int PRIMARY KEY, plain jsonb NOT NULL, doc public.eql_v4_json_tree NOT NULL); - CREATE INDEX json_tree_docs_doc ON json_tree_docs USING gin ((eql_v4.json_tree_query(doc)::jsonb) jsonb_path_ops);", + CREATE INDEX json_tree_docs_doc ON json_tree_docs USING gin ((eql_v4.json_tree_query(doc)) jsonb_path_ops);", ) .unwrap(); @@ -541,7 +541,7 @@ fn postgres_extracts_walks_and_orders_by_path() { .all(|e| e.d.is_some() && e.j.is_some() && e.h.is_some())); let value: Value = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), ) .unwrap(); assert_eq!(value, expected, "row {id}"); @@ -555,6 +555,100 @@ fn postgres_extracts_walks_and_orders_by_path() { .get(0); assert_eq!(garbage, 0, "a wrong token unlocks nothing in any row"); + // The budget: past it the walk returns the sealed root alone, at depth + // 0, and the rows open to the value at the walked path; at it, or + // with the budget off, the walk is complete; a wrong token still + // yields nothing under any budget. + let walked = |db: &mut postgres::Client, budget: i32| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", + &[&nested.token(), &budget], + ) + .unwrap() + .iter() + .map(entry) + .collect() + }; + assert_eq!(walked(&mut db, 6).len(), 6, "at the budget, the subtree"); + assert_eq!(walked(&mut db, 0).len(), 6, "budget 0 never falls back"); + let root = walked(&mut db, 5); + assert_eq!(root.len(), 1, "past the budget, the root alone"); + assert_eq!((root[0].j, root[0].d), (Some(0), Some(0))); + let refused = wait( + &runtime, + cipher.decrypt_as::(JsonTreeSubtree::new(root.clone()), column()), + ); + assert!(refused.is_err(), "the fallback needs the walked path"); + let value: Value = wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(root).below(JsonPath::parse("$.nested").unwrap()), + column(), + ), + ) + .unwrap(); + assert_eq!(value, fixture()["nested"]); + let garbage: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, 1) e", + &[&"00".repeat(16)], + ) + .unwrap() + .get(0); + assert_eq!(garbage, 0); + // The fallback opens to what the walk opens to, as a list too: the + // list's elements, whether the server walked its four entries or + // returned the sealed root, so a caller's result does not change shape + // when a document grows past the budget. + let list = subtree_key("$.nested.list"); + let list_rows = |db: &mut postgres::Client, budget: i32| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", + &[&list.token(), &budget], + ) + .unwrap() + .iter() + .map(entry) + .collect() + }; + let elements = |rows: Vec| -> Vec { + wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(rows).below(JsonPath::parse("$.nested.list").unwrap()), + column(), + ), + ) + .unwrap() + }; + let walked_list = list_rows(&mut db, 0); + assert_eq!(walked_list.len(), 4); + let fallen_back = list_rows(&mut db, 3); + assert_eq!(fallen_back.len(), 1); + assert_eq!( + elements(walked_list), + vec![json!({"x": 1}), json!({"x": 2})] + ); + assert_eq!( + elements(fallen_back), + vec![json!({"x": 1}), json!({"x": 2})] + ); + // The budget is gated on the header's sealed-root flag, the bit + // `JsonHeader` writes: with the flags byte cleared, the same document + // has no root to return and is walked out under any budget. + let unflagged: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_rows, + eql_v4.json_tree_subtree( + jsonb_set(doc::jsonb, '{h}', to_jsonb(encode(set_byte(decode(doc::jsonb ->> 'h', 'base64'), 1, 0), 'base64')))::public.eql_v4_json_tree, + $1::text, 1) e + WHERE id = 1", + &[&nested.token()], + ) + .unwrap() + .get(0); + assert_eq!(unflagged, 6, "an unsealed root is walked out"); + // A wildcard below the anchor: every element's `x`, each a complete // subtree, from the list's token gated on the whole path's selector. let xs = subtree_key("$.nested.list[*].x"); @@ -569,7 +663,29 @@ fn postgres_extracts_walks_and_orders_by_path() { .collect(); let values: Vec = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), + ) + .unwrap(); + assert_eq!(values, vec![json!(1), json!(2)]); + // Past the budget, the gated walk falls back the same way, and the + // wildcard path selects every subtree over the whole document. + let root: Vec = db + .query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text, 1) e WHERE id = 1", + &[&xs.token(), &xs.selector().unwrap().to_string()], + ) + .unwrap() + .iter() + .map(entry) + .collect(); + assert_eq!(root.len(), 1); + assert_eq!(root[0].d, Some(0)); + let values: Vec = wait( + &runtime, + cipher.decrypt_as( + JsonTreeSubtree::new(root).at(JsonPath::parse("$.nested.list[*].x").unwrap()), + column(), + ), ) .unwrap(); assert_eq!(values, vec![json!(1), json!(2)]); @@ -701,3 +817,104 @@ fn postgres_extracts_walks_and_orders_by_path() { } } } + +/// Needs an EQL v4 installation. A stored document is untrusted input: its +/// chain fields are not authenticated, so whoever holds a token can write +/// children whose tokens loop back to their parent's, or an ordinal far +/// past the document's size. Every walk still ends, in time bounded by the +/// document's size, under any budget; `statement_timeout` turns a walk +/// that would not end into a failure instead of a hung suite. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_walks_end_on_hostile_documents() { + let mut db = connect(); + db.batch_execute("SET statement_timeout = '10s'").unwrap(); + // A root and two children at ordinals 0 and 1 under the all-zero + // token, each wrapping that same token as its own: unbounded, the walk + // doubles at every one of its 127 levels. `flags` is the header's flags + // byte, so the same document is built with its root sealed and not. + let looping = |flags: i32| { + format!( + "WITH k AS ( + SELECT set_byte(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 1, {flags}) AS h, + decode(repeat('00', 16), 'hex') AS iv, + decode(repeat('00', 16), 'hex') AS t + ), e AS ( + SELECT o, c, + encode(eql_v4_internal.json_tree_h16('p'::bytea || k.iv || k.t || eql_v4_internal.json_tree_le32(o)), 'hex') AS p, + encode(eql_v4_internal.json_tree_xor16(k.t, eql_v4_internal.json_tree_h16('w'::bytea || k.iv || k.t || decode(c, 'base64'))), 'hex') AS w + FROM k, (VALUES (0, 'AA=='), (1, 'AQ==')) AS v(o, c) + ) + SELECT (jsonb_build_object( + 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), + 'h', (SELECT encode(h, 'base64') FROM k), + 'sv', jsonb_build_array(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==')) + || (SELECT jsonb_agg(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', c, 'o', o, 'p', p, 'w', w) ORDER BY o) FROM e), + 'q', '{{\"sv\": [], \"el\": []}}'::jsonb))::public.eql_v4_json_tree" + ) + }; + let token = "00".repeat(16); + for flags in [0, 1] { + let doc = looping(flags); + let count = |db: &mut postgres::Client, walk: &str| -> i64 { + db.query_one( + &format!("SELECT count(*) FROM ({doc}) AS d(doc), {walk} e"), + &[&token], + ) + .unwrap_or_else(|error| panic!("{walk} (flags {flags}): {error}")) + .get(0) + }; + assert_eq!( + count(&mut db, "eql_v4.json_tree_children(d.doc, $1::text)"), + 2 + ); + for budget in [0, 64] { + let walked = count( + &mut db, + &format!("eql_v4.json_tree_subtree(d.doc, $1::text, {budget})"), + ); + assert!( + walked <= 3, + "a walk returns at most as many rows as the document has entries: {walked} (flags {flags}, budget {budget})" + ); + // No entry carries the selector, so nothing is emitted and the + // budget never trips: only the walk's own bound ends it. + assert_eq!( + count( + &mut db, + &format!("eql_v4.json_tree_subtree_at(d.doc, $1::text, 'ff', {budget})"), + ), + 0 + ); + } + } + + // One entry claiming the largest ordinal an integer holds: the fan-out + // is capped at the document's entry count, so the probe is two tags, + // not two billion. + let wide = "SELECT (jsonb_build_object( + 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), + 'h', encode(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 'base64'), + 'sv', jsonb_build_array( + jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA=='), + jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==', 'o', 2147483646, 'p', '00', 'w', '00')), + 'q', '{\"sv\": [], \"el\": []}'::jsonb))::public.eql_v4_json_tree"; + let children: i32 = db + .query_one( + &format!( + "SELECT eql_v4.json_tree_child_count(d.doc, $1::text) FROM ({wide}) AS d(doc)" + ), + &[&token], + ) + .unwrap() + .get(0); + assert_eq!(children, 0); + let walked: i64 = db + .query_one( + &format!("SELECT count(*) FROM ({wide}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, 0) e"), + &[&token], + ) + .unwrap() + .get(0); + assert_eq!(walked, 0); +} diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 0be913577..173780f49 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -28,7 +28,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, `JsonLeafNotOrdered` and `JsonValueNotScalar`. A data plan reads and writes the index (`"json"`, or `{"json": {"sealed_root": bool}}`) and refuses to run it with - `Reason::JsonIndexPending`. + `Reason::JsonIndexPending`. A `JsonDocument` writes its own containment + projection to the wire as `q`, beside `sv`, for the server to compare + with a needle, and ignores a stored one, deriving it from the entries + again. A walk past its budget returns the sealed root in place of the + subtree: `JsonSubtree::below(path)` (for `json_tree_subtree`) or + `JsonSubtree::at(path)` (for `json_tree_subtree_at`) attaches the walked + path, and the rows then open to the same value either way, over + `JsonPath::select`, which evaluates a path over a value. - **A block ORE index, `OreBlock`** (ADR-0002 and ADR-0009, amended 2026-10-19). It derives ore-rs 0.9.0's 6-bit block ORE under its own key diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index b222c694d..c1f82297d 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -1014,11 +1014,30 @@ impl DecryptInto for JsonNode { /// value, in order. What `json_tree_subtree_at` returns is one subtree per /// node whose selector matched; what `json_tree_subtree` returns is the /// children of one node, which [`DecryptInto`] assembles into that -/// node's value. +/// node's value, or this opens into its children's values. The sealed root +/// a walk fell back to opens whole, and the attached path selects over it +/// to the same values the walked rows open to: one per node the path +/// reaches, for a path attached with [`JsonSubtree::at`]; the children of +/// the one node at the path, for [`JsonSubtree::below`]. An object's +/// members come in an order neither form fixes, and without their keys; +/// open it as a [`Value`] to keep them. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { + if self.is_sealed_root() { + let below = self.below; + return open_fallback(self, cipher, context).try_map(move |(path, whole)| { + let found = path.select(&whole); + if !below { + return Ok(found.into_iter().cloned().collect()); + } + match found.as_slice() { + [node] => Ok(children(node)), + _ => Err(shape("one node at the walked path")), + } + }); + } open_subtree(self, cipher, context) .map(|forest| forest.into_iter().map(|(_, v)| v).collect()) }) @@ -1029,11 +1048,24 @@ impl DecryptInto> for JsonSubtree { /// that token: an array when the rows are elements, an object when they are /// members. No rows is no value ([`Error::OpenedType`]): an empty container /// and a scalar both have no descendants, and `->` on the node's selector -/// returns the node itself. +/// returns the node itself. The sealed root a walk fell back to opens +/// whole, and the path attached with [`JsonSubtree::below`] (or +/// [`JsonSubtree::at`]) selects the one value under it; a path that +/// reaches no node, or more than one, is [`Error::OpenedType`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { Decryption::from_open(move |cipher| { + if self.is_sealed_root() { + return open_fallback(self, cipher, context).try_map(|(path, whole)| { + let mut found = path.select(&whole); + match found.len() { + 1 => Ok(found.remove(0).clone()), + 0 => Err(shape("a node at the walked path")), + _ => Err(shape("one node at the walked path")), + } + }); + } open_subtree(self, cipher, context).try_map(|forest| { let mut members = Map::new(); let mut elements: Vec<(u64, Value)> = Vec::new(); @@ -1102,6 +1134,48 @@ fn open_subtree<'a, K>( }) } +/// The values of a node's children, in the node's order: an array's +/// elements, an object's member values, and none for a scalar. What a +/// `json_tree_subtree` walk's rows open to as a [`Vec`], taken over the +/// whole document when the walk fell back to the sealed root. +fn children(node: &Value) -> Vec { + match node { + Value::Array(items) => items.clone(), + Value::Object(members) => members.values().cloned().collect(), + _ => Vec::new(), + } +} + +/// Open the sealed root a walk fell back to: the whole document, with the +/// path the rows were keyed from, which the caller selects over. Refused +/// without the path, since nothing in the rows records it. +fn open_fallback<'a, K>( + subtree: JsonSubtree, + cipher: &'a StackCipher, + context: AeadContext, +) -> Pending<'a, (JsonPath, Value), K> { + let Some(path) = subtree.path().cloned() else { + return Pending::failed( + cipher, + shape("the walked path, attached with JsonSubtree::below or JsonSubtree::at, to select over the sealed root the walk returned"), + ); + }; + let Some(root) = subtree.nodes().first().cloned() else { + return Pending::failed(cipher, shape("the sealed root")); + }; + let (request, aad) = match retrieval(&root.header, context) { + Ok(parts) => parts, + Err(error) => return Pending::failed(cipher, error), + }; + Pending::request(cipher, vec![request], move |responses| { + let key = responses.next_retrieved_key()?; + let aes = leaf_cipher(&key)?; + let opened = open_entry(&aes, &root.header, &aad, &root.entry, root.j)?; + let whole = opened.whole.ok_or_else(|| shape("a sealed root"))?; + Ok((path, whole)) + }) +} + // ============================================================================= // Extraction keys, on the keyset cipher // ============================================================================= diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index 96085de76..33c47da7f 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -327,8 +327,10 @@ impl JsonEntry { /// [`DecryptInto`](crate::DecryptInto) opens. /// /// Serializes to the wire form the EQL domain stores: `{"h": "…", "sv": -/// [{"s": "…", "vs": "…", "c": "…", …}, …]}`; see the -/// [module docs](self#byte-encodings). +/// [{"s": "…", "vs": "…", "c": "…", …}, …], "q": {"sv": [{"s", "vs"}, …], +/// "el": […]}}`, where `q` is the document's own containment +/// [projection](Self::projection), written once so the server never +/// rebuilds it; see the [module docs](self#byte-encodings). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(try_from = "DocumentWire", into = "DocumentWire")] pub struct JsonDocument { @@ -494,22 +496,80 @@ where /// members an object. A node with no descendants, an empty container or a /// scalar, yields no rows and so no value; `->` on its selector returns the /// node itself. +/// +/// Past its budget, a walk over a document with a sealed root returns the +/// root entry alone, at depth `0`, instead of a subtree it would be +/// cheaper to open whole: the client opens the document and takes the +/// walked path over it. Opening such rows needs that path and which walk +/// it keyed, attached with [`below`](Self::below) for `json_tree_subtree` +/// or [`at`](Self::at) for `json_tree_subtree_at`, so that the rows open +/// to the same value whichever form the server returned; without a path +/// the opening fails with [`Error::OpenedType`](crate::Error::OpenedType). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(transparent)] pub struct JsonSubtree { nodes: Vec, + #[serde(skip)] + path: Option, + #[serde(skip)] + below: bool, } impl JsonSubtree { /// The rows, as returned. pub fn new(nodes: Vec) -> Self { - Self { nodes } + Self { + nodes, + path: None, + below: false, + } + } + + /// The rows of `json_tree_subtree_at`, with the whole path its key was + /// derived from: what the opening takes over the whole document when + /// the rows are the sealed root the server fell back to. They open as + /// the walked rows do, to one value per node the path reaches. + pub fn at(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: false, + ..self + } + } + + /// The rows of `json_tree_subtree`, with the path of the node whose + /// token keyed the walk. They open as the walked rows do, which are + /// that node's descendants: to the node's value, or, as a + /// [`Vec`], to its children's values. Attach the path this way and not + /// with [`at`](Self::at), which would open the fallback to the node + /// itself where the walk opens to its children. + pub fn below(self, path: JsonPath) -> Self { + Self { + path: Some(path), + below: true, + ..self + } } /// The rows. pub fn nodes(&self) -> &[JsonNode] { &self.nodes } + + /// The path attached with [`at`](Self::at) or [`below`](Self::below), + /// if any. + pub fn path(&self) -> Option<&JsonPath> { + self.path.as_ref() + } + + /// Whether the rows are the sealed root the server returned in place + /// of a subtree past its budget: one row, the root, at depth `0`. + pub fn is_sealed_root(&self) -> bool { + matches!( + self.nodes.as_slice(), + [node] if node.j == 0 && node.depth == Some(0) && node.header.sealed_root() + ) + } } impl Decryptable for JsonSubtree { @@ -580,12 +640,16 @@ fn unbase64(text: &str) -> Result, WireError> { #[error("a JSON index wire field does not decode: {0}")] pub struct WireError(&'static str); -/// `h` and `sv`, as stored. +/// `h`, `sv` and `q`, as stored. `q` is written from the entries and not +/// read back: a document deserialized from the wire derives its projection +/// from `sv` again, so the field is accepted and ignored on the way in. #[derive(Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct DocumentWire { h: String, sv: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + q: Option, } /// One entry, as stored: tags in hex, the ciphertext in base64. @@ -648,6 +712,7 @@ impl From for DocumentWire { Self { h: base64(&doc.header.to_bytes()), sv: doc.entries.iter().map(EntryWire::from).collect(), + q: Some(query::GroupWire::from(&doc.projection())), } } } diff --git a/packages/stack-encrypt/src/json/path.rs b/packages/stack-encrypt/src/json/path.rs index 968747b71..87ce2c960 100644 --- a/packages/stack-encrypt/src/json/path.rs +++ b/packages/stack-encrypt/src/json/path.rs @@ -152,6 +152,32 @@ impl JsonPath { !self.segments.iter().any(|s| matches!(s, Segment::Wildcard)) } + /// The values at this path in `value`, in document order: at most one + /// for a positional path, one per element a wildcard reaches, none when + /// the path names nothing. A key step into an array, or a position + /// into an object, reaches nothing, as the token chain reaches nothing + /// there. What a client takes over a whole document when a walk + /// returned the sealed root in place of the subtree. + pub fn select<'a>(&self, value: &'a serde_json::Value) -> Vec<&'a serde_json::Value> { + let mut current = vec![value]; + for segment in &self.segments { + let mut next = Vec::new(); + for node in current { + match segment { + Segment::Key(key) => next.extend(node.as_object().and_then(|o| o.get(key))), + Segment::Index(index) => next.extend( + usize::try_from(*index) + .ok() + .and_then(|i| node.as_array().and_then(|a| a.get(i))), + ), + Segment::Wildcard => next.extend(node.as_array().into_iter().flatten()), + } + } + current = next; + } + current + } + /// This path with one more step. pub(crate) fn then(&self, segment: Segment) -> Self { let mut segments = Vec::with_capacity(self.segments.len() + 1); @@ -311,6 +337,35 @@ mod tests { assert!(JsonPath::parse("$.a[2]").unwrap().is_positional()); } + #[test] + fn selects_what_the_chain_would_reach() { + let doc = serde_json::json!({ + "a": [{"x": 1}, {"x": 2, "y": [3, 4]}, 5], + "b": {"c": "d"}, + }); + let at = |text: &str| JsonPath::parse(text).unwrap().select(&doc); + assert_eq!(at("$"), vec![&doc]); + assert_eq!(at("$.b.c"), vec![&serde_json::json!("d")]); + assert_eq!(at("$.a[1].x"), vec![&serde_json::json!(2)]); + assert_eq!( + at("$.a[*].x"), + vec![&serde_json::json!(1), &serde_json::json!(2)] + ); + assert_eq!( + at("$.a[*].y[*]"), + vec![&serde_json::json!(3), &serde_json::json!(4)] + ); + assert!(at("$.a.x").is_empty(), "a key step into an array"); + assert!(at("$.b[0]").is_empty(), "a position into an object"); + assert!(at("$.a[3]").is_empty(), "past the end"); + assert!(at("$.b[*]").is_empty(), "the wildcard over an object"); + assert!(at("$.z").is_empty()); + assert!(JsonPath::parse("$[18446744073709551615]") + .unwrap() + .select(&doc) + .is_empty()); + } + #[test] fn refuses_what_it_does_not_address() { for (text, at) in [ diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index 944261717..351065929 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -132,9 +132,11 @@ fn group_of(nodes: &[TermNode], parent: Option) -> JsonProjection { } impl JsonDocument { - /// The stored document's projection, from its group tags, as the SQL's - /// `json_tree_query` computes it: what a [`JsonQuery`] is matched - /// against. + /// The stored document's projection, from its group tags: what a + /// [`JsonQuery`] is matched against. The document writes it to the wire + /// as `q`, beside `sv`, and the SQL's `json_tree_query` reads it from + /// there; a document deserialized from the wire derives it again from + /// the entries, so a stored `q` is never trusted over them. pub fn projection(&self) -> JsonProjection { fn belongs(e: &JsonEntry) -> Option<&Tag> { e.n.as_ref().or(e.g.as_ref()) @@ -157,14 +159,15 @@ impl JsonDocument { } } -#[derive(Serialize, Deserialize)] +#[derive(Clone, Serialize, Deserialize)] struct PairWire { s: String, vs: String, } -#[derive(Serialize, Deserialize)] -struct GroupWire { +/// One group, as a needle or a stored document's `q` spells it. +#[derive(Clone, Serialize, Deserialize)] +pub(crate) struct GroupWire { sv: Vec, el: Vec, } diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index 8d4fe2226..ef89f3e69 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -152,7 +152,26 @@ async fn the_wire_form_is_what_the_sql_domain_stores() { .keys() .map(String::as_str) .collect(); - assert_eq!(keys, ["h", "sv"]); + assert_eq!(keys, ["h", "q", "sv"]); + // `q` is the document's own projection, written once: the needle a + // `Contains` of the whole document derives has the same pairs in the + // same groups, and nothing else. + let needle = cipher + .query(op::Contains, &fixture()) + .on(&plan()) + .await + .unwrap(); + assert_eq!(wire["q"], serde_json::to_value(&needle).unwrap()); + assert!( + serde_json::from_value::(json!({"h": wire["h"], "sv": wire["sv"]})).is_ok(), + "a document without `q` still deserializes: the projection is derived from the entries" + ); + let back: JsonDocument = serde_json::from_value(wire.clone()).unwrap(); + assert_eq!( + serde_json::to_value(&back).unwrap(), + wire, + "`q` round-trips through the entries" + ); for (j, entry) in wire["sv"].as_array().unwrap().iter().enumerate() { let entry = entry.as_object().unwrap(); for key in entry.keys() { @@ -1131,6 +1150,226 @@ async fn each_stored_form_opens_as_a_record_field() { assert_eq!(opened, json!([{"x": 1}, {"x": 2}])); } +/// Past its budget the server returns the sealed root alone, at depth 0, +/// in place of the subtree: the rows open to the whole document and the +/// path attached with `at` selects over it. Without the path, or with one +/// that reaches nothing, the opening is refused; a root at depth 0 under +/// an unsealed document is not the fallback. +#[tokio::test] +async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { + let cipher = stack_cipher().await; + let doc = seal(&cipher, &fixture()).await; + let fallback = |doc: &JsonDocument| -> JsonNode { + let mut wire = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + wire["d"] = json!(0); + serde_json::from_value(wire).unwrap() + }; + let root = fallback(&doc); + let rows = JsonSubtree::new(vec![root.clone()]); + assert!(rows.is_sealed_root()); + assert!(rows.path().is_none()); + + let nested = cipher + .decrypt_as::( + rows.clone().at(JsonPath::parse("$.nested").unwrap()), + aead(), + ) + .await + .unwrap(); + assert_eq!(nested, fixture()["nested"]); + let xs = cipher + .decrypt_as::, JsonSubtree>( + rows.clone() + .at(JsonPath::parse("$.nested.list[*].x").unwrap()), + aead(), + ) + .await + .unwrap(); + assert_eq!(xs, vec![json!(1), json!(2)]); + let none = cipher + .decrypt_as::, JsonSubtree>( + rows.clone().at(JsonPath::parse("$.missing[*]").unwrap()), + aead(), + ) + .await + .unwrap(); + assert!(none.is_empty(), "a path that reaches nothing is no subtree"); + + // Each refusal names what it expected: nothing at the path, or one node + // where a wildcard reached two. + for (path, expected) in [ + ("$.missing", "a node at the walked path"), + ("$.nested.list[*].x", "one node at the walked path"), + ] { + let refused = cipher + .decrypt_as::( + rows.clone().at(JsonPath::parse(path).unwrap()), + aead(), + ) + .await; + match refused { + Err(Error::OpenedType { expected: got }) => assert_eq!(got, expected, "{path}"), + other => panic!("{path}: {other:?}"), + } + } + let unattached = cipher + .decrypt_as::(rows.clone(), aead()) + .await; + assert!( + matches!(unattached, Err(Error::OpenedType { .. })), + "the fallback needs the path: {unattached:?}" + ); + + // The same root at depth 1, or at depth 0 under an unsealed document, + // is a walk's row and not the fallback; neither assembles. + let mut as_walked = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + as_walked["d"] = json!(1); + let as_walked: JsonNode = serde_json::from_value(as_walked).unwrap(); + assert!(!JsonSubtree::new(vec![as_walked.clone()]).is_sealed_root()); + let refused = cipher + .decrypt_as::( + JsonSubtree::new(vec![as_walked]).at(JsonPath::parse("$.nested").unwrap()), + aead(), + ) + .await; + assert!(matches!(refused, Err(Error::OpenedType { .. }))); + let open_doc = seal_with(&cipher, &fixture(), JsonOptions { sealed_root: false }).await; + assert!(!JsonSubtree::new(vec![fallback(&open_doc)]).is_sealed_root()); + assert!(!JsonSubtree::new(vec![]).is_sealed_root()); + assert!(!JsonSubtree::new(vec![root.clone(), root]).is_sealed_root()); +} + +/// The sealed root a walk fell back to opens to exactly what the walked +/// rows open to, so a caller's result does not change shape when a +/// document grows past the budget. `json_tree_subtree`'s rows are the +/// descendants of the token's node, so the path is attached with `below` +/// and a `Vec` opens to that node's children; `at` is for +/// `json_tree_subtree_at`, whose rows are the nodes the path reaches. +#[tokio::test] +async fn the_fallback_opens_to_what_the_walk_opens_to() { + let cipher = stack_cipher().await; + let doc = seal(&cipher, &fixture()).await; + let mut root = serde_json::to_value(doc.node(0).unwrap()).unwrap(); + root["d"] = json!(0); + let fallback = JsonSubtree::new(vec![serde_json::from_value(root).unwrap()]); + assert!(fallback.is_sealed_root()); + let path = |text: &str| JsonPath::parse(text).unwrap(); + let walked = |text: &'static str| { + let cipher = &cipher; + let doc = &doc; + async move { + let key = subtree_key(cipher, text, JsonOptions::default()) + .await + .unwrap(); + let rows = doc + .subtree(key.token_bytes()) + .iter() + .map(|(j, _, _)| doc.node(*j).unwrap()) + .collect(); + JsonSubtree::new(rows) + } + }; + let sorted = |mut values: Vec| { + values.sort_by_key(ToString::to_string); + values + }; + + // An array: its elements, in order, either way. + let list = cipher + .decrypt_as::, JsonSubtree>(walked("$.nested.list").await, aead()) + .await + .unwrap(); + assert_eq!(list, vec![json!({"x": 1}), json!({"x": 2})]); + let below = |text: &str| fallback.clone().below(path(text)); + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>(below("$.nested.list"), aead()) + .await + .unwrap(), + list + ); + // An object: its member values, in an order neither form fixes. + let members = cipher + .decrypt_as::, JsonSubtree>(walked("$.nested").await, aead()) + .await + .unwrap(); + assert_eq!(members.len(), 2); + assert_eq!( + sorted( + cipher + .decrypt_as::, JsonSubtree>(below("$.nested"), aead()) + .await + .unwrap() + ), + sorted(members) + ); + // As one value, the node itself, either way. + assert_eq!( + cipher + .decrypt_as::(below("$.nested"), aead()) + .await + .unwrap(), + fixture()["nested"] + ); + // A scalar has no children, as its walk has no rows. + assert!(cipher + .decrypt_as::, JsonSubtree>(below("$.hello"), aead()) + .await + .unwrap() + .is_empty()); + // `below` names one node: nothing at the path, or a wildcard reaching + // two, is refused. + for text in ["$.missing", "$.nested.list[*]"] { + match cipher + .decrypt_as::, JsonSubtree>(below(text), aead()) + .await + { + Err(Error::OpenedType { expected }) => { + assert_eq!(expected, "one node at the walked path", "{text}") + } + other => panic!("{text}: {other:?}"), + } + } + // `at` is the other walk's: the nodes the path reaches, so on this path + // the list itself. Attaching the wrong one is what changes the shape. + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>( + fallback.clone().at(path("$.nested.list")), + aead() + ) + .await + .unwrap(), + vec![fixture()["nested"]["list"].clone()] + ); + // The last attachment decides, both the path and the walk. + let reattached = fallback + .clone() + .below(path("$.nested")) + .at(path("$.nested.list")); + assert_eq!( + reattached.path().map(ToString::to_string).as_deref(), + Some("$.nested.list") + ); + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>(reattached, aead()) + .await + .unwrap(), + vec![fixture()["nested"]["list"].clone()] + ); + assert_eq!( + cipher + .decrypt_as::, JsonSubtree>( + fallback.at(path("$.nested")).below(path("$.nested.list")), + aead() + ) + .await + .unwrap(), + list + ); +} + /// The walk reports each row's depth below where it started, in `d`; an /// extracted node carries none. #[tokio::test]