diff --git a/.changeset/eql-json-tree-empty-container.md b/.changeset/eql-json-tree-empty-container.md new file mode 100644 index 000000000..b0c478075 --- /dev/null +++ b/.changeset/eql-json-tree-empty-container.md @@ -0,0 +1,5 @@ +--- +"@cipherstash/eql": minor +--- + +EQL v4 `json_tree` (unreleased): every subtree read now returns the node it names, so an empty object, an empty array, a scalar and a missing field are four different answers, as `jsonb` gives them. The walk for a top-level field, `eql_v4.json_tree_subtree`, returned the field's descendants and never the field itself, because a walk finds a node by its parent's token and a top-level field has none; so `{}`, `[]`, `"text"` and no field at all each came back as no rows. **SQL signature change:** `json_tree_subtree(doc, token, budget)` becomes `json_tree_subtree(doc, token, selector, budget DEFAULT 64)`. The extra argument is the field's selector, the same one `->` takes for the path: the walk returns the one entry with no parent tag that carries it, at `d = 0`, before its descendants, and returns nothing when the document has no such entry. `json_tree_subtree_at` is unchanged and still returns the node its gates reach at `d = 0`; both now run one internal walk, `eql_v4_internal.json_tree_walk`. No stored field was added. In Stack Encrypt a subtree key for a path that ends in an array position (`$.tags[2]`) now walks from the array's token through a position gate, so the element itself comes back; `JsonSubtreeKey::selector()` is what `json_tree_subtree` takes. In `eql-bindings`, `JsonTreeSubtree` opens to the node's own value, and to `Option`, which is `None` where the row has no node at the path. What the database learns: the selector sent beside a top-level field's token is the one `->` sends; a path ending in an array position sends its array's token, which covers every element of that array, as a wildcard read already does. diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 9c336e483..c0fdbebaa 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -335,7 +335,7 @@ The four above are in the phased plan: the native-`@>` projection and container- - **A build change, additive only.** `tasks/build.sh` renders `src/v3/**` as both bundles and `verify_installer_complete.sh` asserts every ordered v3 file is in the v4 installer. The new family needs the renderer to also order and render a v4-only source tree into the v4 bundle, and the completeness check to accept files that exist in one bundle only. Nothing is removed from either bundle, so this cannot block the v4 release: until the build change lands, the phased plan installs the new family as a separate file over the v4 bundle. - Tests: `mise run test:encryption:postgres` in `packages/eql` installs the v4 bundle built from the tree and runs the Stack Encrypt PostgreSQL tests with fresh encryption under a fake KMS and no credentials (it reached `next` after `3a216441`, the commit the rest of this section was read from, which has only the database-free `test:encryption`). The JSON suite joins it. The SQLx fixture machinery is not needed for this and should not be used: the point of the fake key source is that the whole suite runs on a fork PR. -> **2026-10-22:** as built, the walk functions are `eql_v4.json_tree_subtree(doc, token, budget)`, `json_tree_subtree_at(doc, token, gates, budget)`, `json_tree_children(doc, token, gates)` and `json_tree_child_count(doc, token, gates)`, and there is no `groups_ok`. +> **2026-10-22:** as built, the walk functions are `eql_v4.json_tree_subtree(doc, token, budget)`, `json_tree_subtree_at(doc, token, gates, budget)`, `json_tree_children(doc, token, gates)` and `json_tree_child_count(doc, token, gates)`, and there is no `groups_ok`. (2026-10-24: `json_tree_subtree(doc, token, selector, budget)`, which returns the node itself; see the Phase 4 notes.) **`languages/golang`** @@ -373,7 +373,7 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi - *Nonces are random, not derived from the selector and ordinal.* Every entry seals through the engine's one AEAD path (`vitaminc-encrypt`'s AES-256-GCM with its own random 96-bit nonce, carried in `c`), so there is one leaf cipher in the crate and no second AES-GCM call site. Uniqueness under one document key is then probabilistic (`N² / 2^97` at `N` entries) rather than structural. The entry's AAD still binds its ordinal `j`, so a moved or reordered entry fails to open as the design requires; the cost is twelve bytes per entry, which Phase 3 should count. - *The ordinal is `o`, not `i`*, because `i` is the envelope's identifier, which `->` grafts onto an extracted entry. An extracted entry also carries `j`, its ordinal in the document, which its AAD is bound to, and a walk adds `d`, its depth below the start. (2026-10-22: since Phase 3b `o` is not stored at all.) -- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. (2026-10-22: a gated walk, `json_tree_subtree_at`, now returns the node its gates reach at `d = 0`, with no self tag; the ungated `json_tree_subtree` still returns descendants only.) +- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. (2026-10-22: a gated walk, `json_tree_subtree_at`, now returns the node its gates reach at `d = 0`, with no self tag; the ungated `json_tree_subtree` still returns descendants only.) **Closed (2026-10-24, #1356, PR #1359), with no self tag:** every subtree read now returns the node it names at `d = 0`; see the Phase 4 notes. - *Containment is not `STRICT` in SQL.* PostgreSQL inlines a `STRICT` SQL function only when it can prove the body is NULL for a NULL argument, and the `OR` that adds the needle's `alt` probe is not provably NULL; un-inlined, the call was an opaque filter and the GIN index never engaged. `eql_v4.json_tree_contains` and the `@>` / `<@` wrappers over it are declared without `STRICT` and return NULL for a NULL side on their own. - *`<@` cannot match a stored root scalar inside a needle root array*, since a stored document carries no `alt` pair. The oracle records the case; `@>` is exact. - *Value selectors compare strings byte for byte*, with no Unicode normalization, because `jsonb @>` does, and the oracle is agreement with `jsonb @>`. The scalar equality index normalizes to NFC; a JSON value does not. Numbers canonicalize by value, so `2` and `2.0` are one value, as `jsonb` compares them. @@ -401,7 +401,7 @@ A review of the stored form after the Phase 3 measurements found three fields no - **`vs`, `n` and `g` are gone from the entries.** Once `q` was stored (Phase 3), no SQL function read an entry's value selector or group tags: containment reads `q` alone, and the engine read them only to rebuild `q` on deserialize. The engine now takes the stored `q` as the projection. That is no change in trust: `q` and the entry fields were equally unauthenticated (only `c` is, through its AAD), and a tampered `q` changes which rows a containment query returns and nothing else. On the nested benchmark document (213 entries on average) the values of those three fields were 15% of `sv`. - **Tokens chain by ordinal.** `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` replaces the label-derived step. The server derives every hit's token from its parent's as it walks, so `w` (the wrapped token) is not stored. The cost is that the client cannot derive an object member's token, because an ordinal is a rank among sibling keys it does not know. It derives a chain head's token and those of the array positions that directly follow the head, and sends one gate per further step (`JsonSubtreeKey::gates()`: the step's selector, or `#n` for a position); `json_tree_subtree_at(doc, token, gates, budget)` descends through them, and `json_tree_children` and `json_tree_child_count` take the same gates. A wildcard step's selector is every element's, which is the old `subtree_at` gate generalised to every depth. - **The walk probes to the first miss.** The ordinals under a node are dense from zero (positions for arrays, a permutation for objects), so the walk tries `0, 1, 2, …` and stops at the first miss. `o` is not stored and `json_tree_fanout` is gone. A subtree of `N` nodes costs about `2N` probes (`N` hits, and a miss for every node visited), each a SHA-256 and a map lookup, plus a SHA-256 per hit for its token, where it cost `F` probes per visited node, `F` being the widest node in the whole document; on the nested corpus a wide `items` array taxed every object walk in the same document. -- **Depth is reported from the reached node.** A gated walk returns the node its gates reach with `d = 0` and its descendants beneath it, so the client can tell "the value at the path" (one reached node) from "the children of a node" (no `d = 0` row) without the path. A wildcard reaches several nodes and opens as a list. The ungated `json_tree_subtree(doc, token)` is now `json_tree_subtree_at` with no gates, and returns the token's node's descendants from `d = 1` as before. +- **Depth is reported from the reached node.** A gated walk returns the node its gates reach with `d = 0` and its descendants beneath it, so the client can tell "the value at the path" (one reached node) from "the children of a node" (no `d = 0` row) without the path. A wildcard reaches several nodes and opens as a list. The ungated `json_tree_subtree(doc, token)` is now `json_tree_subtree_at` with no gates, and returns the token's node's descendants from `d = 1` as before. (2026-10-24: no longer; `json_tree_subtree(doc, token, selector)` returns the node at `d = 0` too, and every walk now opens to the node it reaches; see the Phase 4 notes.) - **Leakage.** At rest, the same as Phase 3 less what the stored ordinals told: their multiset gave, for every `k`, how many nodes in the chain have more than `k` children. `p` is still salted by the document `iv`. Under a token, the same: a token still unlocks the shape of that subtree in every row. The `w` mask's binding to `c` (a swapped ciphertext broke the walk on the server) is gone; the swap still fails when the client opens the entry, because `c` is bound to `s` and `j` in its AAD. - **The chain step binds the child's selector (2026-10-23, review of #1329).** As first built, `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` dropped the kind byte the label-derived step had (`0x01` for a key, `0x02` for an index), and an ordinal does not say which kind of child it numbers: an array element's is its position, an object member's its rank. Rows of one column need not agree on a node's kind, so the token the client derives for `$.a[0]` was, in a row whose `$.a` is an object, the token of the member ranked first, and a walk from it returned that member's subtree as the element's; a `#n` gate admitted the same member. `jsonb` has no node there (`jsonb_path_query_first(doc, 'strict $.a[0]')` is NULL), and the sealed-root fallback, which selects over the plaintext, agreed with `jsonb`, so one row answered differently under and over the budget. The step is now `t(child) = H("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`, the child's own selector bound in. The server holds every hit's `s`, so it derives the token as before (`eql_v4_internal.json_tree_child_token(token, s, o)`), and the client knows an element's selector from the path (the wildcard's), so it still derives positional tokens after a head. An element's selector is the wildcard path's and a member's names its key, so they never share a token. A position gate carries the element selector too, `#n:`, and the walk admits the hit at ordinal `n` only if it carries it. Nothing about leakage moves: the selector is stored beside the entry, and a token's holder already learns it with every hit. `postgres_a_position_reaches_array_elements_only` holds both forms against `jsonb`'s strict path over the plaintexts. @@ -495,8 +495,14 @@ Two of the three items that gate a release are closed here; the third, the final - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. -- **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. -- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. +- **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. Filed as #1357 (2026-10-24), to be investigated in its own pull request. +- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. (2026-10-24: closed by the next note; every walk now returns the node it reaches, and a path's last position is a gate.) +- **Taken (2026-10-24, #1356, PR #1359): every subtree read returns the node it names.** The ungated walk, `json_tree_subtree(doc, token, budget)`, returned a node's descendants and never the node, because the walk finds a node by its parent's token and a chain head has none. So `$.profile` as `{}`, as `[]`, as a scalar and absent all came back as no rows, and the client could not tell them apart, where `jsonb` answers `{}`, `[]`, the scalar and NULL. No stored field was added; the 16-byte self tag rejected on size above was not needed: + - **A chain head is found by its selector.** A top-level field's selector is unique among the top-level entries, and the heads are exactly the entries with no `p` (with the root open, only the root is, and its selector is `$`'s). `json_tree_subtree` becomes `json_tree_subtree(doc, token, selector, budget)`: the one entry with no `p` that carries `selector`, at `d = 0`, then its descendants from the token. With no such entry it returns nothing, which is "no node at the path". The body is one internal walk, `eql_v4_internal.json_tree_walk(doc, token, gates, budget, selector)`, which `json_tree_subtree_at` calls with no selector. `JsonSubtreeKey::selector()` is what the client sends. + - **A path's last step is never a chained position.** The client used to derive the token of `$.l[3]` and walk from it, which returns `$.l[3]`'s children and never the element. The key now chains positions up to the last step and makes a last position a gate from its array's token (`#3:`), which the gated walk already returns at `d = 0`. Every key is then either a chain head or the open root with no gates, or ends in a gate. + - **The client opens what the rows reached.** `JsonSubtree` opens to the value of the one node at the path (`{}`, `[]`, the scalar), to `Option` (`None` for no node, matching `->`, which returns SQL `NULL` for a path the row lacks; `Value` refuses no node as `OpenedType`), or to a list (the reached nodes for a gated key, the node's children for a head). The sealed-root fallback selects the same answers from the whole document: `{}` and `[]` select as themselves, and a path the document lacks selects nothing. The checks of #1355 apply unchanged, now with the reached node always present, so a walk with no gates is checked from its head down. + - **What the database learns.** `json_tree_subtree` now sends the head's selector beside its token. The selector is the one `->` sends for the same path, so it is nothing new on its own. With the token beside it, the database learns which top-level field the token is for, which the depth-first stored order already shows it the first time the token reaches one of the field's members (the entry before a subtree's first member is its parent); what is new is that pairing for a field with no members in any row the token reaches. A path that ends in an array position now sends its array's token rather than the element's, with the position as a gate: the database can walk the whole array in every row, not one element, for as long as the column lives, as it already could for `$.l[*]` or any key below `$.l`. The leakage page says both. + - **Verified** against `jsonb_path_query_first(plain, 'strict ')` in `postgres_every_subtree_read_answers_as_jsonb_does`: `{}`, `[]`, scalars (`null` included), absent and non-empty nodes, at a chain head, at array positions after one (chained and not), behind a gate, with the root sealed and open, walked out and past a budget of 1. ## Decisions to take before Phase 1 diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index 69ce58852..de97b9084 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -15,7 +15,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 projection, which the server compares with a needle by native `jsonb @>`. `JsonTreeEntry` is one entry as `->` and the walks return it, `JsonTreeQuery` the containment needle and `JsonTreeLeafQuery` the - ordering operand at a path. `JsonTreeSubtree` collects a walk's rows; past + ordering operand at a path. `JsonTreeSubtree` collects a walk's rows + (`eql_v4.json_tree_subtree(doc, key.token(), key.selector())` for a key + with no gates, `json_tree_subtree_at(doc, key.token(), key.gates())` for + one with), which hold the node at the path itself, so they open to its + value, `{}`, `[]` and scalars included, or, as `Option`, to + `None` where the row has no node there; past the walk's budget the server returns the sealed root instead, which opens at the path the walk's key was derived from: the rows are built with `JsonTreeSubtree::from_key(&key, rows)`, and there is no diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index ba0f77eb7..d4dc84fd2 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -207,18 +207,21 @@ impl JsonTreeExtracted { } } -/// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or -/// `json_tree_subtree_at`), collected with the key that keyed the walk: -/// opens to the value under the walked path, or to the list of complete -/// subtrees the rows hold. +/// The rows a subtree walk returned, collected with the key that keyed +/// the walk: `eql_v4.json_tree_subtree(doc, key.token(), key.selector())` +/// for a key with no gates (a chain head, or the root when it is not +/// sealed), `json_tree_subtree_at(doc, key.token(), key.gates())` for one +/// with. Every walk returns each node the path reaches at depth `0` with +/// everything beneath it, so the rows open to the value at the walked path, +/// `{}`, `[]` and scalars included; to `None` as an `Option` when +/// there is no node there; or to a list. /// /// Past its budget a walk returns the sealed root alone, at depth `0`; /// those rows open to the whole document, and the key's path selects the -/// same value(s) the walked rows open to. The key also says which walk the -/// rows came from (no gates: `json_tree_subtree`, the rows below the path's -/// node; gates: `json_tree_subtree_at`, the nodes the path reaches), so the -/// only constructor, [`from_key`](Self::from_key), takes it, and rows -/// cannot be opened without it. +/// same value(s) the walked rows open to. The only constructor, +/// [`from_key`](Self::from_key), takes the key, which checks every row +/// against its path when the rows are opened, so rows cannot be opened +/// without it. #[derive(Clone, Debug, PartialEq)] pub struct JsonTreeSubtree { rows: Vec, @@ -455,8 +458,8 @@ impl DecryptInto for JsonTreeExtracted { } } -/// The rows of `json_tree_subtree(doc, token)` open to the value under the -/// walked node. +/// The rows of a walk open to the value at the walked path; no node there +/// is an error, which `Option` reads as `None`. impl DecryptInto for JsonTreeSubtree { type Context = NonEmpty; fn decryption(self, context: Self::Context) -> Decryption { @@ -467,8 +470,21 @@ impl DecryptInto for JsonTreeSubtree { } } -/// The rows of a walk open to the complete subtrees they hold, in order: -/// one per node the gates of `json_tree_subtree_at` reached. +/// The rows of a walk open to the value at the walked path, or `None` +/// where the document has no node there: the four answers `jsonb #>` gives. +impl DecryptInto> for JsonTreeSubtree { + type Context = NonEmpty; + fn decryption(self, context: Self::Context) -> Decryption, K> { + match self.subtree() { + Ok(subtree) => subtree.decryption(AeadContext::from(context)), + Err(error) => Decryption::failed(error), + } + } +} + +/// The rows of a walk open to a list: one value per node the gates of +/// `json_tree_subtree_at` reached, or, for a key with no gates, the values +/// of the node's children. impl DecryptInto> for JsonTreeSubtree { type Context = NonEmpty; fn decryption(self, context: Self::Context) -> Decryption, K> { diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index 265aa6fd6..b602d5903 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -33,7 +33,7 @@ A query hands the database a term derived from the plaintext it asks about, and - **Containment (`@>`, `<@`), and equality at a path, which is sent as a containment needle.** The needle's own projection: the selectors of its paths, the value selectors of its nodes, and its element grouping. So the database learns which paths and which (path, value) pairs you asked about, and which rows hold them. - **Path extraction (`->`, `->>`, `json_tree_path_query`, `json_tree_path_exists`).** The selector of the path asked. A position in the path reads as the wildcard, so a position is not sent. The entry returned is a ciphertext. - **Ordering at a path (`<`, `<=`, `>`, `>=`).** The path's selector and the bound's own ordering term, which compares with the stored terms exactly as they compare with each other: the database learns the bound's place in the order of the stored values at every ordered path, and whether it equals one. `ORDER BY ord_term(…)` sends no term and shows nothing the stored terms do not. -- **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token**, and one gate per step below it. The token is not the token of the node you asked for: it is the token of the deepest node on the path the client can derive, which is the top-level field the path starts with or, while the path goes on through array positions, the element they reach. Reading `$.profile.address.city` sends `$.profile`'s token. A gate is the step's selector or, for an array position, `#n:`, with the position `n` in the clear. A token is a capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find each node under that field in any row, see how the nodes nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree, and no token covers a whole document. With it off, every subtree read whose path begins with a key sends the root's token, which covers every row's whole document. +- **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token**, and one gate per step below it. The token is not the token of the node you asked for: it is the token of the deepest node on the path the client can derive, which is the top-level field the path starts with or, while the path goes on through array positions, the element they reach, short of the path's last step. Reading `$.profile.address.city` sends `$.profile`'s token. A gate is the step's selector or, for an array position, `#n:`, with the position `n` in the clear. A path that ends in an array position sends the token of the array and the position as a gate, so that the walk can return the element itself: reading `$.tags[2]` sends `$.tags`'s token, which lets the database walk every element of `$.tags`, as reading `$.tags[*]` does, rather than the third alone. Reading a top-level field itself (`json_tree_subtree`) also sends that field's **selector** beside its token, so the walk can return the field's own entry: the same selector `->` sends for the path, which the database would otherwise have linked to the token only once the token reached one of the field's members (the entry stored before a subtree's first member is its parent). A token is a capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find each node under that field in any row, see how the nodes nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree, and no token covers a whole document. With it off, every subtree read whose path begins with a key sends the root's token, which covers every row's whole document. ## What is authenticated @@ -60,7 +60,7 @@ What the checks do not cover: **which row an entry comes from**. Nothing in an e | which leaves sit in the same array element | no | yes, through the projection's nesting: the one new disclosure, required for exact containment, bounded to array elements | | equal values in sibling elements | ciphertexts equal | ciphertexts differ | | ordering terms | on scalar leaves, with no per-path choice | only at the paths the index orders; none by default | -| a subtree read | reveals the selector asked | reveals a token for the top-level field the path starts with, and with it that field's shape in every row for the life of the column | +| a subtree read | reveals the selector asked | reveals a token for the top-level field the path starts with (or, for a path ending in an array position, for that array), and with it that subtree's shape in every row for the life of the column; reading a top-level field itself also sends its selector | ## The two options that change the picture diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index b325a2e2d..6d1ee108a 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -477,47 +477,51 @@ AS $$ SELECT count(*)::integer FROM {{prefix}}.json_tree_children(e, token, gates) $$; ---! @brief The entries under the node the gates reach from the node whose ---! token is `token`, as rows: the recursive walk of the chain, in ---! core SQL. +--! @brief The walk behind `json_tree_subtree` and `json_tree_subtree_at`: +--! from the node whose token is `token`, through `gates`, each node +--! the gates reach and everything beneath it, as rows. --! --! Each level probes the ordinals under the node in hand until the first ---! miss, derives each hit's token, and recurses. A token that names no node ---! yields no row, and nothing in a row is linkable to anything without the ---! token. At depth `d` (1 below the token) the gate `gates[d]`, if there ---! is one, decides which hits the walk descends through: an entry whose ---! `s` is the gate's selector, or, for a gate `#n:`, the one hit ---! at ordinal `n` if it carries that selector. Rows are returned from the depth of the last gate down: the ---! node a gate chain reaches, with `d = 0`, and everything beneath it, with ---! `d` its depth below that node. The start node is never returned. With ---! no gates, every descendant of the start node, with `d` its depth below ---! it, from 1. +--! miss, derives each hit's token, and recurses; a token that names no node +--! yields no row. At depth `d` (1 below the token) the gate `gates[d]`, if +--! there is one, decides which hits the walk descends through: an entry +--! whose `s` is the gate's selector, or, for `#n:`, the one hit +--! at ordinal `n` if it carries that selector. Rows are returned from the +--! depth of the last gate down: each node a gate chain reaches, with +--! `d = 0`, and everything beneath it, with `d` its depth below that node. +--! +--! No probe finds the start node, since a node is found by its parent's +--! token. With no gates it is returned only when `selector` names it: it +--! is then a chain head, or an open root, which carries no parent tag, so +--! it is the one entry with no `p` whose `s` is `selector`, returned with +--! `d = 0`. When `selector` names no such entry the walk returns nothing: +--! there is no node at the path. --! --! **The budget.** A walk returns about two hundred characters per entry, ---! where a sealed root holds the whole document in one ciphertext of about ---! the plaintext's size. So when the root is sealed and the walk would ---! return more than `budget` entries, it stops and returns the root entry ---! alone, with `d = 0`: the client opens it to the whole document and takes ---! the path itself, which is cheaper on both sides than the walk past that ---! point. The default of 64 is past the crossover by bytes (about a dozen ---! entries) by a margin that keeps a small subtree arriving as the entries ---! it is. A `budget` of `0` never falls back. A token that names no node ---! yields no row under any budget, and a document whose root is not sealed ---! is always walked in full, since its root holds no document to return. +--! where a sealed root holds the whole document in about the plaintext's +--! size. So when the root is sealed and the walk would return more than +--! `budget` entries, it returns the root entry alone, with `d = 0`, and the +--! client takes the path over the whole document. A `budget` of `0` never +--! falls back; a walk that finds no node yields no row under any budget; +--! a document whose root is not sealed is always walked in full. --! +--! @internal --! @param e public.{{prefix}}_json_tree The stored document. --! @param token text The start node's token, base64. --! @param gates text[] One gate per step below the token: a selector, base64, ---! or `#n:` for the array position `n`, with the array ---! elements' selector. Empty for the whole subtree. +--! or `#n:` for the array position `n`. --! @param budget integer The most entries to return before falling back to ---! the sealed root; `0` to always walk out. Default 64. +--! the sealed root; `0` to always walk out. +--! @param selector text The start node's own selector, base64, to return it +--! at `d = 0`; NULL to return what the gates reach. --! @return SETOF public.{{prefix}}_json_tree_entry The entries, each with its ---! depth `d` below the node the gates reach, in document order; or ---! the root entry alone, with `j = 0` and `d = 0`, past the budget. -CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, gates text[], budget integer DEFAULT 64) +--! depth `d` below the node reached, in document order; or the root +--! entry alone, with `j = 0` and `d = 0`, past the budget. +--! @note Not STRICT: `selector` is NULL for a gated walk. The public +--! wrappers are STRICT, so nothing else here is ever NULL. +CREATE FUNCTION {{prefix}}_internal.json_tree_walk(e public.{{prefix}}_json_tree, token text, gates text[], budget integer, selector text) RETURNS SETOF public.{{prefix}}_json_tree_entry - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ WITH RECURSIVE doc AS MATERIALIZED ( @@ -527,14 +531,30 @@ AS $$ e::jsonb -> 'sv' -> 0 AS root, jsonb_array_length(e::jsonb -> 'sv') AS entries, budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped, - COALESCE(array_length(gates, 1), 0) AS gated + COALESCE(array_length(gates, 1), 0) AS gated, + start.entry AS start, + selector IS NULL OR start.entry IS NOT NULL AS found + FROM ( + -- The start node by its selector: the first entry with no parent tag + -- that carries it, with its ordinal `j` in `sv`. + SELECT ( + SELECT elem || jsonb_build_object('j', ord - 1) + FROM jsonb_array_elements(e::jsonb -> 'sv') WITH ORDINALITY AS t(elem, ord) + WHERE selector IS NOT NULL AND NOT (elem ? 'p') AND elem ->> 's' = selector + ORDER BY ord + LIMIT 1 + ) AS entry + ) AS start ), -- One row per hit, after the seed row for the start node. `pt` and `o` -- are where the hit sits, its parent's token and its ordinal; `t` is its -- own token; `ppass` whether every gate down to its parent admitted the - -- parent, `pass` whether this hit's gate admits it too. + -- parent, `pass` whether this hit's gate admits it too. A start node its + -- selector names no entry for seeds nothing. walk(depth, pt, o, entry, t, ppass, pass) AS ( SELECT 0, NULL::bytea, NULL::integer, NULL::jsonb, decode(token, 'base64'), true, true + FROM doc + WHERE doc.found UNION ALL SELECT probe.depth, probe.pt, probe.o, hit.entry, {{prefix}}_internal.json_tree_child_token(probe.pt, decode(hit.entry ->> 's', 'base64'), probe.o), @@ -569,22 +589,30 @@ AS $$ WHERE hit.entry IS NOT NULL ), -- At most one row per entry and the seed (see "The bound" above), - -- whatever the budget: a budget of 0, or an open root, leaves `found` + -- whatever the budget: a budget of 0, or an open root, leaves `reached` -- uncapped, and this is then the walk's only limit. visited AS MATERIALIZED ( SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries + 1 FROM doc) ), -- One row past the budget is enough to know the walk went over; a walk -- under it is the subtree complete. Uncapped, the limit is NULL: all - -- that `visited` holds. - found AS MATERIALIZED ( - SELECT w.depth, w.entry - FROM visited AS w, doc - WHERE w.entry IS NOT NULL AND w.pass AND w.depth >= doc.gated + -- that `visited` holds. The start node, when its selector named it, + -- counts as one of the rows. + reached AS MATERIALIZED ( + SELECT r.depth, r.entry + FROM ( + SELECT 0 AS depth, doc.start AS entry + FROM doc + WHERE doc.start IS NOT NULL + UNION ALL + SELECT w.depth, w.entry + FROM visited AS w, doc + WHERE w.entry IS NOT NULL AND w.pass AND w.depth >= doc.gated + ) AS r LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) ), over AS MATERIALIZED ( - SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc + SELECT doc.capped AND (SELECT count(*) FROM reached) > budget AS fallback FROM doc ) SELECT rows.entry FROM ( @@ -593,28 +621,75 @@ AS $$ FROM doc, over WHERE over.fallback UNION ALL - SELECT (f.entry ->> 'j')::integer, - (doc.meta || f.entry || jsonb_build_object('d', f.depth - doc.gated))::public.{{prefix}}_json_tree_entry - FROM found AS f, doc, over + SELECT (r.entry ->> 'j')::integer, + (doc.meta || r.entry || jsonb_build_object('d', r.depth - doc.gated))::public.{{prefix}}_json_tree_entry + FROM reached AS r, doc, over WHERE NOT over.fallback ) AS rows ORDER BY rows.j $$; ---! @brief Every entry under the node whose token is `token`: ---! `json_tree_subtree_at` with no gates. +--! @brief The nodes the gates reach from the node whose token is `token`, +--! each with everything beneath it, as rows: the recursive walk of +--! the chain, in core SQL. +--! +--! The walk of `{{prefix}}_internal.json_tree_walk` with no start selector. +--! Rows are the node each gate chain reaches, with `d = 0`, and everything +--! beneath it, with `d` its depth below that node, so an empty object, an +--! empty array and a scalar each come back as one row, and no node as none. +--! With no gates the start node itself is not among the rows, since +--! nothing the walk probes names it: `json_tree_subtree` returns it by its +--! selector. Past `budget` entries, with the root sealed, the root entry +--! alone, with `d = 0`. The default budget of 64 is past the crossover by +--! bytes (about a dozen entries) by a margin that keeps a small subtree +--! arriving as the entries it is. +--! +--! @param e public.{{prefix}}_json_tree The stored document. +--! @param token text The start node's token, base64. +--! @param gates text[] One gate per step below the token: a selector, base64, +--! or `#n:` for the array position `n`, with the array +--! elements' selector. +--! @param budget integer The most entries to return before falling back to +--! the sealed root; `0` to always walk out. Default 64. +--! @return SETOF public.{{prefix}}_json_tree_entry The entries, each with its +--! depth `d` below the node the gates reach, in document order; or +--! the root entry alone, with `j = 0` and `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, gates text[], budget integer DEFAULT 64) + RETURNS SETOF public.{{prefix}}_json_tree_entry + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT * FROM {{prefix}}_internal.json_tree_walk(e, token, gates, budget, NULL) +$$; + +--! @brief A chain head (or, with the root not sealed, the root) and every +--! entry under it: the node whose token is `token` and whose +--! selector is `selector`, with `d = 0`, then its descendants. +--! +--! A chain head has no parent token, so no probe finds it; it is the one +--! entry with no parent tag that carries `selector`, which no other +--! top-level node carries. Its descendants are found from `token`. An +--! empty object, an empty array and a scalar come back as one row each, +--! and a node the document does not have as no row, so the four answers +--! `jsonb #>` gives are told apart. A token that is not the node's finds +--! no descendant, and the node alone comes back, as `->` on the selector +--! would return it. Past `budget` entries, with the root sealed, the root +--! entry alone, with `d = 0`. +--! --! @param e public.{{prefix}}_json_tree The stored document. ---! @param token text The subtree's token, base64. +--! @param token text The node's token, base64. +--! @param selector text The node's selector, base64: what `->` takes for the +--! same path. --! @param budget integer The most entries to return before falling back to --! the sealed root; `0` to always walk the subtree out. Default 64. ---! @return SETOF public.{{prefix}}_json_tree_entry The descendants, each with its ---! depth `d` below the node, in document order; or the root entry ---! alone, with `d = 0`, past the budget. -CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, budget integer DEFAULT 64) +--! @return SETOF public.{{prefix}}_json_tree_entry The node with `d = 0` and +--! its descendants, each with its depth `d` below it, in document +--! order; nothing when the document has no such node; or the root +--! entry alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, selector text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT * FROM {{prefix}}.json_tree_subtree_at(e, token, ARRAY[]::text[], budget) + SELECT * FROM {{prefix}}_internal.json_tree_walk(e, token, ARRAY[]::text[], budget, selector) $$; ------------------------------------------------------------------------------ diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index ee9d0fb8e..7e10539d6 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -227,6 +227,34 @@ fn connect() -> postgres::Client { .unwrap() } +/// The rows the walk `key` keys returns from row `id` of `table` (a column +/// `doc`) under `budget`: `json_tree_subtree(doc, token, selector)` for a +/// key with no gates, `json_tree_subtree_at(doc, token, gates)` for one +/// with, as a caller picks between them. +fn subtree_rows( + db: &mut postgres::Client, + table: &str, + id: i32, + key: &JsonSubtreeKey, + budget: i32, +) -> Vec { + let rows = if key.gates().is_empty() { + db.query( + &format!("SELECT e::jsonb::text FROM {table}, eql_v4.json_tree_subtree(doc, $1::text, $2::text, $3::int) e WHERE id = $4"), + &[&key.token(), &key.selector().to_string(), &budget, &id], + ) + } else { + db.query( + &format!("SELECT e::jsonb::text FROM {table}, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], $3::int) e WHERE id = $4"), + &[&key.token(), &key.gates(), &budget, &id], + ) + }; + rows.unwrap() + .iter() + .map(|r| serde_json::from_str(&r.get::<_, String>(0)).unwrap()) + .collect() +} + /// A small deterministic generator over a few keys and values, so needles /// drawn from it overlap stored documents often. struct Lcg(u64); @@ -575,25 +603,31 @@ fn postgres_extracts_walks_and_orders_by_path() { .collect(); assert_eq!(tags, vec![json!("x"), json!("y")]); - // The subtree under `$.nested`, from a token derived on the client from - // the path alone: six entries in row 1, reassembling the sub-document; - // two in row 2. + // The subtree at `$.nested`, from a token derived on the client from + // the path alone and the head's selector: nested itself and its six + // descendants in row 1, reassembling the sub-document; three in row 2. let nested = subtree_key("$.nested"); assert!(nested.gates().is_empty(), "a chain head's token"); + assert_eq!(nested.selector().to_string(), selector("$.nested")); for (id, expected) in [ (1, fixture()["nested"].clone()), (2, rows[1].1["nested"].clone()), ] { let walked: Vec = db .query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text) e WHERE id = $2", - &[&nested.token(), &id], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::text) e WHERE id = $3", + &[&nested.token(), &nested.selector().to_string(), &id], ) .unwrap() .iter() .map(entry) .collect(); - assert_eq!(walked.len(), if id == 1 { 6 } else { 2 }); + assert_eq!(walked.len(), if id == 1 { 7 } else { 3 }); + assert_eq!( + (walked[0].j, walked[0].d), + (Some(2), Some(0)), + "the head itself first, at depth 0" + ); assert!(walked .iter() .all(|e| e.d.is_some() && e.j.is_some() && e.h.is_some())); @@ -606,12 +640,22 @@ fn postgres_extracts_walks_and_orders_by_path() { } let garbage: i64 = db .query_one( - "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text) e", - &[&"00".repeat(16)], + "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::text) e", + &[&"AAAAAAAAAAAAAAAAAAAAAA==", &"AAAAAAAAAAAAAAAAAAAAAA=="], ) .unwrap() .get(0); assert_eq!(garbage, 0, "a wrong token unlocks nothing in any row"); + // A wrong token with the head's selector finds the head, which `->` + // on the selector returns anyway, and nothing beneath it. + let head_only: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::text) e", + &[&"AAAAAAAAAAAAAAAAAAAAAA==", &nested.selector().to_string()], + ) + .unwrap() + .get(0); + assert_eq!(head_only, 2, "nested itself, in the two rows that have it"); // The budget: past it the walk returns the sealed root alone, at depth // 0, and the rows open to the value at the walked path; at it, or @@ -619,17 +663,17 @@ fn postgres_extracts_walks_and_orders_by_path() { // yields nothing under any budget. let walked = |db: &mut postgres::Client, budget: i32| -> Vec { db.query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", - &[&nested.token(), &budget], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::text, $3::int) e WHERE id = 1", + &[&nested.token(), &nested.selector().to_string(), &budget], ) .unwrap() .iter() .map(entry) .collect() }; - assert_eq!(walked(&mut db, 6).len(), 6, "at the budget, the subtree"); - assert_eq!(walked(&mut db, 0).len(), 6, "budget 0 never falls back"); - let root = walked(&mut db, 5); + assert_eq!(walked(&mut db, 7).len(), 7, "at the budget, the subtree"); + assert_eq!(walked(&mut db, 0).len(), 7, "budget 0 never falls back"); + let root = walked(&mut db, 6); assert_eq!(root.len(), 1, "past the budget, the root alone"); assert_eq!((root[0].j, root[0].d), (Some(0), Some(0))); // The key that keyed the walk builds the rows, so the path the @@ -642,8 +686,8 @@ fn postgres_extracts_walks_and_orders_by_path() { assert_eq!(value, fixture()["nested"]); let garbage: i64 = db .query_one( - "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, 1) e", - &[&"00".repeat(16)], + "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::text, 1) e", + &[&"AAAAAAAAAAAAAAAAAAAAAA==", &"AAAAAAAAAAAAAAAAAAAAAA=="], ) .unwrap() .get(0); @@ -666,8 +710,8 @@ fn postgres_extracts_walks_and_orders_by_path() { .unwrap() }; let walked_members = walked(&mut db, 0); - assert_eq!(walked_members.len(), 6); - let fallen_back = walked(&mut db, 5); + assert_eq!(walked_members.len(), 7); + let fallen_back = walked(&mut db, 6); assert_eq!(fallen_back.len(), 1); let members = sorted(vec![json!("constant"), json!([{"x": 1}, {"x": 2}])]); assert_eq!(sorted(open_list(walked_members, &nested)), members); @@ -707,13 +751,13 @@ fn postgres_extracts_walks_and_orders_by_path() { "SELECT count(*) FROM json_tree_rows, eql_v4.json_tree_subtree( jsonb_set(doc::jsonb, '{h}', to_jsonb(encode(set_byte(decode(doc::jsonb ->> 'h', 'base64'), 1, 0), 'base64')))::public.eql_v4_json_tree, - $1::text, 1) e + $1::text, $2::text, 1) e WHERE id = 1", - &[&nested.token()], + &[&nested.token(), &nested.selector().to_string()], ) .unwrap() .get(0); - assert_eq!(unflagged, 6, "an unsealed root is walked out"); + assert_eq!(unflagged, 7, "an unsealed root is walked out"); // A path below the head walks from the head's token through one gate // per step: `$.nested.list[*].x` is the list's selector, then every @@ -1352,6 +1396,173 @@ fn postgres_a_position_reaches_array_elements_only() { ); } +/// Needs an EQL v4 installation. Every subtree read returns the node the +/// path names, with everything beneath it, so it answers as `jsonb` does: +/// an empty object, an empty array, a scalar (`null` included) and a node +/// the document does not have are four different answers, held here to +/// `jsonb_path_query_first(plain, 'strict ')` over the plaintexts (a +/// strict path, since `#>` reads a position as an object key). Every path +/// is read at a chain head, at an array position after one, and behind a +/// gate, with the root sealed and not, walked out (budget 0) and past a +/// budget of 1, where every read of more than one entry falls back to the +/// sealed root and is answered from the whole document. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_every_subtree_read_answers_as_jsonb_does() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = wait(&runtime, common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = connect(); + let documents = [ + json!({"p": {}, "l": [{}, [], "s", {"x": 1}], "h": {"p": {}, "l": [[], {}, null]}}), + json!({"p": [], "l": [[]], "h": {"p": []}}), + json!({"p": "text", "l": ["a", 0, false], "h": {"p": 7, "l": [true]}}), + json!({"q": 1, "h": {"q": null}}), + json!({"p": null, "l": [], "h": []}), + json!({"p": {"a": 1, "b": [2, {"c": null}]}, "l": [{"a": [1, 2]}, 5], "h": {"p": {"a": {"b": {}}}, "l": [{"a": 1}]}}), + json!({"p": [[], {}], "l": [[{}], [[]]], "h": {"l": [[{"x": []}]]}}), + ]; + let paths = [ + "$.p", + "$.q", + "$.l[0]", + "$.l[1]", + "$.l[2]", + "$.l[3]", + "$.l[0][0]", + "$.l[1][0]", + "$.h.p", + "$.h.q", + "$.h.l[0]", + "$.h.l[2]", + "$.h.l[0][0]", + ]; + let mut checked = [0usize; 4]; + for sealed_root in [true, false] { + let options = JsonOptions { + sealed_root, + ..JsonOptions::default() + }; + let table = if sealed_root { + "json_tree_reads_sealed" + } else { + "json_tree_reads_open" + }; + db.batch_execute(&format!( + "CREATE TEMP TABLE {table} (id int PRIMARY KEY, plain jsonb NOT NULL, doc public.eql_v4_json_tree NOT NULL)" + )) + .unwrap(); + for (id, plain) in (1..).zip(&documents) { + let stored = wait( + &runtime, + cipher.default_keyset().run( + JsonTree::new(options.clone()).encryption(), + plain, + column(), + ), + ) + .unwrap(); + db.execute( + &format!("INSERT INTO {table} VALUES ($1, $2::text::jsonb, $3::text::jsonb::public.eql_v4_json_tree)"), + &[&id, &plain.to_string(), &serde_json::to_string(&stored).unwrap()], + ) + .unwrap(); + } + let root_paths: &[&str] = if sealed_root { &[] } else { &["$"] }; + for path in paths.iter().chain(root_paths) { + let key = wait( + &runtime, + keyset.json_subtree_key(&JsonPath::parse(path).unwrap(), &options, column()), + ) + .unwrap(); + for id in 1..=documents.len() as i32 { + let expected: Option = db + .query_one( + &format!("SELECT jsonb_path_query_first(plain, ('strict ' || $1::text)::jsonpath, '{{}}', true)::text FROM {table} WHERE id = $2"), + &[path, &id], + ) + .unwrap() + .get::<_, Option>(0) + .map(|text| serde_json::from_str(&text).unwrap()); + checked[match &expected { + None => 0, + Some(Value::Object(o)) if o.is_empty() => 1, + Some(Value::Array(a)) if a.is_empty() => 2, + Some(_) => 3, + }] += 1; + for budget in [0, 1] { + let rows = subtree_rows(&mut db, table, id, &key, budget); + let fell_back = rows.len() == 1 && rows[0].j == Some(0) && sealed_root; + if expected.is_none() { + assert!( + rows.is_empty(), + "{table} {path}, row {id}: jsonb has no node, the walk returns no row" + ); + } else if budget == 0 || !sealed_root { + assert!( + !fell_back || *path == "$", + "{table} {path}, row {id}: walked out" + ); + } + let opened: Option = wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(&key, rows.clone()), column()), + ) + .unwrap_or_else(|error| { + panic!("{table} {path}, row {id}, budget {budget}: {error}") + }); + assert_eq!( + opened, + expected, + "{table} {path}, row {id}, budget {budget}{}", + if fell_back { + ", from the sealed root" + } else { + "" + } + ); + let one: Result = wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(&key, rows), column()), + ); + match (&expected, one) { + (Some(expected), Ok(one)) => assert_eq!(&one, expected), + (None, Err(stack_encrypt::Error::OpenedType { expected })) => { + assert_eq!(expected, "a node at the walked path") + } + (expected, one) => { + panic!("{table} {path}, row {id}: {expected:?} opened as {one:?}") + } + } + } + } + } + } + assert!( + checked.iter().all(|n| *n > 0), + "the corpus reaches no node, `{{}}`, `[]` and other values: {checked:?}" + ); + // Past the budget of 1 the sealed table answers from the root at least + // once, so the fallback is part of what was held to jsonb. + let fell_back: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_reads_sealed, eql_v4.json_tree_subtree(doc, $1::text, $2::text, 1) e WHERE (e::jsonb ->> 'j')::int = 0", + &[ + &wait(&runtime, keyset.json_subtree_key(&JsonPath::parse("$.p").unwrap(), &JsonOptions::default(), column())).unwrap().token(), + &wait(&runtime, keyset.json_selector(&JsonPath::parse("$.p").unwrap(), column())).unwrap().to_string(), + ], + ) + .unwrap() + .get(0); + assert!( + fell_back > 0, + "some read of `$.p` fell back to the sealed root" + ); +} + /// Needs an EQL v4 installation. What `->` and the walks return is checked /// by the client to be what it asked for: every entry of a row is genuine /// and opens under its own selector and position, so an answer made of the @@ -1416,14 +1627,7 @@ fn postgres_hostile_answers_are_refused_by_the_client() { }; // The walk `key` keys in row `id`, walked out (budget 0). let walked = |db: &mut postgres::Client, id: i32, key: &JsonSubtreeKey| -> Vec { - db.query( - "SELECT e::jsonb::text FROM json_tree_hostile, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 0) e WHERE id = $3", - &[&key.token(), &key.gates(), &id], - ) - .unwrap() - .iter() - .map(|r| serde_json::from_str(&r.get::<_, String>(0)).unwrap()) - .collect() + subtree_rows(db, "json_tree_hostile", id, key, 0) }; let extracted = |asked: &JsonSelector, entry: JsonTreeEntry| -> Result { wait( @@ -1497,7 +1701,11 @@ fn postgres_hostile_answers_are_refused_by_the_client() { ) .unwrap(); let rows = walked(&mut db, 1, &nested); - assert_eq!(rows.len(), 7, "the walk took number for a member"); + assert_eq!( + rows.len(), + 8, + "nested, its six descendants, and number taken for a member" + ); assert_eq!( rows.last().map(|e| (e.j, e.d)), Some((Some(9), Some(1))), @@ -1555,12 +1763,14 @@ fn postgres_walks_end_on_hostile_documents() { 'q', '{{\"sv\": [], \"el\": []}}'::jsonb))::public.eql_v4_json_tree" ) }; + // The token is the root's here, and the root carries the selector + // every entry does: `json_tree_subtree` returns it and the 300 below. let token = "AAAAAAAAAAAAAAAAAAAAAA==".to_owned(); - for (flags, budget, expected) in [(0, 0, 300), (0, 64, 300), (1, 0, 300), (1, 64, 1)] { + for (flags, budget, expected) in [(0, 0, 301), (0, 64, 301), (1, 0, 301), (1, 64, 1)] { let doc = chain(flags); let walked: i64 = db .query_one( - &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, {budget}) e"), + &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, $1::text, {budget}) e"), &[&token], ) .unwrap_or_else(|error| panic!("flags {flags}, budget {budget}: {error}")) @@ -1612,11 +1822,13 @@ fn postgres_walks_do_not_multiply_on_duplicated_entries() { }; let token = "AAAAAAAAAAAAAAAAAAAAAA==".to_owned(); let entries = 1 + 2 * LEVELS; + // `json_tree_subtree` returns the root (the start node, by its + // selector) and one row per level. for (flags, budget, expected) in [ - (0, 0, LEVELS), - (1, 0, LEVELS), - (0, 10, LEVELS), - (1, 200, LEVELS), + (0, 0, LEVELS + 1), + (1, 0, LEVELS + 1), + (0, 10, LEVELS + 1), + (1, 200, LEVELS + 1), (1, 10, 1), ] { let doc = doc(flags); @@ -1630,7 +1842,7 @@ fn postgres_walks_do_not_multiply_on_duplicated_entries() { assert_eq!(stored, entries); let walked: i64 = db .query_one( - &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, {budget}) e"), + &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, $1::text, {budget}) e"), &[&token], ) .unwrap_or_else(|error| panic!("flags {flags}, budget {budget}: {error}")) diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 409199918..69a9330c1 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -52,14 +52,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 run it with `Reason::JsonIndexPending`. `JsonPath` implements `Serialize` and `Deserialize` as its text, read back through `JsonPath::parse`. A subtree key is a token and `JsonSubtreeKey::gates()`, one gate per path step below it, a selector - or `#n:` for an array position. A walk past its budget returns + or `#n:` for an array position; a path's last step is always + a gate when it is a position, and a key with no gates (a chain head, or + the open root) carries the node's selector, `JsonSubtreeKey::selector()`, + which `json_tree_subtree(doc, token, selector)` finds the node by. Every + walk returns the node the path names, at depth `0`, with everything + beneath it, so `JsonSubtree` opens to that node's value, `{}`, `[]` and + scalars included, and, as `Option`, to `None` where the document + has no node at the path. A walk past its budget returns the sealed root in place of the subtree. The key carries the path it was derived from (`JsonSubtreeKey::path`) and builds the rows' wrapper, - `key.open(rows) -> JsonSubtree`, the wrapper's only constructor; from - its gates it knows whether the rows are below the path's node - (`json_tree_subtree`) or at the nodes it reaches (`json_tree_subtree_at`), - so the rows open to the same value either way, over `JsonPath::select`, - which evaluates a path over a value. On the wire every field the SQL reads (`s`, the pairs + `key.open(rows) -> JsonSubtree`, the wrapper's only constructor, so the + fallback opens to the same answer as the walked rows, over + `JsonPath::select`, which evaluates a path over a value. On the wire every field the SQL reads (`s`, the pairs of `q`, `p`, `op`, `h`, tokens and gates) is padded standard base64, and the ciphertext `c` is RFC 1924 base85 (`base85-simd`). The document's plaintext is held under `Protected` throughout, as the scalar paths hold diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 5af1160dc..9240a1fdf 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -1176,138 +1176,140 @@ impl DecryptInto for JsonExtracted { } } -/// Open the rows of a walk: each complete subtree the rows hold, as a -/// value, in order. What `json_tree_subtree_at` returns is one subtree per -/// node its gates reached, each rooted at depth `0`; what -/// `json_tree_subtree` returns is the children of one node, which -/// [`DecryptInto`] assembles into that node's value, or this opens -/// into its children's values. The sealed root a walk fell back to opens -/// whole, and the key's path selects over it to the same values the walked -/// rows open to: one per node the path reaches, for a key with gates -/// (`json_tree_subtree_at`); the children of the one node at the path, for -/// a key without (`json_tree_subtree`). An object's members come in an -/// order neither form fixes, and without their keys; open it as a -/// [`Value`] to keep them. Every row is checked against the key first -/// ([`JsonSubtreeKey::open`]); a row that does not fit is -/// [`Error::JsonWrongEntry`]. -impl DecryptInto> for JsonSubtree { +/// The one node a walk's rows reached, moved out of `reached` (the array +/// of every node the rows reached) into its own `Protected`; `None` when +/// they reached none, and [`Error::OpenedType`] when they reached more +/// than one, as a wildcard does. +fn one_reached(reached: &mut Protected) -> Result>, Error> { + let mut count = 0; + let one = take_protected(reached, |doc: &mut PlainJson| { + let Value::Array(items) = doc.value_mut() else { + return None; + }; + count = items.len(); + match items.as_mut_slice() { + [node] => Some(PlainJson::from_value(std::mem::take(node))), + _ => None, + } + }); + match (one, count) { + (Some(node), _) => Ok(Some(node)), + (None, 0) => Ok(None), + (None, _) => Err(shape("one node at the walked path")), + } +} + +/// Open the rows of a walk into the value of the node they reached: what +/// the path names. An object, an array or a scalar, empty containers +/// included, as `jsonb` holds them at the path. No node at the path, which +/// is no rows (or, past the budget, a document with nothing there), is +/// [`Error::OpenedType`]; open as `Option` to read that as `None`. +/// More than one node (a wildcard) is [`Error::OpenedType`] too, and opens +/// through [`DecryptInto>`] instead. Every row is checked +/// against the key first ([`JsonSubtreeKey::open`]); a row that does not +/// fit is [`Error::JsonWrongEntry`]. +impl DecryptInto for JsonSubtree { type Context = AeadContext; - fn decryption(self, context: Self::Context) -> Decryption, K> { + fn decryption(self, context: Self::Context) -> Decryption { Decryption::from_open(move |cipher| { - if self.is_sealed_root() { - let below = self.below(); - return open_fallback(self, cipher, context).try_map(move |(path, mut whole)| { - // The selected nodes move out of the document's - // `Protected` into one of their own, leaving `null` - // behind, so their plaintext is never copied; what is - // left of the document is wiped when it drops at the - // end of this closure. - let selected = take_protected(&mut whole, |doc: &mut PlainJson| { - let mut found = path.select_mut(doc.value_mut()); - let selected = if below { - match found.as_mut_slice() { - [node] => take_children(node), - _ => return None, - } - } else { - found.into_iter().map(std::mem::take).collect() - }; - Some(PlainJson::from_value(Value::Array(selected))) - }) - .ok_or_else(|| shape("one node at the walked path"))?; - Ok(match hand_over(selected) { - Value::Array(items) => items, - other => vec![other], - }) - }); - } - open_subtree(self, cipher, context) - .map(|forest| forest.into_iter().map(|(_, v)| hand_over(v)).collect()) + open_reached(self, cipher, context).try_map(|mut reached| { + one_reached(&mut reached)? + .map(hand_over) + .ok_or_else(|| shape("a node at the walked path")) + }) + }) + } +} + +/// Open the rows of a walk into the value of the node they reached, or +/// `None` when there is no node at the path: the four answers `jsonb #>` +/// gives, an object, an array, a scalar (`null` included) and SQL `NULL`, +/// as `Some` of the first three and `None`, the way `->` returns SQL `NULL` +/// for a path the row does not have. More than one node (a wildcard) is +/// [`Error::OpenedType`]. +impl DecryptInto> for JsonSubtree { + type Context = AeadContext; + fn decryption(self, context: Self::Context) -> Decryption, K> { + Decryption::from_open(move |cipher| { + open_reached(self, cipher, context) + .try_map(|mut reached| Ok(one_reached(&mut reached)?.map(hand_over))) }) } } -/// Open the rows of a walk into the one value they hold. For -/// `json_tree_subtree(doc, token)`, the rows are the descendants of the -/// token's node and the value is that node's: an array when the rows are -/// elements, an object when they are members. For a gated -/// `json_tree_subtree_at`, the rows are the node the gates reached, at -/// depth `0`, and its descendants, and the value is that node's; more than -/// one reached node (a wildcard) is [`Error::OpenedType`], and opens -/// through [`DecryptInto>`] instead. No rows is no value -/// ([`Error::OpenedType`]): an empty container and a scalar both have no -/// descendants, and `->` on the node's selector returns the node itself. -/// The sealed root a walk fell back to opens whole, and the key's path -/// ([`JsonSubtreeKey::open`]) selects the one value under it; a path that -/// reaches no node, or more than one, is [`Error::OpenedType`]. Every row -/// is checked against the key first ([`JsonSubtreeKey::open`]); a row that +/// Open the rows of a walk into a list. For a key with gates +/// (`json_tree_subtree_at`), one value per node the path reaches, in +/// document order: every element a wildcard reaches, and none where it +/// reaches nothing. For a key without (`json_tree_subtree`, a chain head +/// or the root), the values of the node's children: an array's elements, +/// or an object's member values, in the order of their keys and without +/// them (open it as a [`Value`] to keep them); none for a scalar or an +/// empty container, and [`Error::OpenedType`] when there is no node. The +/// sealed root a walk fell back to opens to the same lists. Every row is +/// checked against the key first ([`JsonSubtreeKey::open`]); a row that /// does not fit is [`Error::JsonWrongEntry`]. -impl DecryptInto for JsonSubtree { +impl DecryptInto> for JsonSubtree { type Context = AeadContext; - fn decryption(self, context: Self::Context) -> Decryption { + fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { - if self.is_sealed_root() { - return open_fallback(self, cipher, context).try_map(|(path, mut whole)| { - // The one node at the path moves out of the document's - // `Protected` into one of its own, leaving `null` - // behind, so its plaintext is never copied; what is - // left of the document is wiped when it drops at the - // end of this closure. - let mut reached = 0; - let selected = take_protected(&mut whole, |doc: &mut PlainJson| { - let mut found = path.select_mut(doc.value_mut()); - reached = found.len(); - match found.as_mut_slice() { - [node] => Some(PlainJson::from_value(std::mem::take(&mut **node))), - _ => None, - } - }); - match (selected, reached) { - (Some(value), _) => Ok(hand_over(value)), - (None, 0) => Err(shape("a node at the walked path")), - (None, _) => Err(shape("one node at the walked path")), - } - }); - } let below = self.below(); - open_subtree(self, cipher, context).try_map(move |forest| { - if !below { - let mut values: Vec> = - forest.into_iter().map(|(_, v)| v).collect(); - return match values.len() { - 1 => Ok(hand_over(values.remove(0))), - _ => Err(shape("one node at the walked path")), - }; - } - let mut members = None; - let mut elements: Vec<(u64, Protected)> = Vec::new(); - for (label, value) in forest { - match label { - Label::Key(key) => { - insert_member(members.get_or_insert_with(protected_object), key, value) - } - Label::Index(i) => elements.push((i, value)), - Label::Root => return Err(shape("a subtree below the root")), - } - } - match (members, elements.is_empty()) { - (Some(members), true) => Ok(hand_over(members)), - (None, false) => { - elements.sort_by_key(|(i, _)| *i); - let mut array = protected_array(elements.len()); - for (_, element) in elements { - push_element(&mut array, element); - } - Ok(hand_over(array)) - } - (None, true) => Err(shape("a node with children")), - (Some(_), false) => Err(shape("the children of one node")), - } + open_reached(self, cipher, context).try_map(move |mut reached| { + let list = if below { + let mut node = one_reached(&mut reached)? + .ok_or_else(|| shape("a node at the walked path"))?; + // The children move out of the node's `Protected` into + // one of their own; the node's keys stay behind and are + // wiped with it. + take_protected(&mut node, |node: &mut PlainJson| { + Some(PlainJson::from_value(Value::Array(take_children( + node.value_mut(), + )))) + }) + .ok_or_else(|| shape("a node at the walked path"))? + } else { + reached + }; + Ok(match hand_over(list) { + Value::Array(items) => items, + other => vec![other], + }) }) }) } } +/// Every node a walk's rows reached, as one JSON array under a +/// `Protected`, in document order. From walked rows, each reached node's +/// subtree is rebuilt and checked ([`place`]). From the sealed root a walk +/// fell back to, the key's path selects over the whole document, and the +/// selected nodes move out of it, leaving `null` behind, so their +/// plaintext is never copied; what is left of the document is wiped when +/// it drops. +fn open_reached<'a, K>( + subtree: JsonSubtree, + cipher: &'a StackCipher, + context: AeadContext, +) -> Pending<'a, Protected, K> { + if subtree.is_sealed_root() { + return open_fallback(subtree, cipher, context).try_map(|(path, mut whole)| { + take_protected(&mut whole, |doc: &mut PlainJson| { + let found = path.select_mut(doc.value_mut()); + Some(PlainJson::from_value(Value::Array( + found.into_iter().map(std::mem::take).collect(), + ))) + }) + .ok_or_else(|| shape("the nodes at the walked path")) + }); + } + open_subtree(subtree, cipher, context).map(|reached| { + let mut array = protected_array(reached.len()); + for node in reached { + push_element(&mut array, node); + } + array + }) +} + /// One row of a walk, opened: the node, and the selector, position and /// depth the server returned it with. The depth is the server's report and /// optional on the wire; the rows are placed by their positions and the @@ -1320,14 +1322,14 @@ struct Row { d: Option, } -/// Open every row and rebuild each complete subtree, labelled, each still -/// under its `Protected`, checking every row against the key as it goes -/// ([`place`]). +/// Open every row and rebuild the subtree of each node the rows reached, +/// each still under its `Protected`, checking every row against the key as +/// it goes ([`place`]). No rows is no node, and retrieves no key. fn open_subtree<'a, K>( subtree: JsonSubtree, cipher: &'a StackCipher, context: AeadContext, -) -> Pending<'a, Vec<(Label, Protected)>, K> { +) -> Pending<'a, Vec>, K> { let (mut nodes, key) = subtree.into_parts(); nodes.sort_by_key(|n| n.j); // A walk returns each entry once. Two rows at one ordinal are the @@ -1365,66 +1367,41 @@ fn open_subtree<'a, K>( }) } -/// Put the opened rows of a walk back into the subtrees they hold, in -/// document order, refusing any row that is not where the key says the -/// walk's rows are ([`Error::JsonWrongEntry`]). +/// Put the opened rows of a walk back into the subtrees of the nodes they +/// reached, in document order, refusing any row that is not where the key +/// says the walk's rows are ([`Error::JsonWrongEntry`]). /// /// Each entry opened under its own selector `s` and position `j`, which its /// ciphertext is bound to, so each is a genuine node of the row; what the -/// check adds is that it is the node the walk asked for. A subtree's rows +/// check adds is that it is the node the walk asked for. Every walk returns +/// the nodes the path reaches, each at depth `0` followed by everything +/// beneath it. A node the path reaches carries the path's selector, and +/// the position the path names if its last step is one. A subtree's rows /// in a document are its root and then every node beneath it, at -/// consecutive positions, each node's children in order; so the rows are -/// placed one by one, each the next position after the one before, one -/// level below its parent, and carrying the selector of its parent's path -/// extended by its own label, which the key derives from the decrypted -/// label. With gates, each subtree's root is a node the path reaches: the -/// path's selector, at depth `0`, at the position the path names if its -/// last step is one. Without gates, the rows are the children of the -/// path's node and their subtrees, side by side: each child carries the -/// selector of the path extended by its label, at depth `1`, right after -/// the subtree before it. -fn place( - rows: Vec, - key: &JsonSubtreeKey, -) -> Result)>, Error> { - let below = key.gates.is_empty(); +/// consecutive positions, each node's children in order; so the rows below +/// a reached node are placed one by one, each the next position after the +/// one before, one level below its parent, and carrying the selector of its +/// parent's path extended by its own label, which the key derives from the +/// decrypted label. +fn place(rows: Vec, key: &JsonSubtreeKey) -> Result>, Error> { let at = Protected::new(key.path.selector_bytes()); - let reached = key.selectors.of(at.clone())?; let mut rest = rows.into_iter(); - let mut forest = Vec::new(); - // Without gates, the position the next child of the path's node must - // have: right after the subtree of the one before. - let mut next: Option = None; + let mut reached = Vec::new(); while let Some(top) = rest.next() { - let (bytes, depth) = if below { - let bytes = child_selector_bytes(&at, &top.node.label) - .ok_or_else(|| wrong("a child of the walked node"))?; - if top.d.is_some_and(|d| d != 1) - || next.is_some_and(|j| j != top.j) - || key.selectors.of(bytes.clone())? != top.s - { - return Err(wrong("a child of the walked node")); - } - (bytes, 1) - } else { - if top.d.is_some_and(|d| d != 0) - || top.s != reached - || !reaches(&key.path, &top.node.label) - { - return Err(wrong("a node the walked path reaches")); - } - (at.clone(), 0) - }; + if top.d.is_some_and(|d| d != 0) + || top.s != key.selector + || !reaches(&key.path, &top.node.label) + { + return Err(wrong("a node the walked path reaches")); + } let mut cursor = top .j .checked_add(1) - .ok_or_else(|| wrong("a node of the walk"))?; - let (label, node) = take_label(top.node); - let value = rebuild_walked(node, &bytes, depth, &mut rest, &mut cursor, &key.selectors)?; - next = Some(cursor); - forest.push((label, value)); + .ok_or_else(|| wrong("a node the walked path reaches"))?; + let value = rebuild_walked(top.node, &at, 0, &mut rest, &mut cursor, &key.selectors)?; + reached.push(value); } - Ok(forest) + Ok(reached) } /// Whether a node with `label` is one `path` reaches by its last step, @@ -1544,8 +1521,8 @@ fn rebuild_walked( /// an array's elements, an object's member values (each leaving `null` /// under its key, which stays in the document to be wiped with it), and /// none for a scalar. What a `json_tree_subtree` walk's rows open to as a -/// [`Vec`], taken out of the whole document when the walk fell back to the -/// sealed root. +/// [`Vec`], taken out of the node the rows reached, or out of the whole +/// document when the walk fell back to the sealed root. fn take_children(node: &mut Value) -> Vec { match node { Value::Array(items) => std::mem::take(items), @@ -1585,18 +1562,28 @@ fn open_fallback<'a, K>( // ============================================================================= /// The key that walks `path`: the token of the deepest node on it the -/// client can derive, and one gate per step below that. +/// client can derive short of the path's last step, unless the path names +/// a chain head (or, with the root open, the root) itself, and one gate per +/// step below that. /// /// A chain head's token is a PRF of its path. An array element's is one /// hash under its parent's, by its position and the element selector, so -/// the client follows array steps as far as they run from the head. An -/// object member's ordinal is a rank among its siblings' keys, which the -/// client does not know, so from the first key step on the client cannot -/// derive tokens and the server walks instead, through gates: the step's -/// selector for a key or a wildcard (every element's selector is the -/// wildcard's, so a wildcard admits them all), and for an array step the -/// position with the element selector, so a member that happens to sit at -/// that ordinal under an object is not taken for an element. +/// the client follows array steps as far as they run from the head, except +/// a position that is the path's last step: that one is a gate, so that the +/// walk reaches the element and returns it. A node is found by its parent's +/// token and its ordinal, and nothing stored names a node by its own +/// token, so a walk from an element's token could return only what is +/// beneath it, and an empty element, a scalar one and a missing one would +/// all come back as no rows. A chain head (and the root) has no parent +/// token, and is found by its selector instead, which the key carries +/// ([`JsonSubtreeKey::selector`]). An object member's ordinal is a rank +/// among its siblings' keys, which the client does not know, so from the +/// first key step on the client cannot derive tokens and the server walks +/// instead, through gates: the step's selector for a key or a wildcard +/// (every element's selector is the wildcard's, so a wildcard admits them +/// all), and for an array step the position with the element selector, so +/// a member that happens to sit at that ordinal under an object is not +/// taken for an element. fn subtree_key_for( selectors: Selectors, options: &JsonOptions, @@ -1618,16 +1605,29 @@ fn subtree_key_for( let head_bytes = head.token_bytes().ok_or(TermError::JsonSealedRoot)?; let mut token = prf16(prf, TOKEN_DOMAIN, context, &head_bytes)?; let mut prefix = head; - let mut rest = rest.iter(); + // The last step is never chained: a position there is a gate, so the + // walk returns the element it reaches. + let (chained, last) = match rest.split_last() { + Some((last, before)) => (before, Some(last)), + None => (rest, None), + }; + let mut rest = chained.iter().chain(last); let mut gates = Vec::new(); - for segment in rest.by_ref() { + for (i, segment) in rest.by_ref().enumerate() { prefix = prefix.then(segment.clone()); match segment { - Segment::Index(index) => { + Segment::Index(index) if i < chained.len() => { let o = fit_ordinal(*index)?; let element = prf16(prf, SELECTOR_DOMAIN, context, &prefix.selector_bytes())?; token = child_token(&token, &element, o); } + Segment::Index(index) => { + gates.push(Gate::Slot( + fit_ordinal(*index)?, + prf16(prf, SELECTOR_DOMAIN, context, &prefix.selector_bytes())?, + )); + break; + } Segment::Key(_) | Segment::Wildcard => { gates.push(Gate::Selector(prf16( prf, @@ -1654,10 +1654,12 @@ fn subtree_key_for( )?), }); } + let selector = prf16(prf, SELECTOR_DOMAIN, context, &path.selector_bytes())?; Ok(JsonSubtreeKey { token, gates, path: path.clone(), + selector, selectors, }) } @@ -1831,11 +1833,30 @@ mod tests { }; let options = JsonOptions::default(); let t = terms(&value, &options); - // A head's token, and the positions under it, derive exactly. - for path in ["$.a", "$.l", "$.l[0]", "$.l[0][0]", "$.l[1]"] { + // A head's token derives exactly, and the key carries the head's + // selector, which finds the head itself. + for path in ["$.a", "$.l"] { let k = key(&options, path).unwrap(); assert_eq!(Some(*k.token_bytes()), at(&t, path).token, "{path}"); assert!(k.gates().is_empty(), "{path}"); + assert_eq!(k.selector().as_bytes(), &at(&t, path).s, "{path}"); + } + // The positions under a head derive too, all but a last one, which + // is a gate from its array's token, so the walk returns the element. + for (path, array) in [ + ("$.l[0]", "$.l"), + ("$.l[1]", "$.l"), + ("$.l[0][0]", "$.l[0]"), + ] { + let k = key(&options, path).unwrap(); + assert_eq!(Some(*k.token_bytes()), at(&t, array).token, "{path}"); + let position = path.rsplit('[').next().unwrap().trim_end_matches(']'); + assert_eq!( + k.gates(), + [format!("#{position}:{}", base64(&at(&t, path).s))], + "{path}" + ); + assert_eq!(k.selector().as_bytes(), &at(&t, path).s, "{path}"); } // The server derives an object member's token the same way, from // its parent's, its selector and its ordinal. @@ -1900,9 +1921,16 @@ mod tests { // as a gate it can cast, and finds nothing there. let last = key("$.a.b[2147483647]").unwrap(); assert!(last.gates()[1].starts_with("#2147483647:")); + let last = key("$.a[2147483647]").unwrap(); + assert!(last.gates()[0].starts_with("#2147483647:")); // One past it fits u32 but not integer: refused here, as a gate and // as a position the client chains itself. - for path in ["$.a.b[2147483648]", "$.a[2147483648]", "$.a[4294967296]"] { + for path in [ + "$.a.b[2147483648]", + "$.a[2147483648]", + "$.a[4294967296]", + "$.a[2147483648][0]", + ] { assert!(matches!(key(path), Err(TermError::JsonTooWide)), "{path}"); } assert_eq!(fit_ordinal(i32::MAX.unsigned_abs()).unwrap(), 2_147_483_647); diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index a4ee371c0..5e044cebf 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -462,7 +462,9 @@ impl JsonDocument { /// Every entry under the node whose token is `token`, with its depth /// below that node, as the server's recursive walk finds them: the node /// itself is not included, since the walk addresses nodes by their - /// parent. In depth-first order, siblings by ordinal. + /// parent (`json_tree_subtree` adds a chain head by its selector, and a + /// gated walk the node its last gate admits). In depth-first order, + /// siblings by ordinal. /// /// A stored document is untrusted, and its tags need not form a tree: /// whoever holds a token can write a chain deeper than any document the @@ -618,28 +620,23 @@ where } /// The entries the server's walk returned for one key, ready to open: -/// what `json_tree_subtree(doc, token)` or `json_tree_subtree_at(doc, -/// token, gates)` yields, collected, and what +/// what `json_tree_subtree(doc, token, selector)` or +/// `json_tree_subtree_at(doc, token, gates)` yields, collected, and what /// [`DecryptInto`](crate::DecryptInto) reassembles into the value -/// under the walked path. Built only by the key that keyed the walk +/// at the walked path. Built only by the key that keyed the walk /// ([`JsonSubtreeKey::open`]), so it always carries the walked path, and /// what checks every row against it when the rows are opened (see /// [`JsonSubtreeKey::open`] for the checks). /// -/// The ungated walk returns the node's descendants and not the node -/// itself, so the node's kind is read from its children's labels: elements -/// make an array, members an object. A node with no descendants, an empty -/// container or a scalar, yields no rows and so no value; `->` on its -/// selector returns the node itself. A gated walk also returns each node -/// its gates reach, at depth `0`, with that node's descendants beneath it. +/// Every walk returns each node the path reaches at depth `0`, with that +/// node's descendants beneath it, so the rows open to the node's own +/// value: `{}`, `[]` and a scalar each come back as themselves, and no rows +/// is no node at the path, which `Option` opens to `None`. /// /// Past its budget, a walk over a document with a sealed root returns the /// root entry alone, at depth `0`, instead of a subtree it would be /// cheaper to open whole: the client opens the document and takes the -/// walked path over it, to the same value the walked rows open to. Whether -/// the path names the node walked below (`json_tree_subtree`) or the nodes -/// reached (`json_tree_subtree_at`) is the key's to say, by whether it has -/// gates, so no caller attaches it. +/// walked path over it, to the same value the walked rows open to. #[derive(Clone, Debug, PartialEq, Eq)] pub struct JsonSubtree { nodes: Vec, @@ -647,9 +644,9 @@ pub struct JsonSubtree { } impl JsonSubtree { - /// The rows of a walk `key` keyed: below the path's node when the key - /// has no gates (`json_tree_subtree`), at the nodes it reaches - /// otherwise. + /// The rows of a walk `key` keyed: the nodes the key's path reaches, + /// each with everything beneath it, from `json_tree_subtree` for a key + /// with no gates and `json_tree_subtree_at` for one with. pub(crate) fn keyed(nodes: Vec, key: JsonSubtreeKey) -> Self { Self { nodes, key } } @@ -664,9 +661,9 @@ impl JsonSubtree { &self.key.path } - /// Whether the rows are the descendants of the path's node - /// (`json_tree_subtree`, a key with no gates) rather than the nodes - /// the path reaches. + /// Whether the key has no gates: it names a chain head or the root, its + /// walk is `json_tree_subtree`, and its rows open as a list to the + /// node's children rather than to the nodes the path reaches. pub(crate) fn below(&self) -> bool { self.key.gates.is_empty() } diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index 1600fbe47..d650f76f4 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -375,8 +375,21 @@ impl Gate { /// The key of a subtree walk: the token of the deepest node on the path /// the client can derive, one gate per step below it, and the path it was /// derived from, which [`open`](Self::open) attaches to the rows the walk -/// returns. Serializes as `{"t": "…", "g": ["…", "#0:…", …]}`, `g` omitted -/// when empty; the path is not part of the wire. +/// returns. Serializes as `{"t": "…", "g": ["…", "#0:…", …]}` for a key +/// with gates and `{"t": "…", "s": "…"}` for one without; the path is not +/// part of the wire. +/// +/// Every walk returns the node the path names (each node, for a +/// wildcard) at depth `0`, with everything beneath it, so an empty object, +/// an empty array, a scalar and a missing node are four different answers. +/// A key with [`gates`](Self::gates) keys `json_tree_subtree_at(doc, +/// token, gates)`, which reaches the node through them. A key without +/// names a chain head, or with the root open the root, which no parent +/// token reaches: it keys `json_tree_subtree(doc, token, selector)`, which +/// finds the node by its [`selector`](Self::selector) and its descendants +/// by the token. A path that ends in an array position walks from the +/// array's token through a position gate, so that the element itself comes +/// back. /// /// The key also holds what derives selectors under the column it was /// derived for (the keyset's index-key PRF, shared with the keyset handle, @@ -388,6 +401,8 @@ pub struct JsonSubtreeKey { pub(crate) token: Tag, pub(crate) gates: Vec, pub(crate) path: JsonPath, + /// The selector of `path`: what the nodes the walk reaches carry. + pub(crate) selector: Tag, pub(crate) selectors: Selectors, } @@ -407,12 +422,26 @@ impl JsonSubtreeKey { /// The gates the walk descends through below the token, one per step, /// as `json_tree_subtree_at` takes them: a selector's base64, or /// `#n:` for an array position, the selector being the array - /// elements' own. Empty when the token names the walked node itself, - /// which `json_tree_subtree` takes. + /// elements' own. Empty when the token names the walked node itself (a + /// chain head, or the open root), whose walk is `json_tree_subtree`. pub fn gates(&self) -> Vec { self.gates.iter().map(Gate::text).collect() } + /// The selector of the key's path: what `json_tree_subtree(doc, token, + /// selector)` takes beside the token when the key has no gates, to + /// return the node itself. A chain head has no parent token, so the + /// walk finds it by its selector, which no other top-level node + /// carries. It is the selector `->` sends for the same path, so the + /// database learns nothing from it it could not from `->`; with the + /// token beside it, it learns which top-level field the token is for, + /// which the stored order shows it the first time the token reaches + /// one of the field's members (the entry before the first member is its + /// parent). + pub fn selector(&self) -> JsonSelector { + JsonSelector(self.selector) + } + /// The path the key was derived from. pub fn path(&self) -> &JsonPath { &self.path @@ -420,12 +449,13 @@ impl JsonSubtreeKey { /// The rows the walk this key keyed returned, ready to open: the only /// way to build a [`JsonSubtree`], so the rows always carry the path - /// a sealed-root fallback is opened at. With no gates the rows came - /// from `json_tree_subtree(doc, token)` and are the descendants of the - /// path's node; with gates they came from `json_tree_subtree_at(doc, - /// token, gates)` and are the nodes the path reaches, with theirs. The - /// key knows which, so the fallback opens to what the walked rows - /// open to either way. + /// a sealed-root fallback is opened at. The rows are the nodes the path + /// reaches, each at depth `0` with everything beneath it, from + /// `json_tree_subtree(doc, token, selector)` for a key with no gates and + /// `json_tree_subtree_at(doc, token, gates)` for one with; no rows means + /// no node at the path. Past the walk's budget they are the sealed root + /// alone, and the key's path selects over the whole document to the + /// same answer. /// /// Opening the rows checks each one against the key, and refuses the /// rows with [`Error::JsonWrongEntry`] if one does not fit: a node the @@ -438,11 +468,9 @@ impl JsonSubtreeKey { /// A row whose parent is not among the rows is refused. /// /// Not covered: which row of the column the entries come from (ADR - /// 0010), and, until the walk returns the node it starts from, which - /// node at the path's selector a walk with no gates descends from. An - /// array position before the last step of a path reads as the wildcard - /// to the selectors, so another element's subtree at the same depth - /// passes too. + /// 0010). An array position before the last step of a path reads as + /// the wildcard to the selectors, so another element's subtree at the + /// same depth passes too. pub fn open(&self, rows: Vec) -> JsonSubtree { JsonSubtree::keyed(rows, self.clone()) } @@ -464,10 +492,13 @@ impl Serialize for JsonSubtreeKey { t: String, #[serde(skip_serializing_if = "Vec::is_empty")] g: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + s: Option, } Wire { t: base64(&self.token), g: self.gates(), + s: self.gates.is_empty().then(|| base64(&self.selector)), } .serialize(s) } @@ -828,6 +859,7 @@ mod tests { token: [0xcd; 16], gates: vec![Gate::Selector([0xab; 16]), Gate::Slot(3, [0xab; 16])], path: JsonPath::parse("$.a.b[3]").unwrap(), + selector: [0xab; 16], selectors: Selectors::for_tests(), }; assert_eq!(key.token(), "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="); @@ -849,12 +881,15 @@ mod tests { token: [0xcd; 16], gates: Vec::new(), path: JsonPath::parse("$.a").unwrap(), + selector: [0xab; 16], selectors: Selectors::for_tests(), }; assert_eq!( serde_json::to_value(&bare).unwrap(), - serde_json::json!({"t": "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="}) + serde_json::json!({"t": "zc3Nzc3Nzc3Nzc3Nzc3NzQ==", "s": "q6urq6urq6urq6urq6urqw=="}), + "a key with no gates carries the head's selector" ); + assert_eq!(bare.selector(), JsonSelector([0xab; 16])); } #[test] diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index 420e82254..e0d1fa088 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -516,10 +516,14 @@ async fn a_token_derived_from_the_path_alone_unlocks_exactly_its_subtree() { assert_eq!(open_key.token_bytes(), root.token_bytes()); assert_eq!(open_key.gates().len(), 3); let array_doc = seal_with(&cipher, &json!([[1, 2], [3]]), &open_root()).await; - let second_row = subtree_key(&cipher, "$[1]", open_root()).await.unwrap(); - assert!(second_row.gates().is_empty(), "positions derive"); + let below_second = subtree_key(&cipher, "$[1][0]", open_root()).await.unwrap(); + assert_eq!( + below_second.gates().len(), + 1, + "positions derive up to the last, which is a gate" + ); let found: Vec = array_doc - .subtree(second_row.token_bytes()) + .subtree(below_second.token_bytes()) .iter() .map(|(j, _, _)| *j) .collect(); @@ -598,11 +602,7 @@ async fn an_extracted_node_opens_to_its_scalar_and_a_subtree_to_its_value() { let key = subtree_key(&cipher, "$.nested", JsonOptions::default()) .await .unwrap(); - let rows: Vec = doc - .subtree(key.token_bytes()) - .iter() - .map(|(j, _, _)| node(*j)) - .collect(); + let rows = gated_walk(&doc, &key); let nested = cipher .decrypt_as::(key.open(rows.clone()), aead()) .await @@ -1463,17 +1463,21 @@ async fn a_position_reaches_no_object_member() { let first = subtree_key(&cipher, "$.a[0]", JsonOptions::default()) .await .unwrap(); - assert!( - first.gates().is_empty(), - "a position after the head derives" + assert_eq!( + first.gates().len(), + 1, + "a last position is a gate from its array's token" ); let array = seal(&cipher, &json!({"a": [{"x": 2}]})).await; - let found: Vec = array - .subtree(first.token_bytes()) + let found: Vec = gated_walk(&array, &first) .iter() - .map(|(j, _, _)| *j) + .map(JsonNode::ordinal) .collect(); - assert_eq!(found, [3], "$, $.a, $.a[0], $.a[0].x"); + assert_eq!(found, [2, 3], "$, $.a, $.a[0], $.a[0].x"); + // A position the client chains, before the last step. + let below = subtree_key(&cipher, "$.a[0].x", JsonOptions::default()) + .await + .unwrap(); for object in [ json!({"a": {"k": {"x": 1}}}), json!({"a": {"k": [1, 2]}}), @@ -1481,9 +1485,13 @@ async fn a_position_reaches_no_object_member() { ] { let doc = seal(&cipher, &object).await; assert!( - doc.subtree(first.token_bytes()).is_empty(), + gated_walk(&doc, &first).is_empty(), "{object}: an object member is not an element" ); + assert!( + gated_walk(&doc, &below).is_empty(), + "{object}: nor beneath one" + ); } // The same below a key step, where the position is a gate: the walk // from the head reaches `$.h.a` by its selector, and the gate at the @@ -1722,13 +1730,32 @@ async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { /// walked here as the SQL walks them: from the key's token, through each /// gate (a selector the entry at that depth must carry, or `#n:`, /// the child at ordinal `n` carrying it), then everything beneath each node -/// the gates reach, each row with its depth below that node. +/// the gates reach, each row with its depth below that node. For a key +/// with no gates, what `json_tree_subtree(doc, token, selector)` returns: +/// the entry with no parent tag that carries the key's selector, then its +/// descendants. fn gated_walk(doc: &JsonDocument, key: &JsonSubtreeKey) -> Vec { let with_depth = |j: usize, d: u32| -> JsonNode { let mut wire = serde_json::to_value(doc.node(j).unwrap()).unwrap(); wire["d"] = json!(d); serde_json::from_value(wire).unwrap() }; + if key.gates().is_empty() { + let Some(head) = doc + .entries() + .iter() + .position(|e| e.parent_tag().is_none() && e.selector() == key.selector().as_bytes()) + else { + return Vec::new(); + }; + let mut rows = vec![with_depth(head, 0)]; + rows.extend( + doc.subtree(key.token_bytes()) + .iter() + .map(|(j, _, d)| with_depth(*j, *d)), + ); + return rows; + } let mut frontier = vec![(None, *key.token_bytes())]; for gate in key.gates() { let mut next = Vec::new(); @@ -1796,7 +1823,7 @@ async fn the_key_built_fallback_opens_to_what_the_walk_opens_to() { ("$.list", false), ("$.obj", false), ("$.hello", false), - ("$.list[1]", false), + ("$.list[1]", true), ("$.obj.b", true), ("$.obj.c.d", true), ("$.obj.b[0]", true), @@ -1808,14 +1835,7 @@ async fn the_key_built_fallback_opens_to_what_the_walk_opens_to() { .await .unwrap(); assert_eq!(!key.gates().is_empty(), gated, "{text}"); - let walked = if gated { - gated_walk(&doc, &key) - } else { - doc.subtree(key.token_bytes()) - .iter() - .map(|(j, _, _)| doc.node(*j).unwrap()) - .collect() - }; + let walked = gated_walk(&doc, &key); let fallen_back = key.open(vec![root.clone()]); assert!(fallen_back.is_sealed_root(), "{text}"); let as_walked = cipher @@ -1828,11 +1848,6 @@ async fn the_key_built_fallback_opens_to_what_the_walk_opens_to() { .unwrap(); // An object's members come back in an order neither form fixes. assert_eq!(sorted(as_fallen), sorted(as_walked), "{text}, as a list"); - // A walk with no rows (a scalar's, or an empty container's) is - // never past a budget, so it never falls back: nothing to compare. - if walked.is_empty() { - continue; - } let one_walked = cipher .decrypt_as::(key.open(walked), aead()) .await; @@ -1866,7 +1881,7 @@ async fn the_key_built_fallback_opens_to_what_the_walk_opens_to() { .await { Err(Error::OpenedType { expected }) => { - assert_eq!(expected, "one node at the walked path") + assert_eq!(expected, "a node at the walked path") } other => panic!("{other:?}"), } diff --git a/packages/stack-encrypt/tests/json_read_checks.rs b/packages/stack-encrypt/tests/json_read_checks.rs index 963b37ac8..e9f2b8db5 100644 --- a/packages/stack-encrypt/tests/json_read_checks.rs +++ b/packages/stack-encrypt/tests/json_read_checks.rs @@ -118,16 +118,28 @@ fn row(doc: &JsonDocument, j: usize, d: Option) -> JsonNode { } /// The rows the SQL walk returns for `key`, walked as it walks: with no -/// gates, every descendant of the token's node from depth 1; with gates, -/// through each gate (a selector, or `#n:`), then each node the -/// gates reach at depth 0 with everything beneath it. +/// gates (`json_tree_subtree`), the entry with no parent tag that carries +/// the key's selector at depth 0, then every descendant of the token's +/// node, or nothing when no such entry exists; with gates +/// (`json_tree_subtree_at`), through each gate (a selector, or +/// `#n:`), then each node the gates reach at depth 0 with +/// everything beneath it. fn walk(doc: &JsonDocument, key: &JsonSubtreeKey) -> Vec { let gates = key.gates(); if gates.is_empty() { - return doc - .subtree(key.token_bytes()) + let Some(head) = doc + .entries() .iter() - .map(|(j, _, d)| row(doc, *j, Some(*d))) + .position(|e| e.parent_tag().is_none() && e.selector() == key.selector().as_bytes()) + else { + return Vec::new(); + }; + return std::iter::once(row(doc, head, Some(0))) + .chain( + doc.subtree(key.token_bytes()) + .iter() + .map(|(j, _, d)| row(doc, *j, Some(*d))), + ) .collect(); } let mut frontier = vec![(None, *key.token_bytes())]; @@ -294,7 +306,7 @@ mod given_the_entry_arrow_returned { mod given_the_rows_of_a_walk_without_gates { use super::*; - const CHILD: &str = "a child of the walked node"; + const REACHED: &str = "a node the walked path reaches"; const NEXT: &str = "the next node of the walked subtree"; #[tokio::test] @@ -304,7 +316,7 @@ mod given_the_rows_of_a_walk_without_gates { let nested = key(&cipher, "$.nested").await; assert!(nested.gates().is_empty(), "a chain head's token"); let rows = walk(&doc, &nested); - assert_eq!(rows.len(), 6, "nested's descendants"); + assert_eq!(rows.len(), 7, "nested and its descendants"); let value = open_value(&cipher, nested.open(rows.clone())).await; assert_eq!(value.unwrap(), fixture()["nested"], "the sub-document"); let list = open_list(&cipher, nested.open(rows)).await; @@ -321,10 +333,10 @@ mod given_the_rows_of_a_walk_without_gates { let doc = seal(&cipher, &json!({"a": {"x": 1}, "b": {"x": 2}})).await; let a = key(&cipher, "$.a").await; let b = key(&cipher, "$.b").await; - // The database answers `$.a` with `$.b`'s subtree: one genuine - // member, in its place, but under `$.b`'s path. + // The database answers `$.a` with `$.b`'s subtree: genuine + // entries, in their place, but at `$.b`'s path. let refused = open_value(&cipher, a.open(walk(&doc, &b))).await; - assertions::wrong_entry(refused, CHILD); + assertions::wrong_entry(refused, REACHED); } #[tokio::test] @@ -338,7 +350,7 @@ mod given_the_rows_of_a_walk_without_gates { let mut rows = walk(&doc, &nested); rows.push(row(&doc, 9, Some(1))); let refused = open_value(&cipher, nested.open(rows)).await; - assertions::wrong_entry(refused, CHILD); + assertions::wrong_entry(refused, REACHED); } #[tokio::test] @@ -391,7 +403,7 @@ mod given_the_rows_of_a_walk_without_gates { }) .collect(); let refused = open_value(&cipher, nested.open(rows)).await; - assertions::wrong_entry(refused, CHILD); + assertions::wrong_entry(refused, NEXT); } #[tokio::test] @@ -399,11 +411,26 @@ mod given_the_rows_of_a_walk_without_gates { let (cipher, _) = cipher().await; let doc = seal(&cipher, &fixture()).await; let nested = key(&cipher, "$.nested").await; - // `deep` (3) and then the second element's `x` (8) alone: its - // parent and everything between are missing. - let rows = vec![row(&doc, 3, Some(1)), row(&doc, 8, Some(3))]; + // nested, `deep` (3) and then the second element's `x` (8) alone: + // its parent and everything between are missing. + let rows = vec![ + row(&doc, 2, Some(0)), + row(&doc, 3, Some(1)), + row(&doc, 8, Some(3)), + ]; let refused = open_value(&cipher, nested.open(rows)).await; - assertions::wrong_entry(refused, CHILD); + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn the_descendants_without_the_node_are_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // What the walk returned before it returned the node itself. + let rows: Vec = walk(&doc, &nested).into_iter().skip(1).collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, REACHED); } #[tokio::test] @@ -418,7 +445,7 @@ mod given_the_rows_of_a_walk_without_gates { .filter(|n| n.ordinal() != 3) .collect(); let refused = open_value(&cipher, a.open(rows)).await; - assertions::wrong_entry(refused, CHILD); + assertions::wrong_entry(refused, NEXT); } #[tokio::test] @@ -433,7 +460,7 @@ mod given_the_rows_of_a_walk_without_gates { let doc = seal_with(&cipher, &value, &open_root()).await; let root = key_with(&cipher, "$", &open_root()).await; let rows = walk(&doc, &root); - assert_eq!(rows.len(), 127, "every node below the root"); + assert_eq!(rows.len(), 128, "the root and every node below it"); let opened = open_value(&cipher, root.open(rows)).await; assert_eq!(opened.unwrap(), value, "the whole document"); } @@ -529,3 +556,215 @@ mod given_the_rows_of_a_gated_walk { assertions::wrong_entry(refused, REACHED); } } + +/// Every subtree read returns the node the path names, so a node with +/// nothing beneath it (`{}`, `[]`, a scalar) opens to itself, and no node +/// is told apart from all three: `Option` opens to `None`, as `->` +/// returns SQL `NULL`. The sealed root a walk falls back to past its budget +/// opens to the same answers. +mod given_a_node_with_nothing_beneath_it { + use super::*; + + async fn open_option(cipher: &Cipher, rows: JsonSubtree) -> Result, Error> { + cipher + .decrypt_as::, JsonSubtree>(rows, aead()) + .await + } + + /// The rows a walk past its budget returns: the sealed root alone. + fn fallback(doc: &JsonDocument) -> Vec { + vec![row(doc, 0, Some(0))] + } + + /// `path` in `value` read every way the client can read it, walked and + /// (with the root sealed) from the fallback, against what the path + /// selects in the plaintext. + async fn holds(cipher: &Cipher, value: &Value, path: &str, options: &JsonOptions) { + let doc = seal_with(cipher, value, options).await; + let key = key_with(cipher, path, options).await; + let expected = JsonPath::parse(path) + .unwrap() + .select(value) + .first() + .map(|v| (*v).clone()); + let mut answers = vec![("walked", walk(&doc, &key))]; + if options.sealed_root { + answers.push(("past the budget", fallback(&doc))); + } + for (how, rows) in answers { + let opened = open_option(cipher, key.open(rows.clone())).await; + assert_eq!(opened.unwrap(), expected, "{path} in {value}, {how}"); + let one = open_value(cipher, key.open(rows)).await; + match &expected { + Some(expected) => assert_eq!(&one.unwrap(), expected, "{path} in {value}, {how}"), + None => match one { + Err(Error::OpenedType { expected }) => assert_eq!( + expected, "a node at the walked path", + "{path} in {value}, {how}: no node is no value" + ), + other => panic!("{path} in {value}, {how}: {other:?}"), + }, + } + } + } + + const NOTHING_BENEATH: [&str; 7] = [ + "{}", + "[]", + "\"text\"", + "0", + "null", + "false", + "{\"a\": [1, {\"b\": null}]}", + ]; + + fn each_node() -> impl Iterator { + NOTHING_BENEATH + .iter() + .map(|text| serde_json::from_str(text).unwrap()) + } + + #[tokio::test] + async fn a_chain_head_opens_to_its_own_value() { + let (cipher, _) = cipher().await; + for node in each_node() { + holds( + &cipher, + &json!({"p": node, "q": 1}), + "$.p", + &JsonOptions::default(), + ) + .await; + } + } + + #[tokio::test] + async fn an_array_position_after_a_head_opens_to_its_own_value() { + let (cipher, _) = cipher().await; + let elements: Vec = each_node().collect(); + let value = json!({ "l": elements }); + for n in 0..=elements.len() { + holds( + &cipher, + &value, + &format!("$.l[{n}]"), + &JsonOptions::default(), + ) + .await; + } + // A position in an array of arrays, chained through the first. + let nested = json!({"l": [[{}, [], 3]]}); + for n in 0..4 { + holds( + &cipher, + &nested, + &format!("$.l[0][{n}]"), + &JsonOptions::default(), + ) + .await; + } + } + + #[tokio::test] + async fn a_node_behind_a_gate_opens_to_its_own_value() { + let (cipher, _) = cipher().await; + for node in each_node() { + let value = json!({"h": {"p": node.clone(), "l": [node]}}); + for path in ["$.h.p", "$.h.l[0]", "$.h.l[1]", "$.h.q"] { + holds(&cipher, &value, path, &JsonOptions::default()).await; + } + } + } + + #[tokio::test] + async fn no_node_opens_to_none_and_retrieves_no_key() { + let (cipher, retrieved) = cipher().await; + for path in ["$.p", "$.l[0]", "$.h.p", "$.h.l[0]"] { + holds( + &cipher, + &json!({"q": 1, "h": []}), + path, + &JsonOptions::default(), + ) + .await; + } + let doc = seal(&cipher, &json!({"q": 1})).await; + let p = key(&cipher, "$.p").await; + assert!(walk(&doc, &p).is_empty(), "no entry carries `$.p`"); + let before = retrieved.load(Ordering::SeqCst); + let opened = open_option(&cipher, p.open(Vec::new())).await; + assert_eq!(opened.unwrap(), None, "no node"); + assert_eq!( + retrieved.load(Ordering::SeqCst), + before, + "no rows retrieve no key" + ); + } + + #[tokio::test] + async fn with_the_root_open_every_read_opens_to_its_own_value() { + let (cipher, _) = cipher().await; + for node in each_node() { + let value = json!({"p": node.clone(), "l": [node]}); + for path in ["$", "$.p", "$.l[0]", "$.l[1]", "$.q"] { + holds(&cipher, &value, path, &open_root()).await; + } + } + } + + #[tokio::test] + async fn a_heads_children_open_as_a_list_walked_and_past_the_budget() { + let (cipher, _) = cipher().await; + for (node, children) in [ + (json!({}), vec![]), + (json!([]), vec![]), + (json!("text"), vec![]), + (json!([1, {}]), vec![json!(1), json!({})]), + (json!({"b": [], "a": 1}), vec![json!(1), json!([])]), + ] { + let doc = seal(&cipher, &json!({"p": node})).await; + let p = key(&cipher, "$.p").await; + for (how, rows) in [ + ("walked", walk(&doc, &p)), + ("past the budget", fallback(&doc)), + ] { + let list = open_list(&cipher, p.open(rows)).await; + assert_eq!(list.unwrap(), children, "{node}, {how}"); + } + } + let doc = seal(&cipher, &json!({"q": 1})).await; + let p = key(&cipher, "$.p").await; + for (how, rows) in [ + ("walked", walk(&doc, &p)), + ("past the budget", fallback(&doc)), + ] { + match open_list(&cipher, p.open(rows)).await { + Err(Error::OpenedType { expected }) => { + assert_eq!(expected, "a node at the walked path", "{how}") + } + other => panic!("{how}: no node has no children to list: {other:?}"), + } + } + } + + #[tokio::test] + async fn a_wildcard_reaching_nodes_with_nothing_beneath_them_lists_each() { + let (cipher, _) = cipher().await; + let elements: Vec = each_node().collect(); + let doc = seal(&cipher, &json!({"h": {"l": elements.clone()}})).await; + let all = key(&cipher, "$.h.l[*]").await; + for (how, rows) in [ + ("walked", walk(&doc, &all)), + ("past the budget", fallback(&doc)), + ] { + let list = open_list(&cipher, all.open(rows.clone())).await; + assert_eq!(list.unwrap(), elements, "{how}"); + match open_option(&cipher, all.open(rows)).await { + Err(Error::OpenedType { expected }) => { + assert_eq!(expected, "one node at the walked path", "{how}") + } + other => panic!("{how}: a wildcard is no one node: {other:?}"), + } + } + } +}