fix(eql): return the node itself from every json_tree subtree read

.changeset/eql-json-tree-empty-container.md

@@ -0,0 +1,5 @@
+---
+"@cipherstash/eql": minor
+---
+
+EQL v4 `json_tree` (unreleased): every subtree read now returns the node it names, so an empty object, an empty array, a scalar and a missing field are four different answers, as `jsonb` gives them. The walk for a top-level field, `eql_v4.json_tree_subtree`, returned the field's descendants and never the field itself, because a walk finds a node by its parent's token and a top-level field has none; so `{}`, `[]`, `"text"` and no field at all each came back as no rows. **SQL signature change:** `json_tree_subtree(doc, token, budget)` becomes `json_tree_subtree(doc, token, selector, budget DEFAULT 64)`. The extra argument is the field's selector, the same one `->` takes for the path: the walk returns the one entry with no parent tag that carries it, at `d = 0`, before its descendants, and returns nothing when the document has no such entry. `json_tree_subtree_at` is unchanged and still returns the node its gates reach at `d = 0`; both now run one internal walk, `eql_v4_internal.json_tree_walk`. No stored field was added. In Stack Encrypt a subtree key for a path that ends in an array position (`$.tags[2]`) now walks from the array's token through a position gate, so the element itself comes back; `JsonSubtreeKey::selector()` is what `json_tree_subtree` takes. In `eql-bindings`, `JsonTreeSubtree` opens to the node's own value, and to `Option<Value>`, which is `None` where the row has no node at the path. What the database learns: the selector sent beside a top-level field's token is the one `->` sends; a path ending in an array position sends its array's token, which covers every element of that array, as a wildcard read already does.

packages/eql/crates/eql-bindings/CHANGELOG.md

@@ -15,7 +15,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   projection, which the server compares with a needle by native `jsonb @>`.
   `JsonTreeEntry` is one entry as `->` and the walks return it,
   `JsonTreeQuery` the containment needle and `JsonTreeLeafQuery` the
-  ordering operand at a path. `JsonTreeSubtree` collects a walk's rows; past
+  ordering operand at a path. `JsonTreeSubtree` collects a walk's rows
+  (`eql_v4.json_tree_subtree(doc, key.token(), key.selector())` for a key
+  with no gates, `json_tree_subtree_at(doc, key.token(), key.gates())` for
+  one with), which hold the node at the path itself, so they open to its
+  value, `{}`, `[]` and scalars included, or, as `Option<Value>`, to
+  `None` where the row has no node there; past
   the walk's budget the server returns the sealed root instead, which
   opens at the path the walk's key was derived from: the rows are built
   with `JsonTreeSubtree::from_key(&key, rows)`, and there is no

packages/eql/crates/eql-bindings/src/v4/json_tree.rs

@@ -207,18 +207,21 @@ impl JsonTreeExtracted {
     }
 }
 
-/// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or
-/// `json_tree_subtree_at`), collected with the key that keyed the walk:
-/// opens to the value under the walked path, or to the list of complete
-/// subtrees the rows hold.
+/// The rows a subtree walk returned, collected with the key that keyed
+/// the walk: `eql_v4.json_tree_subtree(doc, key.token(), key.selector())`
+/// for a key with no gates (a chain head, or the root when it is not
+/// sealed), `json_tree_subtree_at(doc, key.token(), key.gates())` for one
+/// with. Every walk returns each node the path reaches at depth `0` with
+/// everything beneath it, so the rows open to the value at the walked path,
+/// `{}`, `[]` and scalars included; to `None` as an `Option<Value>` when
+/// there is no node there; or to a list.
 ///
 /// Past its budget a walk returns the sealed root alone, at depth `0`;
 /// those rows open to the whole document, and the key's path selects the
-/// same value(s) the walked rows open to. The key also says which walk the
-/// rows came from (no gates: `json_tree_subtree`, the rows below the path's
-/// node; gates: `json_tree_subtree_at`, the nodes the path reaches), so the
-/// only constructor, [`from_key`](Self::from_key), takes it, and rows
-/// cannot be opened without it.
+/// same value(s) the walked rows open to. The only constructor,
+/// [`from_key`](Self::from_key), takes the key, which checks every row
+/// against its path when the rows are opened, so rows cannot be opened
+/// without it.
 #[derive(Clone, Debug, PartialEq)]
 pub struct JsonTreeSubtree {
     rows: Vec<JsonTreeEntry>,
@@ -455,8 +458,8 @@ impl DecryptInto<Value> for JsonTreeExtracted {
     }
 }
 
-/// The rows of `json_tree_subtree(doc, token)` open to the value under the
-/// walked node.
+/// The rows of a walk open to the value at the walked path; no node there
+/// is an error, which `Option<Value>` reads as `None`.
 impl DecryptInto<Value> for JsonTreeSubtree {
     type Context = NonEmpty<Identifier>;
     fn decryption<K: 'static>(self, context: Self::Context) -> Decryption<Value, K> {
@@ -467,8 +470,21 @@ impl DecryptInto<Value> for JsonTreeSubtree {
     }
 }
 
-/// The rows of a walk open to the complete subtrees they hold, in order:
-/// one per node the gates of `json_tree_subtree_at` reached.
+/// The rows of a walk open to the value at the walked path, or `None`
+/// where the document has no node there: the four answers `jsonb #>` gives.
+impl DecryptInto<Option<Value>> for JsonTreeSubtree {
+    type Context = NonEmpty<Identifier>;
+    fn decryption<K: 'static>(self, context: Self::Context) -> Decryption<Option<Value>, K> {
+        match self.subtree() {
+            Ok(subtree) => subtree.decryption(AeadContext::from(context)),
+            Err(error) => Decryption::failed(error),
+        }
+    }
+}
+
+/// The rows of a walk open to a list: one value per node the gates of
+/// `json_tree_subtree_at` reached, or, for a key with no gates, the values
+/// of the node's children.
 impl DecryptInto<Vec<Value>> for JsonTreeSubtree {
     type Context = NonEmpty<Identifier>;
     fn decryption<K: 'static>(self, context: Self::Context) -> Decryption<Vec<Value>, K> {

packages/eql/docs/reference/json-tree-leakage.md

@@ -33,7 +33,7 @@ A query hands the database a term derived from the plaintext it asks about, and
 - **Containment (`@>`, `<@`), and equality at a path, which is sent as a containment needle.** The needle's own projection: the selectors of its paths, the value selectors of its nodes, and its element grouping. So the database learns which paths and which (path, value) pairs you asked about, and which rows hold them.
 - **Path extraction (`->`, `->>`, `json_tree_path_query`, `json_tree_path_exists`).** The selector of the path asked. A position in the path reads as the wildcard, so a position is not sent. The entry returned is a ciphertext.
 - **Ordering at a path (`<`, `<=`, `>`, `>=`).** The path's selector and the bound's own ordering term, which compares with the stored terms exactly as they compare with each other: the database learns the bound's place in the order of the stored values at every ordered path, and whether it equals one. `ORDER BY ord_term(…)` sends no term and shows nothing the stored terms do not.
-- **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token**, and one gate per step below it. The token is not the token of the node you asked for: it is the token of the deepest node on the path the client can derive, which is the top-level field the path starts with or, while the path goes on through array positions, the element they reach. Reading `$.profile.address.city` sends `$.profile`'s token. A gate is the step's selector or, for an array position, `#n:<selector>`, with the position `n` in the clear. A token is a capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find each node under that field in any row, see how the nodes nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree, and no token covers a whole document. With it off, every subtree read whose path begins with a key sends the root's token, which covers every row's whole document.
+- **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token**, and one gate per step below it. The token is not the token of the node you asked for: it is the token of the deepest node on the path the client can derive, which is the top-level field the path starts with or, while the path goes on through array positions, the element they reach, short of the path's last step. Reading `$.profile.address.city` sends `$.profile`'s token. A gate is the step's selector or, for an array position, `#n:<selector>`, with the position `n` in the clear. A path that ends in an array position sends the token of the array and the position as a gate, so that the walk can return the element itself: reading `$.tags[2]` sends `$.tags`'s token, which lets the database walk every element of `$.tags`, as reading `$.tags[*]` does, rather than the third alone. Reading a top-level field itself (`json_tree_subtree`) also sends that field's **selector** beside its token, so the walk can return the field's own entry: the same selector `->` sends for the path, which the database would otherwise have linked to the token only once the token reached one of the field's members (the entry stored before a subtree's first member is its parent). A token is a capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find each node under that field in any row, see how the nodes nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree, and no token covers a whole document. With it off, every subtree read whose path begins with a key sends the root's token, which covers every row's whole document.
 
 ## What is authenticated
 
@@ -60,7 +60,7 @@ What the checks do not cover: **which row an entry comes from**. Nothing in an e
 | which leaves sit in the same array element | no | yes, through the projection's nesting: the one new disclosure, required for exact containment, bounded to array elements |
 | equal values in sibling elements | ciphertexts equal | ciphertexts differ |
 | ordering terms | on scalar leaves, with no per-path choice | only at the paths the index orders; none by default |
-| a subtree read | reveals the selector asked | reveals a token for the top-level field the path starts with, and with it that field's shape in every row for the life of the column |
+| a subtree read | reveals the selector asked | reveals a token for the top-level field the path starts with (or, for a path ending in an array position, for that array), and with it that subtree's shape in every row for the life of the column; reading a top-level field itself also sends its selector |
 
 ## The two options that change the picture