diff --git a/.changeset/eql-json-tree-ordinal-chain.md b/.changeset/eql-json-tree-ordinal-chain.md new file mode 100644 index 000000000..8d40e3357 --- /dev/null +++ b/.changeset/eql-json-tree-ordinal-chain.md @@ -0,0 +1,13 @@ +--- +"@cipherstash/eql": minor +--- + +EQL v4 `json_tree` (unreleased): the token chain is derived by ordinal and the walk probes to the first miss. + +A stored entry now carries four fields, `s`, `c`, `p` and `op`. The value selector `vs` lives only in the document's projection `q`, which the engine now reads back as the projection instead of rebuilding it from the entries; the group tags `n` and `g` are gone with it, since nothing read them once `q` was stored. The wrapped token `w` and the ordinal `o` are gone too: a child's token is `H("t" ‖ t(parent) ‖ s ‖ LE32(o))`, `s` being the child's own selector, which the server derives as it walks, and the ordinals under a node are dense from zero, so the walk probes them in turn and stops at the first miss. A subtree of `N` nodes costs about `2N` probes (a SHA-256 and a lookup each) and `N` token hashes whatever the document's width, where it cost a probe for every ordinal up to the document's widest node at every node visited. The entry validator accepts the four fields only, so a document written by the earlier build of this unreleased family is refused. + +The client can derive a token only for a chain head and the array positions that directly follow it, since an object member's ordinal is a rank among its siblings' keys. `JsonSubtreeKey::gates()` gives the rest of a path as one gate per step, a selector, or `#n:` for an array position with the array elements' selector, and `eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64)` walks them, replacing the earlier `json_tree_subtree_at(doc, token, selector, budget)`. `json_tree_children` and `json_tree_child_count` take the same `gates`, optional and empty by default. A gated walk returns the node the gates reach with `d = 0` and its descendants beneath it, so a wildcard reaches several nodes and opens as a list; `json_tree_subtree(doc, token, budget)` is now `json_tree_subtree_at` with no gates and still returns the descendants of the token's node from `d = 1`. Past the budget a walk still returns the sealed root alone, with `j = 0` and `d = 0`. `eql_v4_internal.json_tree_unwrap`, `json_tree_xor16` and `json_tree_fanout` are removed; `eql_v4_internal.json_tree_child_token(token, s, o)` is added. Binding the child's selector into its token, and the element selector into a position gate, is what makes a position name an array element only: an ordinal alone is an element's position or an object member's rank, so in a row whose node at the path is an object, `$.a[0]` would otherwise reach the member that ranks first, where `jsonb` has no node. + +**Byte encodings.** Every field the SQL compares, derives or decodes is padded standard base64, as PostgreSQL's `encode(x, 'base64')` writes it: the selectors `s`, the `s` and `vs` pairs of `q`, the parent tag `p`, the ordering term `op`, and the tokens and gates `json_tree_subtree_at`, `json_tree_children` and `->` take, where they were hex; the header `h` was base64 already. The ciphertext `c`, which no SQL function decodes, is RFC 1924 base85. The parent tag `p` is the first eight bytes of its hash, since it is a lookup key within one document and nothing authenticates through it. `eql_v4_internal.json_tree_ope_cllw` is added for `eql_v4.ord_term`, because the `ope_cllw(jsonb)` the scalar types share reads hex. A document written by the earlier build of this unreleased family is refused by the engine. Re-encoding 1,000 stored rows of each benchmark corpus: a nested document measures 22% smaller on disk (`pg_column_size` 40,808 to 31,894 bytes) and a flat one 12% (1,596 to 1,398). + +Measured on 10,000 and 100,000 rows of a nested corpus (about 213 entries a document), query-only medians: subtree walks 30 to 55% faster (a 122-entry subtree walked out, 23 ms to 11 ms), `json_tree_children` and `json_tree_child_count` 6 to 9%, a stored nested document 25% smaller (heap 810 MB to 621 MB at 10,000 rows), containment and field queries unchanged within the run's noise. diff --git a/.changeset/eql-json-tree-walk-and-projection.md b/.changeset/eql-json-tree-walk-and-projection.md index d44a6560b..e9341e9a2 100644 --- a/.changeset/eql-json-tree-walk-and-projection.md +++ b/.changeset/eql-json-tree-walk-and-projection.md @@ -5,9 +5,11 @@ **The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured (cipherstash/benches, `report/JSON_TREE_COMPARISON.md` at 57abcf9) on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`: - **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.5 to 0.7 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK. -- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size: it probes at most as many ordinals as the document has entries and returns at most as many rows, so a stored document whose unauthenticated chain fields claim a huge ordinal, or wrap a child's token back to its parent's, can no longer keep a query running until `statement_timeout`. -- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to: attach the path with `below(path)` for `json_tree_subtree` rows and `at(path)` for `json_tree_subtree_at` rows (`JsonSubtree` in `stack-encrypt`, `JsonTreeSubtree::new(rows)` in `eql-bindings`). Rows that fell back and carry no path refuse to open. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. +- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size, so a stored document whose unauthenticated chain fields claim a huge ordinal, or loop back to an ancestor, can no longer keep a query running until `statement_timeout`. +- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to. The rows are opened through the key that keyed the walk, which carries the path it was derived from and knows from its gates which walk the rows came from: `key.open(rows)` in `stack-encrypt`, `JsonTreeSubtree::from_key(&key, rows)` in `eql-bindings`. Rows cannot be opened without it, so a document that grows past the budget does not break a caller. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out. Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (16.9 ms down to 0.11 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed. -The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows`, an optional `path` and `below`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::below`, `at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. +The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct of the rows and the key that keyed the walk; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtreeKey::path` and `open`, `JsonSubtree::path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged. + +**The leaf ordering term `op` is opt-in per path.** A stored leaf carried the order-preserving term on every number and string, about 80 bytes each and the largest field of a nested document, whether or not anything ordered by it. Now the JSON index's options name the paths that order (`{"json": {"ordered": ["$.age", "$.orders[*].total"]}}` on the data plan; `JsonOptions::ordering([...])` in Rust), a wildcard covering every element of an array, and no path is ordered by default. A position in an ordered path (`$.scores[2]`), or in the path an ordering asks at, is refused: the comparison extracts by selector, which reads a position as the wildcard and reaches the first element, never the one named. An ordering query at a path the index does not order is refused before any key loads rather than silently matching no row. The SQL is unchanged: `eql_v4.ord_term` was already NULL for a leaf without a term. `eql-bindings` gains `v4::JsonTree`, the `json_tree` layout under options, and `stack-encrypt` gains `target::Layout` and `Plan::value().encrypt_as(layout)`, a plan laid out by a value whose queries run under its options. diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index baa8e8cd8..ad76faf70 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -78,6 +78,7 @@ jobs: - { task: "fuzz:term-decode", slug: term-decode } - { task: "fuzz:check-record", slug: check-record } - { task: "fuzz:plan-build", slug: plan-build } + - { task: "fuzz:json-wire", slug: json-wire } steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: @@ -126,6 +127,7 @@ jobs: - { task: "fuzz:term-decode", slug: term-decode, dir: packages/stack-encrypt, target: term_decode } - { task: "fuzz:check-record", slug: check-record, dir: packages/stack-encrypt, target: check_record } - { task: "fuzz:plan-build", slug: plan-build, dir: packages/stack-encrypt, target: plan_build } + - { task: "fuzz:json-wire", slug: json-wire, dir: packages/stack-encrypt, target: json_wire_decode } steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: diff --git a/Cargo.lock b/Cargo.lock index bc8fce38a..5bf1172ff 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -285,6 +285,12 @@ version = "1.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" +[[package]] +name = "base85-simd" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd16ef96137ca89cc38b0acb5bee36fada9707c044bc9c753f55c816bfa719" + [[package]] name = "bit-set" version = "0.8.0" @@ -981,7 +987,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -1480,7 +1486,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.2", "tokio", "tower-service", "tracing", @@ -2138,7 +2144,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -2508,7 +2514,7 @@ dependencies = [ "quinn-udp", "rustc-hash", "rustls", - "socket2 0.5.10", + "socket2 0.6.2", "thiserror 2.0.18", "tokio", "tracing", @@ -2547,7 +2553,7 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.2", "tracing", "windows-sys 0.60.2", ] @@ -2864,7 +2870,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.12.1", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -2921,7 +2927,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -3358,6 +3364,7 @@ name = "stack-encrypt" version = "0.3.0" dependencies = [ "base64ct", + "base85-simd", "cllw-ore", "miette", "ore-rs", @@ -3592,7 +3599,7 @@ dependencies = [ "getrandom 0.4.2", "once_cell", "rustix 1.1.5", - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] @@ -4776,7 +4783,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.52.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/docs/fuzzing.md b/docs/fuzzing.md index e1d7ace9e..ca130747b 100644 --- a/docs/fuzzing.md +++ b/docs/fuzzing.md @@ -28,6 +28,7 @@ Current targets: | `fuzz:term-decode` | `stack-encrypt` | `term_decode` | the SEM term decoders (`EqualityTerm`, `MatchTerms`, `OreTerm`, `OpeTerm` `from_bytes`) | | `fuzz:check-record` | `stack-encrypt` | `check_record` | `dynamic::record::check_record` over an `Arbitrary`-derived plan and tree, against a model of the record rules (structure-aware) | | `fuzz:plan-build` | `stack-encrypt` | `plan_build` | `plan::FieldsBuilder::build` over `Arbitrary`-derived untrusted names (context, fields, identities); a built field's label must render and parse back, and passthrough names are never refused (structure-aware) | +| `fuzz:json-wire` | `stack-encrypt` | `json_wire_decode` | the JSON index's wire decoders (`JsonDocument`, `JsonNode`, `JsonProjection`, `JsonLeafQuery` from JSON text; `JsonHeader::from_bytes`), over base64 tags and base85 ciphertexts; an accepted document or node must round-trip through its wire form | Each target is a few lines — `libfuzzer-sys` hands a `&str` (or `&[u8]` for the byte decoders) to the parser via the `arbitrary` crate: diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index d46293c6e..f6d7f822f 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -6,6 +6,8 @@ **Relates to:** #1231 (containment matches across array elements), #1209 (`Contains`, `ContainedBy`, `AtPath` declared), ADR-0009 (operators select the index), system ADR-0002 (EQL v4 by producer), `packages/eql/docs/plans/stevec-envelope-key-header-rfc.md` (the v3 envelope), `docs/plans/2026-10-04-plan-builder.md` ("The JSON index" under Additions) **Prototypes:** `2026-10-17-json-tree-index/` beside this file, three SQL files that run on a stock PostgreSQL 16 with no extension +> **Note (2026-10-22).** The stored form this document designs has changed twice since, and the implementation notes at the end record each change. As of Phase 3b a stored entry carries only `s`, `c`, `p` and `op`. The value selectors live only in the stored projection `q`. The group tags `n` and `g`, the wrapped token `w` and the stored ordinal (`i` here, `o` in the code) are gone. A child's token is derived from its parent's token and its ordinal, not its label, and the walk probes ordinals until the first miss. The sections below are kept as written, with short dated notes where they would otherwise mislead. + ## Summary Stack Encrypt has no JSON index yet. The operators exist (#1209 merged `op::Contains`, `op::ContainedBy` and `op::AtPath` on `next`), the plan-builder document reserves `Json: Index` as "a new core operation", and EQL v4 has a storage-only `public.eql_v4_json` but "no SteVec bindings yet". The v4 JSON wire format is therefore unconstrained by anything shipped. This document records what to build and the phased plan that proves it, and the implementation notes at its end record what each phase built. @@ -74,6 +76,8 @@ The flattening is unchanged in what it visits: every node, with arrays indexed u The value entry and its sentinel disappear: `vs` sits on the node's own entry. The entry count halves. `a` disappears: `n` marks an array element, and the chain enumerates an array's elements. +> **2026-10-22:** of the fields in this table a stored entry now keeps `s`, `p`, `c` and `op` (Phase 3b). `vs` moved to the stored projection `q` (Phase 3 stored it, Phase 3b removed it from the entries), and `w`, `n`, `g` and the ordinal are not stored. + What the client gets back for a scalar path (`doc -> 'sel'` where the leaf is a string) is unchanged in shape: one entry, decryptable with the grafted `h`, its own `s` and `c`, and its ordinal `j` (grafted by the extractor, as `h` is today). For a container path the client gets the subtree's entries and rebuilds the value from each entry's decrypted `(label, kind, scalar)` and the depth the walk reports. Whole-document decryption is one AEAD on the root entry when the sealed-root option (next section, on by default) is set, and otherwise the same walk, run by the client over the whole `sv` with the root token it can derive itself. ### 2. The token chain @@ -86,6 +90,8 @@ p(child) = H("p" ‖ iv ‖ t(parent))[..16] stored w(child) = t(child) XOR H("w" ‖ iv ‖ t(parent) ‖ c(child))[..16] stored ``` +> **2026-10-22:** superseded by Phase 3b. A child's token is `H("t" ‖ t(parent) ‖ LE32(o))`, keyed by its ordinal rather than its label, and nothing stores it: the server derives it as it walks, so there is no `w`, no unwrapping and no XOR, and the parent tag is the position-keyed one below. The client can then derive a token only for a chain head and the array positions that directly follow it, and sends gates for the rest of a path. The recursive query, the paragraph on the mask, and "Wildcard paths" below describe the original design. + `H` is SHA-256. Only `t(root)` needs the index key; everything below it is hashing, so the server can do its half with core functions. The tokens are a function of the column and the **positional** path, so the client derives a start token for any path it can name without a wildcard: `$.nested`, `$.a[2].b`. Labels are bound with a kind byte and as the final variable-length field, so `("ab", "c")` and `("a", "bc")` cannot collide. Server side, the subtree under a start token is one recursive query. From `token-chain.sql`, which implements the derivations above (`chain.parent_tag(iv, t)` is `h16("p" ‖ iv ‖ t)` and `chain.mask(iv, t, c)` is `h16("w" ‖ iv ‖ t ‖ c)`) and runs as written on a stock PostgreSQL: @@ -142,6 +148,8 @@ The sealed root caps how much a token unlocks. It does nothing about how long: a `p = H("p" ‖ iv ‖ t(parent))` is the same value on every child of a node, so sibling groups are visible to anyone holding the row, with no token. That is the main at-rest residue of the chain. Keying the tag by the child's position removes it: `p_i = H("p" ‖ iv ‖ t(parent) ‖ i)`, with the child's ordinal `i` stored in the clear on the entry. At rest an observer learns only "this is some node's `i`-th child". The walk changes from one hash per walked node to `F` hashes per walked node, where `F` is one more than the largest `i` in the document (public), followed by the same equality join as now; `W × F` hashes for a walk of `W` nodes, which Phase 3 measures. Array children take `i` from their index, which the client needs for reconstruction anyway. Object children take `i` from a per-document shuffled order, derived from the document key, so the ordinal does not rank key names. The storage cost is one small integer per entry. +> **2026-10-22:** the parent tag is still position-keyed, but since Phase 3b the ordinal is not stored and there is no `F`: the ordinals under a node are dense from zero, so the walk probes them in turn and stops at the first miss, about `2N` probes for a subtree of `N` nodes. + ### 3. Exact containment (#1231) **The gap, verified.** `jsonb-semantics.sql` pins PostgreSQL's own answers, and `containment-oracle.sql` runs a plaintext-selector simulation of the v3 flattening (the PRF is injective in practice, so string selectors give the same containment answers as MAC'd ones) beside it: @@ -186,6 +194,8 @@ OR (q.alt IS NOT NULL AND to_query(doc) @> q.alt) -- a second GIN probe, bitmap Then `to_query(a) @> to_query(b)` is exact for arrays of containers with no plpgsql at all: the `sv` arrays check selector presence, and each needle `el` object has to fit inside one stored `el` object, which is the same-element rule. The GIN `jsonb_path_ops` index still serves it, because GIN is lossy for `@>` and the executor rechecks the real operator on the heap tuple; and the projection is computed once per row inside the expression index, so a query pays nothing for it. `alt` stays as it is. Two ways to obtain the grouping server-side, with identical leakage: build the projection from the `g` and `n` tags (a plpgsql `to_query`, immutable, run at index time), or store `sv` itself nested by element, which removes the tags from the wire and makes `->` by selector a `$.**` jsonpath lookup. Phase 2 tries the projection first, measures both, and keeps `groups_ok` only if neither engages the index. Whichever form wins, `n` and `g` are emitted only for elements that are **containers**: a scalar element carries one constraint, so grouping cannot be wrong for it, and the tags would reveal element membership for nothing. +> **2026-10-22:** the projection won, and it is stored rather than built: the client writes it beside `sv` as `q` (Phase 3), and since Phase 3b the entries carry no `n` or `g`. Element grouping is visible at rest through the nesting of `q` instead. The `grp` needle and `groups_ok` above were never built. + **`ContainedBy` (`<@`).** `col <@ needle` asks whether the stored document is contained by the query value, so the roles swap: the query term has to carry the whole projection of its own document, in the same element-nested shape a stored document projects to (so the same-element rule still holds), and the row's projection is the side tested for containment inside it. `jsonb_path_ops` indexes `@>`, `@?` and `@@`, not `<@`, so no GIN index serves this direction: it is a filter evaluated per row, as `<@` already is in v3, and a query that uses it at scale needs another predicate to bound the rows. The root-scalar `alt` does not carry over. It lets a scalar needle match a stored array, and for `<@` it is the stored side that would need it, which a stored document does not carry; so a stored root scalar is not found inside a needle root array, which is the one case where `<@` would disagree with `jsonb <@`. Either a stored root scalar gains the pair, or the case is documented as a known difference. **Portability to v3.** The group tags and the recheck do not depend on the chain. If #1231 must also be fixed for rows cipherstash-client writes, the same `g`/`n` fields and the same `groups_ok` can be added to the v3 wire and SQL, with `g` derived from the document key instead of a token. This document does not propose that; it records that the fix is separable. @@ -225,6 +235,8 @@ An `AtPath` needle is `{"sv": [{"vs": "…"}]}`. An `AtPath` term is th With the sealed-root option off, the root entry's `c` is the root's label and kind only, and the top-level entries carry `p` and `w` like every other node. The header records which shape the document has. +> **2026-10-22:** as built, a stored document is `{"v", "i", "h", "sv": [{s, c, p?, op?}], "q"}`, with no `k`; a containment needle is a projection, `{"sv", "el", "alt"?}`, with no `grp`; and a subtree request is a token and one gate per step below it. The size estimates below predate both; the measured sizes are under Phase 3b. + **Size.** Per 16-byte field, hex costs 32 characters plus about six of framing. Measured against the envelope RFC's fixture document (`{"hello":"world-1","nested":{"deep":"constant"},"number":1}`, 5 nodes): | | v3 today | proposed | @@ -264,6 +276,8 @@ On a flat five-node document the proposal is about 8% larger: the two chain fiel | the broadest token a legitimate query can send | none exist | with the sealed root: one top-level field's subtree. Without it: `t($)`, the whole column's shape | | how long a revealed token stays useful | not applicable | for as long as the column lives; epochs (recorded above, not scheduled) would bound it to rows written under the epoch it was issued for | +> **2026-10-22:** parent tags are position-keyed (Phase 1), so siblings do not share a `p` at rest. Since Phase 3b there is no `g`: which leaves sit in the same array element is visible at rest through the nesting of the stored projection `q` instead. + ### 7. Performance - **Encrypt:** per node two PRFs (`s`, `vs`), three hashes (`t`, `p`, mask), one small AEAD. v3 does two MACs and one AEAD over the subtree bytes. Fewer bytes encrypted at every depth above the leaves. @@ -273,10 +287,14 @@ On a flat five-node document the proposal is about 8% larger: the two chain fiel - **Decrypt a document:** with the sealed root, one AEAD on the root entry, as today. Without it, the client walks the chain locally: N hashes plus N small AEADs; the client has the index key, so no extra round trip either way. - **Encrypt, with the sealed root:** one more AEAD, over the whole document, beside the N small ones. +> **2026-10-22:** as built, encrypt is two PRFs per node (`s`, `vs`) and up to three hashes (`t`, `p`, and an object member's rank), with no mask. The walk reads the document once into a map keyed by parent tag (Phase 3) and probes it, about `2N` probes and `3N` hashes for a subtree of `N` nodes (Phase 3b); measurements are under Phase 3 and Phase 3b. + ### 8. Standard PostgreSQL Everything the server does: `sha256(bytea)` (core since PostgreSQL 11), `get_byte` / `set_byte` for a 16-byte XOR in plpgsql, `decode` / `encode`, `WITH RECURSIVE`, `jsonb_array_elements WITH ORDINALITY`, and `jsonb_path_ops` GIN. No pgcrypto, no superuser, no C. The same posture as the v3 surface Supabase runs today. +> **2026-10-22:** with no XOR since Phase 3b, the walk needs no `get_byte` / `set_byte`: it is `sha256`, `decode` / `encode`, a jsonb object lookup and `WITH RECURSIVE`. + ## Further mechanisms (added 2026-10-18) The four above are in the phased plan: the native-`@>` projection and container-only tags under containment, the sealed root and position-keyed parent tags under the token chain. Four more are recorded here, with what each buys and costs, so that none is rediscovered as free. @@ -311,6 +329,8 @@ The four above are in the phased plan: the native-`@>` projection and container- - **A build change, additive only.** `tasks/build.sh` renders `src/v3/**` as both bundles and `verify_installer_complete.sh` asserts every ordered v3 file is in the v4 installer. The new family needs the renderer to also order and render a v4-only source tree into the v4 bundle, and the completeness check to accept files that exist in one bundle only. Nothing is removed from either bundle, so this cannot block the v4 release: until the build change lands, the phased plan installs the new family as a separate file over the v4 bundle. - Tests: `mise run test:encryption:postgres` in `packages/eql` installs the v4 bundle built from the tree and runs the Stack Encrypt PostgreSQL tests with fresh encryption under a fake KMS and no credentials (it reached `next` after `3a216441`, the commit the rest of this section was read from, which has only the database-free `test:encryption`). The JSON suite joins it. The SQLx fixture machinery is not needed for this and should not be used: the point of the fake key source is that the whole suite runs on a fork PR. +> **2026-10-22:** as built, the walk functions are `eql_v4.json_tree_subtree(doc, token, budget)`, `json_tree_subtree_at(doc, token, gates, budget)`, `json_tree_children(doc, token, gates)` and `json_tree_child_count(doc, token, gates)`, and there is no `groups_ok`. + **`languages/golang`** - Nothing in the phased plan. After it, `JSON` moves from "the JSON index is a new operation in the engine" to producible in `encrypt/eql`'s table, and the data plan carries the `json` index with its options (plan-builder Additions, item 7). @@ -339,36 +359,135 @@ The index, the SQL and the live oracle landed in the pull request stacked on thi **Where it is.** The engine half is `packages/stack-encrypt/src/json/`: `Json` is the index (`Index`, term `JsonTerms`), `target::json_document` the sealing operation, `JsonDocument` the stored term and an `EncryptFrom` / `DecryptInto` target, `JsonQuery` the needle for `Contains`, `ContainedBy` and `AtPath`, `JsonLeafQuery` the operand for the orderings at a path, and `JsonSelector` / `JsonSubtreeKey` the extraction keys, derived through `KeysetCipher::json_selector` and `json_subtree_key`. `IndexSpec::Json(JsonOptions)` is the data form, with the key `"json"` and `{"json": {"sealed_root": …}}` as its wire form; a data plan reads it and refuses to run it (`Reason::JsonIndexPending`). The SQL is `packages/eql/src/v3/json_tree/`, confined to EQL v4; the EQL types are `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; the oracle is `packages/eql/tests/encryption/tests/json_tree.rs`, which stores the design's corpus and random documents encrypted beside their plaintexts and checks that `@>` and `<@` over the encrypted column select the rows PostgreSQL's own `jsonb` operators select over the plaintext column, with the GIN index shown in `EXPLAIN`. The pure-Rust half of that oracle is `packages/stack-encrypt/tests/json_index.rs`, against a model of `jsonb @>` pinned to the answers in `jsonb-semantics.sql`. -**Decisions taken by the code.** Of the eleven listed below: 1, hex for every sixteen-byte field and base64 for `c` and `h`, pending Phase 3; 2, leaf ordering terms bound to the column only, numbers as the `f64` they round to and strings as text, under the scalar `Ope` index's derivation; 3, wildcard arrays only; 4, group tags on, and only on container array elements; 5, the child count is in every container node's ciphertext, and a truncated subtree is `OpenedType`, not a shorter array; 7, plain SHA-256; 8, the option is the boolean `sealed_root`, on by default, recorded in the header's flags; 9, the projection is built from the tags server-side (`eql_v4.json_tree_query`) and `groups_ok` was not needed; 10, position-keyed parent tags on, the ordinal stored as `o`, object members ranked under the parent's token; 11, keyed: a child's token is `PRF(json/token-step ‖ column, PAE(t(parent), 0x01, key))` (`0x02, LE64(index)` for an element) and an object member's rank is `PRF(json/rank ‖ column, PAE(t(parent), key))`, each under its own PAE domain rather than the heads' `json/token/v1`, so a token's holder can neither test key names against the child tokens the walk unwraps nor rank guessed names against the ordinals. Decision 6 is out of scope, as proposed. The first implementation hashed both steps unkeyed, as the token chain section above writes them; it was corrected before any release, so no stored document carries the unkeyed form. +**Decisions taken by the code.** Of the eleven listed below: 1, hex for every sixteen-byte field and base64 for `c` and `h`, pending Phase 3 (taken on 2026-10-23, Phase 3d: base64 for every field the SQL reads, base85 for `c`); 2, leaf ordering terms bound to the column only, numbers as the `f64` they round to and strings as text, under the scalar `Ope` index's derivation; 3, wildcard arrays only; 4, group tags on, and only on container array elements; 5, the child count is in every container node's ciphertext, and a truncated subtree is `OpenedType`, not a shorter array; 7, plain SHA-256; 8, the option is the boolean `sealed_root`, on by default, recorded in the header's flags; 9, the projection is built from the tags server-side (`eql_v4.json_tree_query`) and `groups_ok` was not needed; 10, position-keyed parent tags on, the ordinal stored as `o`, object members ranked under the parent's token; 11, keyed: in this first form a child's token was `PRF(json/token-step ‖ column, PAE(t(parent), 0x01, key))` (`0x02, LE64(index)` for an element) and an object member's rank `PRF(json/rank ‖ column, PAE(t(parent), key))`, each under its own PAE domain, so a token's holder could neither test key names against the child tokens the walk unwraps nor rank guessed names against the ordinals. Decision 6 is out of scope, as proposed. (Both steps were first hashed unkeyed, as the token chain section above writes them, and were corrected before any release.) + +> **2026-10-22:** 4 and 10 have since moved. The grouping lives only in the stored projection `q`, with no `n` or `g` on an entry (Phase 3 and 3b). Parent tags are still position-keyed and an object member's ordinal is still its rank under the parent's token, keyed (decision 11), but the ordinal is no longer stored (Phase 3b). The ordinal chain of Phase 3b derives a child's token from its ordinal, which carries no label, so the keyed label step of decision 11 is gone with it and only the keyed rank remains. **Deviations.** - *Nonces are random, not derived from the selector and ordinal.* Every entry seals through the engine's one AEAD path (`vitaminc-encrypt`'s AES-256-GCM with its own random 96-bit nonce, carried in `c`), so there is one leaf cipher in the crate and no second AES-GCM call site. Uniqueness under one document key is then probabilistic (`N² / 2^97` at `N` entries) rather than structural. The entry's AAD still binds its ordinal `j`, so a moved or reordered entry fails to open as the design requires; the cost is twelve bytes per entry, which Phase 3 should count. -- *The ordinal is `o`, not `i`*, because `i` is the envelope's identifier, which `->` grafts onto an extracted entry. An extracted entry also carries `j`, its ordinal in the document, which its AAD is bound to, and a walk adds `d`, its depth below the start. -- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. +- *The ordinal is `o`, not `i`*, because `i` is the envelope's identifier, which `->` grafts onto an extracted entry. An extracted entry also carries `j`, its ordinal in the document, which its AAD is bound to, and a walk adds `d`, its depth below the start. (2026-10-22: since Phase 3b `o` is not stored at all.) +- *The subtree walk returns a node's descendants and not the node itself.* The walk addresses nodes by their parent, and the design's wire has no field that names a node by its own token. The client reads the node's kind from its children's labels; a node with no descendants, an empty container or a scalar, yields no rows, and `->` on its selector returns the node. A self tag per node would close this at sixteen bytes an entry and is left to Phase 4. (2026-10-22: a gated walk, `json_tree_subtree_at`, now returns the node its gates reach at `d = 0`, with no self tag; the ungated `json_tree_subtree` still returns descendants only.) - *Containment is not `STRICT` in SQL.* PostgreSQL inlines a `STRICT` SQL function only when it can prove the body is NULL for a NULL argument, and the `OR` that adds the needle's `alt` probe is not provably NULL; un-inlined, the call was an opaque filter and the GIN index never engaged. `eql_v4.json_tree_contains` and the `@>` / `<@` wrappers over it are declared without `STRICT` and return NULL for a NULL side on their own. - *`<@` cannot match a stored root scalar inside a needle root array*, since a stored document carries no `alt` pair. The oracle records the case; `@>` is exact. - *Value selectors compare strings byte for byte*, with no Unicode normalization, because `jsonb @>` does, and the oracle is agreement with `jsonb @>`. The scalar equality index normalizes to NFC; a JSON value does not. Numbers canonicalize by value, so `2` and `2.0` are one value, as `jsonb` compares them. - *`AtPath` is the containment of the needle document that holds the value at the path*, with a position in the path read as the wildcard, rather than a single `{vs}` entry: a leaf inside an array element sits in that element's group of the projection, and the needle has to say which group. Containment of that needle is equality only for a scalar, so a container value is refused (`TermError::JsonValueNotScalar`; `op::Contains` asks the containment question), and the needle carries no `alt`, so `("$", 1)` does not match a stored `[1]`. - *The data-plan form and the catalog are not done.* The Go guest, `stashgen`, the target table and `v4::all()` do not know the family; `IndexSpec::Json` crosses the wire and is refused with a reason. The family is hand-written in `eql-bindings` as the SteVec structs are. Both are the next bindings issue. - *A document is at most 127 levels deep*, under either root option. The sealed root's plaintext is the document as JSON, and `serde_json` parses JSON back only to 128 levels of nesting, so the 512 levels the walk first admitted sealed documents that then never opened. The bound is the reader's, so a document too deep fails when it is sealed, while the caller still holds it, and not when it is read. -- *A stored document is read as untrusted input.* Its chain fields are not authenticated, and whoever holds a token can write tags that form a cycle or a chain deeper than any sealed document. `JsonDocument::children` checks each entry at its own ordinal instead of looping over a fan-out the document chooses; `JsonDocument::subtree` and the SQL walks stop below 127 levels, and the Rust walk also stops once it has returned as many entries as the document holds; the rebuild of a subtree's rows refuses a repeated row and a nesting deeper than 127 levels as `OpenedType`. Each would otherwise have overflowed the stack, looped, or allocated what the document asked for. -- *Nothing wipes the plaintext copies the walk makes* (the per-node scalars it clones before sealing), where the scalar paths hold theirs in `Zeroizing`. A `serde_json::Value` does not zeroize; a follow-up should walk the caller's value without copying scalars out of it. +- *A stored document is read as untrusted input.* Its chain fields are not authenticated, and whoever holds a token can write tags that form a cycle or a chain deeper than any sealed document. `JsonDocument::children` checks each entry at its own ordinal instead of looping over a fan-out the document chooses; `JsonDocument::subtree` and the SQL walks stop below 127 levels, and the Rust walk also stops once it has returned as many entries as the document holds; the rebuild of a subtree's rows refuses a repeated row and a nesting deeper than 127 levels as `OpenedType`. Each would otherwise have overflowed the stack, looped, or allocated what the document asked for. (Phase 3b replaced `children` with a probe from ordinal 0 to the first miss, bounded by the entry count, and derives a child's token from its ordinal by hash, so no stored chain can form a cycle and the SQL walks need no depth bound; the Rust walk keeps both of its bounds and the rebuild keeps its own.) +- *Nothing wipes the plaintext copies the walk makes* (the per-node scalars it clones before sealing), where the scalar paths hold theirs in `Zeroizing`. A `serde_json::Value` does not zeroize; a follow-up should walk the caller's value without copying scalars out of it. (2026-10-23: done. The JSON plaintext is held under vitaminc's `Protected` as the scalar paths hold theirs. The walk no longer copies a scalar or a key out of the caller's value: it encodes each node's plaintext straight into a `Protected>`, sized before it is written so it never reallocates, and hands that to the cipher, which takes a `Protected` anyway; the sealed root's JSON is measured first and written once into its own. On the way back, the bytes the cipher opens stay in its `Protected>` while they are parsed, and every key, scalar and sealed-root document read out of them is wrapped as it is made (`zip_ref` over the bytes), in `Protected` for a value: `PlainJson` is a newtype over `serde_json::Value` whose `Zeroize` wipes every string, values and object keys alike, and overwrites numbers and booleans in place. A subtree is rebuilt inside a `Protected`, each member moved in from its own, and a value leaves the wrapper only where it is handed to the caller. The sealed-root fallback selects the walked path inside the document's `Protected`, so the whole document is wiped when the selection is made and only the selected nodes come out. What remains: numbers and booleans have no volatile overwrite without unsafe code, which this crate denies, so theirs is best-effort; and outside the decrypted plaintext, the query and term side still holds plaintext in places vitaminc does not reach from here: an accessor's needle document (`JsonPath::needle`), the canonical bytes of a value selector inside vitaminc's PAE `Context`, and the key names in each term node's positional path.) ## Implementation notes (2026-10-21, Phase 3: first measurements and the fixes they forced) The family was measured at 10,000 rows in the `cipherstash/benches` repository (branch `feat/json-tree-benches`, report `report/JSON_TREE_COMPARISON.md`; the first run at commit 076ebfc, the re-measurement of this phase's fixes at 57abcf9, both at 10,000 and 100,000 rows) against the v3 SteVec and native `jsonb`, on a flat five-node corpus and a nested corpus of about 220 nodes per document. The figures below are that report's at 10,000 rows unless they say otherwise. The flat corpus was on par with v3 (containment 0.47 ms against 0.23 ms, equality at a path 0.70 against 0.81 ms, the containment index eleven times smaller). The nested corpus was not: containment 70 to 80 ms per query, a 244-entry subtree 10 s, the GIN index 44 s to build. Three mechanisms accounted for it, and the fixes landed in the pull request stacked on the family's: - **The projection is stored, not rebuilt.** Decision 9 above chose to build the element-nested projection server-side from the tags (`eql_v4_internal.json_tree_group`, recursive plpgsql). That function ran once per candidate row in the lossy GIN recheck, and again in the Filter, at about 3 ms per 220-entry document: the whole cost of containment on the nested corpus, and of the index build. The client already derives the shape for the needle, so the document now writes it to the wire as `q` beside `sv`, the domain CHECK validates it as a group, and `eql_v4.json_tree_query(doc)` is `doc -> 'q'`. It returns `jsonb` rather than `eql_v4.query_json_tree`, because a domain cast runs the needle's recursive CHECK, and the recheck would pay it per row. And it is plpgsql with `COST 1000`, not an inlinable SQL function, for a reason the first re-measurement found: inlined to `value -> 'q'` the filter is costed as one operator, and the nested table's rows are TOASTed, so its main heap is a few hundred pages and a sequential scan over it was costed below the GIN bitmap scan and chosen, at 1.5 s for a one-row match on 10,000 rows (17 s on 100,000), because every row is decompressed to read `q`. Un-inlined, the call keeps its cost, the index expression `eql_v4.json_tree_query(col)` still matches it, and the planner takes the index: 0.5 to 0.7 ms for a one-row match, 3.2 ms for ten deep-object rows (0.36 ms native), and 5 s to build the index. Nothing new is revealed: `q` is the selectors already in `sv`, grouped as `sv`'s own `n` and `g` tags already group them. It is not small: 19,100 of a nested document's 114,309 characters, about 17 per cent, though nothing measurable on disk, since its hex pairs compress against `sv`. And it is trusted as written: the CHECK validates its shape, not that it is `sv`'s, so a row written by anything but this index can answer `@>` differently from `->`, the walks and decryption, which answer from `sv`. The server-built projection could not disagree with the entries; verifying `q` against `sv` at write time would restore that at roughly the old per-row cost, paid once per write instead of once per recheck, and is left open. The deviation this replaces, "9, the projection is built from the tags server-side", is withdrawn; the `groups_ok` fallback remains unneeded. -- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The `$.orders` subtree walks out in 23 ms (the six-entry `$.company` subtree in 2.8 ms); `children` and `child_count` in 1.6 ms, from 7.1. The report's prose calls `$.orders` the same 244 entries as the first run's 10.2 s, while its table counts 122 rows for the walked scenario; which count the 23 ms is for is still to be confirmed. The walk is bounded by the document as well as by depth: the fan-out it probes is at most the document's entry count, where a stored `o` could otherwise ask for two billion, and it returns at most as many rows as the document has entries, where a stored `w` that wraps a child's token to its parent's could otherwise double the rows at every level. Neither changes a sealed document's walk; the Postgres suite holds both against hostile documents (`postgres_walks_end_on_hostile_documents`). -- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it, over the new `JsonPath::select`, and the result is what the walked rows open to: the path is attached with `JsonSubtree::below(path)` for `json_tree_subtree`, whose rows are the descendants of the path's node, and `JsonSubtree::at(path)` for `json_tree_subtree_at`, whose rows are the nodes the path reaches (the same two on `JsonTreeSubtree` in the bindings). The two differ when the rows are opened as a list, to the node's children or to the node, which is why the rows cannot be opened without saying which walk they came from. A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. `$.orders` returns the root in 9.5 ms, against 23 ms walked out; `json_tree_subtree_at` over its items in 13 ms against 23. Two costs are not in those numbers: the client's, since the benchmarks do not decrypt, which for the fallback is one AES-GCM open and a JSON parse of a document about 1.3 times the plaintext's size; and the plaintext the client then holds, which is every field of the document and not the subtree asked for. The data key is the same, so nothing is readable that was not, but the note above that a `serde_json::Value` is never zeroized now covers the whole document on every subtree read past the budget. This is the sealed root paying for itself a second time: it is also what makes the fallback possible. +- **The walk reads the document once, keyed by parent tag.** The shipped walk put `json_tree_parent_tag(iv, t, o)` inside a LATERAL scan of `sv`, so it was evaluated once per (node, ordinal, entry): about 670,000 SHA-256 evaluations for the 244-entry subtree. The walk now builds one jsonb object of the chained entries keyed by `p` (`eql_v4_internal.json_tree_by_parent`) and looks each computed tag up in it, so a level costs its fan-out in hashes plus one lookup each, which is the design's `W × F` as written. The `$.orders` subtree walks out in 23 ms (the six-entry `$.company` subtree in 2.8 ms); `children` and `child_count` in 1.6 ms, from 7.1. The report's prose calls `$.orders` the same 244 entries as the first run's 10.2 s, while its table counts 122 rows for the walked scenario; which count the 23 ms is for is still to be confirmed. The walk is bounded by the document as well as by depth: the fan-out it probes is at most the document's entry count, where a stored `o` could otherwise ask for two billion, and it returns at most as many rows as the document has entries, where a stored `w` that wraps a child's token to its parent's could otherwise double the rows at every level. Neither changes a sealed document's walk; the Postgres suite held both against hostile documents (`postgres_walks_end_on_hostile_documents`). The ordinal chain of Phase 3b removes both cases rather than bounding them: a child's token is a hash of its parent's, so a chain cannot loop back, and the walk probes ordinals until the first miss rather than up to a stored fan-out. Its walk carries no depth bound either, so the same test now holds it to one row per entry on a chain of 300, deeper than any sealed document. +- **A walk has a budget and falls back to the sealed root.** With the root sealed, the root's ciphertext is the whole document at about 1.3 times the plaintext, where a walk returns about 440 characters per entry and pays the fan-out in hashes per node visited. Past a dozen or so entries the document is cheaper on both sides, so `json_tree_subtree` and `json_tree_subtree_at` take a `budget` (default 64) and, once the walk passes it, return the root entry alone with `d = 0`. The client opens the root to the whole document and takes the walked path over it, over the new `JsonPath::select`, and the result is what the walked rows open to: the path is attached with `JsonSubtree::below(path)` for `json_tree_subtree`, whose rows are the descendants of the path's node, and `JsonSubtree::at(path)` for `json_tree_subtree_at`, whose rows are the nodes the path reaches (the same two on `JsonTreeSubtree` in the bindings). The two differ when the rows are opened as a list, to the node's children or to the node, which is why the rows cannot be opened without saying which walk they came from. (2026-10-23, the owner's option (c) from the #1325 review: the caller no longer says it. The path rides on the key it was derived from, `JsonSubtreeKey::path`, and the key builds the rows' wrapper, `key.open(rows)` in stack-encrypt and `JsonTreeSubtree::from_key(&key, rows)` in the bindings; a key with no gates keyed `json_tree_subtree` and opens below the path's node, a key with gates keyed `json_tree_subtree_at` and opens at the nodes the path reaches. There is no path-less constructor and no `below` or `at` to attach, so rows that cannot open past the budget cannot be built, and a caller unaware of the budget is not caught out by a document that grows past it.) A budget of 0 never falls back, a token that names no node yields no row under any budget, and an unsealed root is always walked out. `$.orders` returns the root in 9.5 ms, against 23 ms walked out; `json_tree_subtree_at` over its items in 13 ms against 23. Two costs are not in those numbers: the client's, since the benchmarks do not decrypt, which for the fallback is one AES-GCM open and a JSON parse of a document about 1.3 times the plaintext's size; and the plaintext the client then holds, which is every field of the document and not the subtree asked for. The data key is the same, so nothing is readable that was not, but the note above that a `serde_json::Value` is never zeroized now covers the whole document on every subtree read past the budget. (2026-10-23: no longer; the fallback's document is held under `Protected` and wiped once the walked path is selected from it, and only the selected nodes reach the caller. See the Phases 1 and 2 note on plaintext copies.) This is the sealed root paying for itself a second time: it is also what makes the fallback possible. - **`json_tree_path_query_first` is costed at 1000.** `(col -> sel) > $1` extracts through it, and under a prepared statement's generic plan (what every driver reaches after five executions) the default cost chose a sequential scan with the extraction as a Filter over the functional btree: 16.9 ms against 0.11 ms with the index, at both tiers. Still open from the measurements, filed as a follow-up: `->` scans `sv` on every extraction (0.6 ms on a nested document, 0.2 on a flat one, against 0.1 native; an index of `sv` by `s` would make it a lookup); `json_tree_contains` plans a second GIN probe for `alt` even when the needle carries none; `ORDER BY ord_term(col -> sel)` measured 0.96 ms against v3's 0.25 ms on an index scan both ways, unexplained; and the nested document is 29 times its plaintext in characters and 34 times on disk (hex tags, a random nonce per entry, the sealed-root copy, and now `q`), which decisions 1 and 6 above and the opaque-subtree policy are the levers on. Also open, for the owner: whether the budget should be on by default (a caller who opens a walk's rows without attaching a path works until a document passes the budget, then fails), and whether to verify `q` at write time. +## Implementation notes (2026-10-22, Phase 3b: the ordinal-chained walk) + +A review of the stored form after the Phase 3 measurements found three fields nothing read and two the walk did not need, and replaced the chain step. The wire is now `{"v", "i", "h", "sv": [{s, c, p?, op?}], "q"}`. + +- **`vs`, `n` and `g` are gone from the entries.** Once `q` was stored (Phase 3), no SQL function read an entry's value selector or group tags: containment reads `q` alone, and the engine read them only to rebuild `q` on deserialize. The engine now takes the stored `q` as the projection. That is no change in trust: `q` and the entry fields were equally unauthenticated (only `c` is, through its AAD), and a tampered `q` changes which rows a containment query returns and nothing else. On the nested benchmark document (213 entries on average) the values of those three fields were 15% of `sv`. +- **Tokens chain by ordinal.** `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` replaces the label-derived step. The server derives every hit's token from its parent's as it walks, so `w` (the wrapped token) is not stored. The cost is that the client cannot derive an object member's token, because an ordinal is a rank among sibling keys it does not know. It derives a chain head's token and those of the array positions that directly follow the head, and sends one gate per further step (`JsonSubtreeKey::gates()`: the step's selector, or `#n` for a position); `json_tree_subtree_at(doc, token, gates, budget)` descends through them, and `json_tree_children` and `json_tree_child_count` take the same gates. A wildcard step's selector is every element's, which is the old `subtree_at` gate generalised to every depth. +- **The walk probes to the first miss.** The ordinals under a node are dense from zero (positions for arrays, a permutation for objects), so the walk tries `0, 1, 2, …` and stops at the first miss. `o` is not stored and `json_tree_fanout` is gone. A subtree of `N` nodes costs about `2N` probes (`N` hits, and a miss for every node visited), each a SHA-256 and a map lookup, plus a SHA-256 per hit for its token, where it cost `F` probes per visited node, `F` being the widest node in the whole document; on the nested corpus a wide `items` array taxed every object walk in the same document. +- **Depth is reported from the reached node.** A gated walk returns the node its gates reach with `d = 0` and its descendants beneath it, so the client can tell "the value at the path" (one reached node) from "the children of a node" (no `d = 0` row) without the path. A wildcard reaches several nodes and opens as a list. The ungated `json_tree_subtree(doc, token)` is now `json_tree_subtree_at` with no gates, and returns the token's node's descendants from `d = 1` as before. +- **Leakage.** At rest, the same as Phase 3 less what the stored ordinals told: their multiset gave, for every `k`, how many nodes in the chain have more than `k` children. `p` is still salted by the document `iv`. Under a token, the same: a token still unlocks the shape of that subtree in every row. The `w` mask's binding to `c` (a swapped ciphertext broke the walk on the server) is gone; the swap still fails when the client opens the entry, because `c` is bound to `s` and `j` in its AAD. +- **The chain step binds the child's selector (2026-10-23, review of #1329).** As first built, `t(child) = H("t" ‖ t(parent) ‖ LE32(o))` dropped the kind byte the label-derived step had (`0x01` for a key, `0x02` for an index), and an ordinal does not say which kind of child it numbers: an array element's is its position, an object member's its rank. Rows of one column need not agree on a node's kind, so the token the client derives for `$.a[0]` was, in a row whose `$.a` is an object, the token of the member ranked first, and a walk from it returned that member's subtree as the element's; a `#n` gate admitted the same member. `jsonb` has no node there (`jsonb_path_query_first(doc, 'strict $.a[0]')` is NULL), and the sealed-root fallback, which selects over the plaintext, agreed with `jsonb`, so one row answered differently under and over the budget. The step is now `t(child) = H("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`, the child's own selector bound in. The server holds every hit's `s`, so it derives the token as before (`eql_v4_internal.json_tree_child_token(token, s, o)`), and the client knows an element's selector from the path (the wildcard's), so it still derives positional tokens after a head. An element's selector is the wildcard path's and a member's names its key, so they never share a token. A position gate carries the element selector too, `#n:`, and the walk admits the hit at ordinal `n` only if it carries it. Nothing about leakage moves: the selector is stored beside the entry, and a token's holder already learns it with every hit. `postgres_a_position_reaches_array_elements_only` holds both forms against `jsonb`'s strict path over the plaintexts. + +### Measured (cipherstash/benches 2d9d5f2) + +Measured in `cipherstash/benches` (branch `feat/json-tree-benches`, commit 2d9d5f2, `report/JSON_TREE_COMPARISON.md`, "Third measurement") on the machine of the second Phase 3 measurement (Apple M1 Max, PostgreSQL 17.10), against the same documents: the ingest resealed the plaintext each row already held, so every scenario returned the same rows before and after. "Before" is the Phase 3 branch with its fixes, "after" this change. Query-only criterion medians over 10 samples, no decrypt, in milliseconds, on the nested corpus. + +| scenario | 10,000 rows | 100,000 rows | +|---|--:|--:| +| `subtree/orders`, default budget (falls back to the sealed root) | 9.48 → 5.71 (−40%) | 8.77 → 5.46 (−38%) | +| `subtree_at/items`, `$.orders[*].items`, default budget | 12.95 → 8.21 (−37%) | 10.92 → 7.18 (−34%) | +| `subtree/orders`, walked out (budget 0; 122 / 130 entries) | 22.97 → 10.92 (−52%) | 24.57 → 11.35 (−54%) | +| `subtree_at/items`, walked out (budget 0; 87 / 105 entries) | 23.12 → 10.93 (−53%) | 24.83 → 11.27 (−55%) | +| `subtree/company` | 2.84 → 1.99 (−30%) | 2.94 → 1.96 (−33%) | +| `subtree/user` | 3.90 → 2.38 (−39%) | 3.99 → 2.45 (−39%) | +| `children/orders` | 1.63 → 1.54 (−6%) | 1.64 → 1.49 (−9%) | +| `child_count/orders` | 1.62 → 1.53 (−6%) | 1.66 → 1.52 (−9%) | +| `path_query/wildcard` | 2.24 → 1.64 (−27%) | 1.80 → 1.41 (−22%) | +| `field_order/nested` | 4.81 → 3.81 (−21%) | 5.23 → 4.12 (−21%) | +| `extract/nested` | 0.59 → 0.47 (−19%) | 0.63 → 0.54 (−15%) | + +The walks the change was aimed at halved when walked out. Scenarios the walk does not touch improved too (`path_query`, `field_order`, `extract`): they read entries through `->` and the path functions, which scan an `sv` 30% shorter in a datum 24% smaller to decompress. `children` and `child_count` moved least; the report's explanation, not yet tested, is that both are dominated by the per-call fixed cost of detoasting the document and building the parent map over all its entries, which this change does not touch, and probing five to seven children instead of the document's widest fan-out saves only a few dozen hashes on top of it. That fixed cost is the next lever for every walk. + +Containment and field queries did not change beyond noise. The nested corpus's containment scenarios moved −12% to +7%. The flat corpus read +7% to +13% on the GIN-served containment and equality scenarios, but its operands, SQL, plans, projection and index are byte-identical before and after, and the native `jsonb` containment controls over unchanged plaintext, with freshly built indexes, drifted +5% to +19% in the same run; that points at the machine, not the format. The 100,000-row nested GIN build went from 58 s to 65 s on one sample, with the same projection and a smaller heap, which reads the same way. + +| size | before | after | +|---|--:|--:| +| nested document, characters (213 entries) | 113,458 | 85,157 (−25%) | +| nested `sv`, characters | 94,324 | 66,023 (−30%) | +| nested `q`, characters | 18,961 | 18,961 | +| nested `pg_column_size`, 10,000 rows | 81,264 B | 62,143 B (−24%) | +| nested heap with TOAST, 10,000 / 100,000 rows | 810 MB / 8,093 MB | 621 MB / 6,201 MB (−23%) | +| nested containment GIN, 10,000 / 100,000 rows | 40 MB / 358 MB | 40 MB / 358 MB | +| nested GIN build, 10,000 / 100,000 rows | 5 s / 58 s | 5 s / 65 s | +| flat document, characters (5 entries) | 2,346 | 2,136 (−9%) | +| flat `pg_column_size` | 2,452 B | 2,250 B (−8%) | + +`sv` shrank by 30% where 15% was expected from the three unread fields alone: the five removed fields carried 20,827 characters of values per nested document, and their keys and quotes about 7,500 more. The nested document is now 21 times its plaintext in characters (was 28) and 26 times in heap (was 34). The largest field is now `op`, the CLLW OPE term on the 156 number and string leaves: 24,832 characters a document. + +**Correctness.** A new benches task, `mise run check:v4:json_tree `, ran after ingest at both tiers. For 25 rows in a fixed pseudo-random order it decrypted every walk form the SQL returns and compared it with the row's plaintext: `json_tree_subtree` at `$.company`, `$.user` and `$.orders`, walked out and under the default budget (opening the sealed root at the path when the walk fell back); `json_tree_subtree_at` for `$.user.contact` (one gate) and `$.orders[*].items` (two gates); `json_tree_children` and `json_tree_child_count` at `$.orders`; every order by its position-derived token; `json_tree_path_query`; and `->`. For 10 sampled needles per predicate it compared the full id set of the encrypted containment, equality, range and ordering queries with native `jsonb` over the plaintext. 462 checks passed and none failed at each of 10,000 and 100,000 rows (350 walk checks, 112 predicate checks). + +## Implementation notes (2026-10-22, Phase 3c: the ordering term is opt-in per path) + +After the Phase 3b measurement the largest field in a stored document was `op`, the CLLW OPE term on every number and string leaf: 24,832 of the nested document's 85,157 characters, about 80 bytes of hex a leaf, on 156 leaves of which the benchmark's queries order two. An ordering term is also the one field that reveals something across rows at rest (the order of the leaf's value among every row that holds one), on fields nobody asked to order. "Further mechanisms" above proposed a per-path index policy with "everything indexed" as the default; the `op` half of it is now built, and the default is reversed: **no path is ordered unless the index says so.** + +- **`JsonOptions` gains `ordered`**, a set of `JsonPath`s, beside `sealed_root`. The walk derives a leaf's `op` only when some ordered path covers the node's positional path (`JsonPath::covers`): a key step matches the same key, a wildcard step covers any array position and the wildcard, a position covers that position alone. So `$.orders[*].total` orders the total of every order and `$.scores[0]` the first score only; literal positions and wildcards are both allowed. The set is part of the index's identity like `sealed_root`, and crosses the data-plan wire as `{"json": {"sealed_root": true, "ordered": ["$.age", "$.orders[*].total"]}}`, paths as text, written in canonical order. It is not recorded in the header: a stored leaf shows whether it carries a term. +- **An ordering query at a path the index does not order is refused** (`TermError::JsonPathNotOrdered`) when the query is awaited, before any keyset loads. Without the refusal, `ord_term` returns NULL for every stored leaf at the path and the comparison silently matches no row. A wildcard query path is covered by a wildcard only, since a position orders elements the wildcard would compare against. +- **A position in an ordered path is refused (2026-10-23, review of #1329).** As first built, `$.scores[2]` ordered the third score alone, and an ordering query at `$.scores[2]` under an index ordering `$.scores[*]` was accepted. Neither works: an ordering compares the leaf `->` extracts by the path's selector, and a selector reads every position as the wildcard, so `->` returns the first element at the path. A query at an ordered `$.scores[2]` compared the first score, which carries no term, and matched no row, which is the failure the refusal above exists to prevent; a query at `$.scores[2]` under `$.scores[*]` compared the first score and returned rows for the wrong element. `JsonOptions::ordering`, the data plan's `ordered` and an ordering query now refuse a path that names a position (`TermError::JsonOrderingAtPosition`), and sealing under options whose `ordered` field holds one, which only direct construction can produce, is refused the same way. Ordering at an array element by position would need an extractor that reaches a position, which `->` by selector is not; it is not scheduled. +- **A plan can be laid out by a value.** The typed plan had no place for options: `encrypt_into::()` reads the type's own `EncryptFrom`, which is static, so every typed plan sealed under the defaults and a query under other options was refused as `PlanError::IndexOptions`. The new `target::Layout` trait is a target's layout chosen by value, with the index it answers queries through; `Json::new(options)` implements it for `JsonDocument`, and `Plan::value().encrypt_as(Json::new(options))` builds a `ValuePlan<_, Laid>` that seals under the options and resolves every query through the plan's own index, so a query does not name the options again. The EQL type has the same by value: `eql_bindings::v4::JsonTree::new(options)` lays a value out as a `JsonTreeDocument` under an identifier. +- **Nothing changes in SQL.** `ord_term` was already NULL for an entry without `op`, and the entry validator already admitted its absence; the functional btree on `eql_v4.ord_term(col -> sel)` indexes the leaves at one selector, which under the opt-in are the leaves of one ordered path. + +What remains of the per-path policy is the `vs` half (equality off for a low-cardinality field) and opaque subtrees; neither is scheduled. + +### Measured (cipherstash/benches cd9643d) + +The benches re-sealed both corpora with one ordered path each, the one their range and ordering scenarios use (`$.age` on the flat corpus, `$.company.salary` on the nested), at 10,000 and 100,000 rows, and the correctness check passed at both tiers (462 of 462). "Before" is the Phase 3b form with `op` on every number and string leaf. + +| | flat, 10,000 rows | nested, 10,000 rows | nested, 100,000 rows | +|---|--:|--:|--:| +| `pg_column_size` per row | 2,250 → 1,596 B (−29%) | 62,143 → 40,578 B (−35%) | 62,092 → 40,549 B | +| heap with TOAST | 27.1 → 17.0 MiB | 620.9 → 407.5 MiB (−34%) | 6,201 → 4,069 MiB | +| containment GIN | 0.98 → 1.03 MiB | 39.9 MiB, unchanged | 358 MiB, unchanged | +| total with indexes | 28.4 → 18.4 MiB | 662.0 → 448.6 MiB | 6,569 → 4,437 MiB | + +The nested document is now 18 times its plaintext on disk (was 26 at Phase 3b, 40 at Phase 3). Two side effects: a flat document at 1,596 bytes sits under PostgreSQL's TOAST threshold, so the flat heap moves from TOAST into the main relation (0.5 to 17 MiB at 10,000 rows) and the earlier note about a TOASTed column making a sequential scan look cheap no longer applies to that corpus; and the flat GIN grew 12% at 100,000 rows with an identical projection, because the rows now spread over many more heap pages and the index's row-address deltas are wider. Query timings were not re-run for this measurement. + +## Implementation notes (2026-10-23, Phase 3d: byte encodings and the eight-byte parent tag) + +Decision 1 below (the token encoding on the wire) is taken, and the parent tag is shortened with it. The rule is the one "Further mechanisms" set out: a field the SQL compares, derives or decodes must be in an alphabet core PostgreSQL encodes and decodes natively, and only a field the SQL never opens may take a denser one. + +- **Every field the SQL reads is padded standard base64**, as `encode(x, 'base64')` writes it and `decode(x, 'base64')` reads it: the selectors `s` and the `s`/`vs` pairs of `q` (sixteen bytes, 24 characters where hex took 32), the parent tag `p`, the ordering term `op` (54 characters where hex took 80), and the tokens and gates `json_tree_subtree_at`, `json_tree_children` and `->` take. `h` already was. The padding stays because `decode(x, 'base64')` refuses the unpadded form (`invalid base64 end sequence`, PostgreSQL 17.10): two characters on a sixteen-byte field and one on an eight-byte one, against a padding step written into every SQL site that would ever decode one. base64url was not taken for the same reason: PostgreSQL has no decoder for it. +- **The ciphertext `c` is RFC 1924 base85** (`base85-simd` 0.1.0: NEON on aarch64, AVX2 on x86_64, portable scalar code elsewhere, wasm32 included), five characters per four bytes against base64's eight per six, on the field that is most of a document. No SQL function decodes `c`. The alphabet has no `"` or `\`, so a JSON string carries it unescaped. `h` stays base64: the SQL reads the `iv` and the sealed-root flag out of it, and it is one field a document. +- **The parent tag `p` is the first eight bytes of its hash.** A parent tag is a lookup key for the server's walk and nothing else: an entry's AAD binds its selector and its ordinal, not its parent tag, so truncation authenticates nothing less. It has to be distinct among the entries of one document only, where eight bytes put a collision in a ten-thousand-node document below one in 10^11. The walk's misses are the other place a short tag could bite: a probe for an ordinal that does not exist that matches some stored tag would return an unrelated entry as a child. A walk of `N` nodes makes about `N` such probes against `N` stored tags, so the chance is the same order, about `N² / 2^64` a walk and row. The row it would add is authentic and opens; the client's rebuild checks each container's child count and each element's position, which catches it in most shapes but not all, so the bound is the probability, not a check. `eql_v4_internal.json_tree_parent_tag` is `encode(substr(h16(…), 1, 8), 'base64')`, twelve characters where hex took 32. +- **`op` needs its own extractor in SQL**, `eql_v4_internal.json_tree_ope_cllw(jsonb)`, because the `ope_cllw(jsonb)` every ordered scalar type shares reads hex and those wires are released. `eql_v4.ord_term` on an entry and on an ordering operand call it; the ordering operators and the functional btree are unchanged. + +### Measured (1,000 stored rows of each corpus, re-encoded and reloaded as jsonb, PostgreSQL 17.10) + +The rows were taken from the Phase 3c tables (one ordered path a corpus), re-encoded field by field the way the code now writes them, and both forms loaded into fresh tables; `pg_column_size` is the average over the 1,000 rows after `VACUUM FULL`. + +| | flat (5 nodes) | nested (219 nodes) | +|---|--:|--:| +| wire characters per document | 1,419 → 1,224 (−14%) | 58,111 → 47,038 (−19%) | +| `pg_column_size` per row | 1,596 → 1,398 B (−12%) | 40,808 → 31,894 B (−22%) | +| table with TOAST, 1,000 rows | 1,648 → 1,608 kB | 41 → 33 MB | + +The nested document is now about 14 times its plaintext on disk (18 at Phase 3c, 40 at Phase 3). The byte saving exceeds the character saving: pglz compressed hex better than it compresses base64 and base85, but not by enough to matter. The benchmark corpora are not re-ingested and no query is re-timed in this note; the walk, containment and ordering paths are covered by the oracle suite, which passed against the bundle built from the tree. Of the two levers left from the Phase 3 list, one is closed and one is open. **The derived nonce is rejected (2026-10-23).** It would save the twelve random bytes in every `c`, about 7% of a nested document, but it needs vitaminc to expose a caller-chosen nonce (a nonce strategy that reads the leaf's AAD, and a nonce-less leaf wire form), and a caller-chosen nonce is exactly what vitaminc's AEAD layer refuses to offer: its misuse resistance rests on never letting a caller repeat a (key, nonce) pair, and the safety argument here ("every seal mints a fresh data key") is an invariant of stack-encrypt that vitaminc could not check. Seven percent does not buy weakening that promise; the random nonce stays. Unpadded base64 (two characters a tag, about 3%) remains open and untaken. + ## Decisions to take before Phase 1 -1. **Token encoding on the wire:** hex, matching v3's `s`, or base64url, saving about 40 characters per entry. Proposal: measure in Phase 3, default to base64url. +> **2026-10-22:** these are the open questions as they stood before Phase 1; the implementation notes above record how each was taken. Since then, decision 4's group tags exist only as the nesting of the stored projection `q`, decision 9 was settled by storing the projection (Phase 3), and decision 10's ordinal is not stored (Phase 3b). + +1. **Token encoding on the wire:** hex, matching v3's `s`, or base64url, saving about 40 characters per entry. Proposal: measure in Phase 3, default to base64url. Taken 2026-10-23 (Phase 3d): padded standard base64, not base64url, since PostgreSQL decodes only the former; base85 for `c`. 2. **Leaf ordering terms bound to the column only, or to the column and the selector.** Column-only reuses the v4 scalar ordering query domains unchanged and matches the ADR-0009 appendix; selector-bound stops cross-path order comparison at the cost of a JSON-specific query domain. Proposal: column-only for the phased plan, revisit in the ADR. 3. **Array index modes.** Wildcard only, with the chain serving positional access, retiring `[@]` and `[n]`. Proposal: yes. 4. **Group tags on by default**, or opt-in per column for callers who accept the v3 behaviour in exchange for not revealing element grouping. Proposal: on by default; `Contains` is declared exact and that declaration has to be true. diff --git a/languages/golang/encrypt/guest/Cargo.lock b/languages/golang/encrypt/guest/Cargo.lock index c715159ff..c932c38f4 100644 --- a/languages/golang/encrypt/guest/Cargo.lock +++ b/languages/golang/encrypt/guest/Cargo.lock @@ -168,6 +168,12 @@ version = "1.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" +[[package]] +name = "base85-simd" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd16ef96137ca89cc38b0acb5bee36fada9707c044bc9c753f55c816bfa719" + [[package]] name = "bitflags" version = "2.13.1" @@ -2163,6 +2169,7 @@ name = "stack-encrypt" version = "0.3.0" dependencies = [ "base64ct", + "base85-simd", "cllw-ore", "miette", "ore-rs", diff --git a/packages/eql/Cargo.lock b/packages/eql/Cargo.lock index 0b689259b..24b13d5f4 100644 --- a/packages/eql/Cargo.lock +++ b/packages/eql/Cargo.lock @@ -279,6 +279,12 @@ dependencies = [ "thiserror 1.0.69", ] +[[package]] +name = "base85-simd" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd16ef96137ca89cc38b0acb5bee36fada9707c044bc9c753f55c816bfa719" + [[package]] name = "bit-set" version = "0.8.0" @@ -4266,6 +4272,7 @@ name = "stack-encrypt" version = "0.3.0" dependencies = [ "base64ct", + "base85-simd", "cllw-ore 0.5.2-rc.1", "miette", "ore-rs 0.9.0", diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index f110c3e99..a3f5249f8 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -16,13 +16,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `JsonTreeEntry` is one entry as `->` and the walks return it, `JsonTreeQuery` the containment needle and `JsonTreeLeafQuery` the ordering operand at a path. `JsonTreeSubtree` collects a walk's rows; past - the walk's budget the server returns the sealed root instead, and - `JsonTreeSubtree::below(path)` (for `json_tree_subtree`) or `at(path)` - (for `json_tree_subtree_at`) attaches the path the rows then open at. + the walk's budget the server returns the sealed root instead, which + opens at the path the walk's key was derived from: the rows are built + with `JsonTreeSubtree::from_key(&key, rows)`, and there is no + constructor without the key. The document is an `EncryptFrom` and `DecryptInto` target, the entry and the subtree open through `DecryptInto`, and the needle and the ordering operand convert from Stack Encrypt's `JsonQuery` and - `JsonLeafQuery`. + `JsonLeafQuery`. `JsonTree::new(options)` is the `json_tree` layout under + index options other than the defaults (a `Layout`, so a program + seals with `keyset.run(JsonTree::new(options).encryption(), ..)`), such + as the paths that carry an ordering term. The wire fields are typed apart + from the v3 hex terms: `JsonTreeSelector` (`s`, and the pairs of `q`), + `JsonTreeTag` (`p`) and `JsonTreeOrderingTerm` (`op`), padded base64, and + `JsonTreeCiphertext` (`c`), base85. - **Range query operands for `eql_v4.between`** (`stack-encrypt` feature). Every ordered v4 family has two: `IntegerRangeOreQuery` and `IntegerRangeOpeQuery`, `TextRangeOreQuery` and `TextRangeOpeQuery`, and so diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index 0629678bf..d93b9a919 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -23,31 +23,54 @@ use schemars::{schema_for, JsonSchema, Schema}; use serde::{Deserialize, Serialize}; use serde_json::Value; use stack_encrypt::json::{ - JsonDocument, JsonLeafQuery, JsonNode, JsonPath, JsonQuery, JsonSubtree, + JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree, + JsonSubtreeKey, }; -use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec}; +use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec, Json, Layout}; use stack_encrypt::{ DecryptField, DecryptInto, Decryptable, Decryption, EncryptFrom, Encryption, Error, NonEmpty, }; use ts_rs::TS; -use crate::v3::terms::{EntryCiphertext, KeyHeader, OpeCllw, Selector}; +use crate::v3::terms::KeyHeader; use crate::v4::{DomainType, SchemaVersion}; use crate::Identifier; -/// A sixteen-byte chain tag, hex: a parent tag `p`, a wrapped token `w`, -/// a group tag `n` or `g`. Per document, salted by the document's `iv`, so -/// unlinkable across rows at rest. +/// A path selector `s` (and the value selector `vs` of a projection pair): +/// sixteen bytes, padded standard base64, as PostgreSQL's `encode(x, +/// 'base64')` writes it. Column-wide: the same path has the same selector +/// in every row, which is what `->` and containment match on. +#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)] +pub struct JsonTreeSelector(pub String); + +/// A node's ciphertext `c`: RFC 1924 base85 (`base85-simd`), which no SQL +/// function decodes. The AES-GCM envelope of the node's label, kind, child +/// count and scalar value under the document's data key. +#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)] +pub struct JsonTreeCiphertext(pub String); + +/// An eight-byte parent tag `p`, padded base64: the first eight bytes of +/// the parent-tag hash, per document, salted by the document's `iv`, so +/// unlinkable across rows at rest. `eql_v4.json_tree_subtree_at` derives +/// the same text with `encode(substr(…, 1, 8), 'base64')` to find a child. #[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)] pub struct JsonTreeTag(pub String); +/// A leaf ordering term `op`, padded base64: the CLLW OPE term of a number +/// or string leaf at a path the index orders. `eql_v4.ord_term` decodes it +/// to the bytea-backed `eql_v4_internal.ope_cllw` domain. +#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)] +pub struct JsonTreeOrderingTerm(pub String); + /// `public.eql_v4_json_tree`: a JSON tree document, `{v, i, h, sv: /// [entry], q: group}`. Strict. `h` is the document's key header, stored /// once (every entry is sealed under the document's one data key); `sv` /// holds one entry per node in depth-first order, the root first; `q` is -/// the document's containment projection, its `{s, vs}` pairs nested by -/// array element, which the client writes beside the entries so the -/// server's `eql_v4.json_tree_query` reads it rather than rebuilding it. +/// the document's containment projection, one `{s, vs}` pair per node +/// nested by array element, which the client writes beside the entries +/// and which is the only place the value selectors live: the server's +/// `eql_v4.json_tree_query` reads it, and the engine takes it as the +/// projection when it reads a document back. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeDocument { @@ -58,36 +81,22 @@ pub struct JsonTreeDocument { pub q: JsonTreeQueryGroup, } -/// `public.eql_v4_json_tree_entry`: one node. `s` and `vs` are the path -/// and value selectors, `c` the node's own ciphertext; `o`, `p` and `w` -/// the chain fields below the heads; `n` and `g` the group tags of -/// container array elements and what sits inside them; `op` the leaf -/// ordering term. `h`, `j`, `d`, `v` and `i` are what the extractors graft -/// onto an entry they return, so it opens on its own. +/// `public.eql_v4_json_tree_entry`: one node. `s` is the path selector, +/// `c` the node's own ciphertext; `p` the parent tag below the chain's +/// heads, which the walk probes by ordinal; `op` the leaf ordering term. +/// `h`, `j`, `d`, `v` and `i` are what the extractors graft onto an entry +/// they return, so it opens on its own. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeEntry { - pub s: Selector, - pub vs: Selector, - pub c: EntryCiphertext, - #[serde(default, skip_serializing_if = "Option::is_none")] - #[ts(optional)] - pub o: Option, + pub s: JsonTreeSelector, + pub c: JsonTreeCiphertext, #[serde(default, skip_serializing_if = "Option::is_none")] #[ts(optional)] pub p: Option, #[serde(default, skip_serializing_if = "Option::is_none")] #[ts(optional)] - pub w: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - #[ts(optional)] - pub n: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - #[ts(optional)] - pub g: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - #[ts(optional)] - pub op: Option, + pub op: Option, #[serde(default, skip_serializing_if = "Option::is_none")] #[ts(optional)] pub h: Option, @@ -109,8 +118,8 @@ pub struct JsonTreeEntry { #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeQueryPair { - pub s: Selector, - pub vs: Selector, + pub s: JsonTreeSelector, + pub vs: JsonTreeSelector, } /// One group of a containment needle: the pairs that sit directly in it, @@ -143,63 +152,45 @@ pub struct JsonTreeQuery { #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeLeafQuery { - pub s: Selector, - pub op: OpeCllw, + pub s: JsonTreeSelector, + pub op: JsonTreeOrderingTerm, } /// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or -/// `json_tree_subtree_at`), collected: opens to the value under the -/// walked node, or to the list of complete subtrees the rows hold. +/// `json_tree_subtree_at`), collected with the key that keyed the walk: +/// opens to the value under the walked path, or to the list of complete +/// subtrees the rows hold. /// /// Past its budget a walk returns the sealed root alone, at depth `0`; -/// those rows open to the whole document, and the path the walk was keyed -/// from selects the same value(s) the walked rows open to. Attach it with -/// [`below`](Self::below) for `json_tree_subtree` rows and -/// [`at`](Self::at) for `json_tree_subtree_at` rows. The path is not part -/// of the wire. -#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] -#[serde(transparent)] +/// those rows open to the whole document, and the key's path selects the +/// same value(s) the walked rows open to. The key also says which walk the +/// rows came from (no gates: `json_tree_subtree`, the rows below the path's +/// node; gates: `json_tree_subtree_at`, the nodes the path reaches), so the +/// only constructor, [`from_key`](Self::from_key), takes it, and rows +/// cannot be opened without it. +#[derive(Clone, Debug, PartialEq)] pub struct JsonTreeSubtree { - pub rows: Vec, - #[serde(skip)] - pub path: Option, - /// Whether `path` names the node `json_tree_subtree` walked below - /// ([`below`](Self::below)), rather than the nodes a - /// `json_tree_subtree_at` path reaches ([`at`](Self::at)). - #[serde(skip)] - pub below: bool, + rows: Vec, + key: JsonSubtreeKey, } impl JsonTreeSubtree { - /// The rows, as returned. - pub fn new(rows: Vec) -> Self { + /// The rows the walk `key` keyed returned, as the server sent them. + pub fn from_key(key: &JsonSubtreeKey, rows: Vec) -> Self { Self { rows, - path: None, - below: false, + key: key.clone(), } } - /// The rows of `json_tree_subtree_at`, with the whole path its key was - /// derived from: past the budget they open to one value per node the - /// path reaches, as the walked rows do. - pub fn at(self, path: JsonPath) -> Self { - Self { - path: Some(path), - below: false, - ..self - } + /// The rows. + pub fn rows(&self) -> &[JsonTreeEntry] { + &self.rows } - /// The rows of `json_tree_subtree`, with the path of the node whose - /// token keyed the walk: past the budget they open to that node, or as - /// a list to its children, as the walked rows do. - pub fn below(self, path: JsonPath) -> Self { - Self { - path: Some(path), - below: true, - ..self - } + /// The path the walk was keyed from. + pub fn path(&self) -> &JsonPath { + self.key.path() } } @@ -261,9 +252,11 @@ impl JsonTreeDocument { serde_json::from_value(value).map_err(codec) } - /// The engine's document: the header and entries without the envelope. + /// The engine's document: the header, entries and projection without + /// the envelope. fn document(&self) -> Result { - serde_json::from_value(serde_json::json!({ "h": self.h, "sv": self.sv })).map_err(codec) + serde_json::from_value(serde_json::json!({ "h": self.h, "sv": self.sv, "q": self.q })) + .map_err(codec) } } @@ -305,19 +298,62 @@ impl JsonTreeEntry { // Stack Encrypt targets // ============================================================================= -/// A document seals as the JSON index lays it out, under the column it is -/// stored in: the [`Identifier`] is the context the data key is minted -/// under and every selector derives under, and is written to `i`. +/// The `json_tree` layout under the JSON index's options: what seals a +/// value into a [`JsonTreeDocument`] under options other than the +/// defaults, such as the paths that carry an ordering term +/// ([`JsonOptions::ordered`]). A [`Layout`], so a Rust program runs it as +/// `keyset.run(JsonTree::new(options).encryption(), &value, column)`, and +/// its index is the engine's [`Json`] under the same options, which a +/// query derives through. [`JsonTreeDocument`]'s own [`EncryptFrom`] is +/// this layout under the defaults. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct JsonTree { + index: Json, +} + +impl JsonTree { + /// The layout under `options`. + pub fn new(options: JsonOptions) -> Self { + Self { + index: Json::new(options), + } + } + + /// The index the layout seals under, with its options. + pub fn index(&self) -> &Json { + &self.index + } +} + +/// A document seals as the JSON index lays it out under the layout's +/// options, under the column it is stored in: the [`Identifier`] is the +/// context the data key is minted under and every selector derives +/// under, and is written to `i`. +impl Layout for JsonTree { + type Target = JsonTreeDocument; + type Index = Json; + fn encryption<'s, K: 'static>( + &self, + ) -> Encryption<'s, Value, JsonTreeDocument, K, NonEmpty> { + Layout::::encryption::(&self.index) + .accepting::>() + .map_with_context(|document, i: NonEmpty| (i, document)) + .try_map(|(i, document)| JsonTreeDocument::envelope(i.into_inner(), &document)) + } + fn index(&self) -> &Json { + &self.index + } +} + +/// A document seals as the JSON index lays it out under the default +/// options ([`JsonTree::default`]): no path ordered, the root sealed. impl EncryptFrom for JsonTreeDocument { type Context = NonEmpty; fn encryption<'s, K: 'static>() -> Encryption<'s, Value, Self, K, Self::Context> { - >::encryption() - .accepting::>() - .map_with_context(|document, i: NonEmpty| (i, document)) - .try_map(|(i, document)| Self::envelope(i.into_inner(), &document)) + JsonTree::default().encryption() } fn indexes() -> Vec { - >::indexes() + JsonTree::default().specs() } } @@ -377,7 +413,7 @@ impl DecryptInto for JsonTreeSubtree { } /// The rows of a walk open to the complete subtrees they hold, in order: -/// one per node whose selector `json_tree_subtree_at` matched. +/// one per node the gates of `json_tree_subtree_at` reached. impl DecryptInto> for JsonTreeSubtree { type Context = NonEmpty; fn decryption(self, context: Self::Context) -> Decryption, K> { @@ -390,17 +426,12 @@ impl DecryptInto> for JsonTreeSubtree { impl JsonTreeSubtree { fn subtree(&self) -> Result { - let rows = JsonSubtree::new( + Ok(self.key.open( self.rows .iter() .map(JsonTreeEntry::node) .collect::>()?, - ); - Ok(match &self.path { - Some(path) if self.below => rows.below(path.clone()), - Some(path) => rows.at(path.clone()), - None => rows, - }) + )) } } @@ -446,7 +477,7 @@ mod tests { fn a_document_carries_its_projection() { let document = serde_json::json!({ "v": 4, "i": {"t": "users", "c": "profile"}, "h": "AA==", - "sv": [{"s": "aa", "vs": "bb", "c": "AA=="}], + "sv": [{"s": "aa", "c": "AA=="}], "q": {"sv": [{"s": "aa", "vs": "bb"}], "el": []}, }); let stored: JsonTreeDocument = serde_json::from_value(document.clone()).unwrap(); @@ -460,31 +491,10 @@ mod tests { ); } - #[test] - fn a_subtree_carries_its_path_beside_the_wire() { - let rows: JsonTreeSubtree = serde_json::from_value(serde_json::json!([])).unwrap(); - assert!(rows.path.is_none()); - let at = rows.at(JsonPath::parse("$.a").unwrap()); - assert_eq!( - at.path.as_ref().map(ToString::to_string).as_deref(), - Some("$.a") - ); - assert_eq!(serde_json::to_value(&at).unwrap(), serde_json::json!([])); - assert!(at.subtree().unwrap().path().is_some()); - assert!(!at.below); - let below = at.below(JsonPath::parse("$.b").unwrap()); - assert!(below.below); - assert_eq!( - below.subtree().unwrap(), - JsonSubtree::new(Vec::new()).below(JsonPath::parse("$.b").unwrap()) - ); - assert_eq!(serde_json::to_value(&below).unwrap(), serde_json::json!([])); - } - #[test] fn an_entry_without_the_grafted_fields_does_not_open() { let entry: JsonTreeEntry = serde_json::from_value(serde_json::json!({ - "s": "00", "vs": "00", "c": "AA==" + "s": "00", "c": "AA==" })) .unwrap(); assert!(entry.node().is_err()); diff --git a/packages/eql/src/v3/json_tree/functions.sql b/packages/eql/src/v3/json_tree/functions.sql index 3b35559fb..3208ccafa 100644 --- a/packages/eql/src/v3/json_tree/functions.sql +++ b/packages/eql/src/v3/json_tree/functions.sql @@ -14,8 +14,8 @@ --! extension, no superuser, so the surface installs wherever the rest of --! {{prefix}} does. --! ---! `selector` parameters are *encrypted-side* selectors, the hex the index ---! emits in `s`; `token` parameters are the hex of a chain token the client +--! `selector` parameters are *encrypted-side* selectors, the base64 the index +--! emits in `s`; `token` parameters are the base64 of a chain token the client --! derived from a path. Plaintext paths are never accepted. ------------------------------------------------------------------------------ @@ -33,26 +33,6 @@ AS $$ SELECT substr(sha256(x), 1, 16) $$; ---! @brief XOR of two sixteen-byte strings. ---! @internal ---! @param a bytea Sixteen bytes. ---! @param b bytea Sixteen bytes. ---! @return bytea `a XOR b`, byte by byte. -CREATE FUNCTION {{prefix}}_internal.json_tree_xor16(a bytea, b bytea) - RETURNS bytea - LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE -AS $$ -DECLARE - r bytea := a; - i int; -BEGIN - FOR i IN 0..15 LOOP - r := set_byte(r, i, get_byte(a, i) # get_byte(b, i)); - END LOOP; - RETURN r; -END; -$$; - --! @brief A 32-bit integer as four little-endian bytes. --! @internal --! @param n integer The integer, non-negative. @@ -79,56 +59,40 @@ AS $$ $$; --! @brief The parent tag a child at ordinal `o` under the node with token ---! `token` carries: `H("p" ‖ iv ‖ token ‖ LE32(o))`. +--! `token` carries: the first eight bytes of +--! `H("p" ‖ iv ‖ token ‖ LE32(o))`. Eight, since the tag is a lookup +--! key within one document and nothing else: an entry's AAD binds +--! its selector and ordinal, not its parent tag. --! @internal --! @param iv bytea The document's iv. --! @param token bytea The parent's token, sixteen bytes. --! @param o integer The child's ordinal. ---! @return text The tag, hex, as the entry's `p` spells it. +--! @return text The tag, padded base64, as the entry's `p` spells it. CREATE FUNCTION {{prefix}}_internal.json_tree_parent_tag(iv bytea, token bytea, o integer) RETURNS text LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT encode({{prefix}}_internal.json_tree_h16('p'::bytea || iv || token || {{prefix}}_internal.json_tree_le32(o)), 'hex') + SELECT encode(substr({{prefix}}_internal.json_tree_h16('p'::bytea || iv || token || {{prefix}}_internal.json_tree_le32(o)), 1, 8), 'base64') $$; ---! @brief Unwrap a child's token from its entry, given its parent's token: ---! `w XOR H("w" ‖ iv ‖ token ‖ c)`. +--! @brief The token of the child at ordinal `o` under the node with token +--! `token`, whose selector is `s`: `H("t" ‖ token ‖ s ‖ LE32(o))`. +--! Column-wide like every token, with no document salt, so a client +--! derives an array element's token from the path alone and the walk +--! derives a hit's token from its parent's and the hit's own `s` on +--! the spot. Nothing stores a token. The selector keeps an array +--! element apart from an object member at the same ordinal, which a +--! row whose node at the path is an object would otherwise put there. --! @internal ---! @param iv bytea The document's iv. ---! @param token bytea The parent's token. ---! @param entry jsonb The child's entry, carrying `w` and `c`. +--! @param token bytea The parent's token, sixteen bytes. +--! @param s bytea The child's selector, sixteen bytes. +--! @param o integer The child's ordinal. --! @return bytea The child's token, sixteen bytes. -CREATE FUNCTION {{prefix}}_internal.json_tree_unwrap(iv bytea, token bytea, entry jsonb) +CREATE FUNCTION {{prefix}}_internal.json_tree_child_token(token bytea, s bytea, o integer) RETURNS bytea LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}_internal.json_tree_xor16( - decode(entry ->> 'w', 'hex'), - {{prefix}}_internal.json_tree_h16('w'::bytea || iv || token || decode(entry ->> 'c', 'base64')) - ) -$$; - ---! @brief How many ordinals a walk tries under each node: one more than the ---! largest ordinal any entry of the document carries, and never more ---! than the document has entries. ---! ---! A node's children sit at ordinals below its child count, which is below ---! the document's entry count, so the cap changes nothing for a sealed ---! document. It is there for a stored one: `o` is not authenticated, and ---! without the cap one entry carrying `o = 2147483646` would make every ---! level of every walk hash two billion candidate tags. ---! ---! @internal ---! @param val jsonb The document. ---! @return integer The fan-out bound; `0` for a document with no chained ---! entry. -CREATE FUNCTION {{prefix}}_internal.json_tree_fanout(val jsonb) - RETURNS integer - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE -AS $$ - SELECT LEAST(COALESCE(MAX((elem ->> 'o')::integer), -1) + 1, count(*)::integer) - FROM jsonb_array_elements(val -> 'sv') AS elem + SELECT {{prefix}}_internal.json_tree_h16('t'::bytea || token || s || {{prefix}}_internal.json_tree_le32(o)) $$; --! @brief Whether the document's root is sealed: its header's sealed-root @@ -145,12 +109,30 @@ AS $$ SELECT (get_byte(decode(h, 'base64'), 1) & 1) = 1 $$; +--! @brief The CLLW OPE term of a JSON tree entry or ordering operand: its +--! `op`, base64-decoded. The JSON tree spells `op` in base64 like +--! its other fields, where the scalar types' shared +--! `{{prefix}}_internal.ope_cllw(jsonb)` reads hex, so the family +--! has its own extractor. +--! @internal +--! @param val jsonb An entry or a `query_json_tree_leaf`, as jsonb. +--! @return {{prefix}}_internal.ope_cllw The term; NULL when there is none. +CREATE FUNCTION {{prefix}}_internal.json_tree_ope_cllw(val jsonb) + RETURNS {{prefix}}_internal.ope_cllw + LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE +AS $$ + SELECT decode(val ->> 'op', 'base64')::{{prefix}}_internal.ope_cllw +$$; + +COMMENT ON FUNCTION {{prefix}}_internal.json_tree_ope_cllw(jsonb) IS + 'eql-inline-critical: raw-jsonb CLLW OPE extractor of the JSON tree family; must stay inlinable (unpinned search_path)'; + --! @brief The chained entries of a document keyed by parent tag: one object --! whose keys are each entry's `p` and whose values are the entry --! with its ordinal `j` in `sv` added. A walk looks a child up by --! the tag it computes, `map -> tag`, instead of scanning `sv` for ---! it, so the document is read once per walk and a level costs its ---! fan-out in hashes, not its fan-out times the document's size. +--! it, so the document is read once per walk and a probe costs one +--! hash and one lookup, not a pass over the document. --! @internal --! @param val jsonb The document. --! @return jsonb The map; `{}` for a document with no chained entry. @@ -209,7 +191,7 @@ $$; --! own: it carries the document's header `h` and its own ordinal `j`. --! --! @param val jsonb The document. ---! @param selector text The path selector, hex. +--! @param selector text The path selector, base64. --! @return SETOF public.{{prefix}}_json_tree_entry The matching entries, in --! document order. CREATE FUNCTION {{prefix}}.json_tree_path_query(val jsonb, selector text) @@ -235,7 +217,7 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_query(jsonb, text) IS --! operand's selectivity is a guess, the default cost chose the scan. --! --! @param val jsonb The document. ---! @param selector text The path selector, hex. +--! @param selector text The path selector, base64. --! @return public.{{prefix}}_json_tree_entry The entry, or NULL when the document --! has no node at that path. CREATE FUNCTION {{prefix}}.json_tree_path_query_first(val jsonb, selector text) @@ -255,7 +237,7 @@ COMMENT ON FUNCTION {{prefix}}.json_tree_path_query_first(jsonb, text) IS --! @brief Whether the document has a node at the path `selector` names. --! @param val jsonb The document. ---! @param selector text The path selector, hex. +--! @param selector text The path selector, base64. --! @return boolean True when some entry carries the selector. CREATE FUNCTION {{prefix}}.json_tree_path_exists(val jsonb, selector text) RETURNS boolean @@ -372,142 +354,207 @@ $$; -- The token chain: children and subtrees by token ------------------------------------------------------------------------------ ---! The walk, in every function below: for the node in hand, compute the ---! parent tag a child at each ordinal under the document's fan-out would ---! carry, and look each tag up in the document's parent map ---! ({{prefix}}_internal.json_tree_by_parent). A hit is a child; its token ---! unwraps under the parent's and the walk recurses. The map and the ---! document's header fields are computed once per call, so a level costs ---! its fan-out in SHA-256 evaluations and one lookup each, and the ---! document is never scanned per node. - ---! @brief The children of the node whose token is `token`, as rows. +--! The walk, in every function below. A node's children sit at the dense +--! ordinals `0, 1, 2, …` under it, and a child's parent tag is +--! `H("p" ‖ iv ‖ t(parent) ‖ LE32(o))`. So from a token the walk probes +--! ordinal 0, then 1, and so on, looking each tag up in the document's +--! parent map ({{prefix}}_internal.json_tree_by_parent), and stops at the +--! first miss. A hit's own token is `H("t" ‖ t(parent) ‖ s ‖ LE32(o))`, +--! `s` being the hit's own selector, derived on the spot +--! ({{prefix}}_internal.json_tree_child_token), and the walk recurses under +--! it. A subtree of `N` nodes therefore costs about +--! `2N` probes, `N` hits and a miss for every node visited (the end of a +--! parent's children, or a leaf's lack of any), each probe one SHA-256 and +--! one map lookup, plus one SHA-256 per hit for its token: about `3N` +--! hashes whatever the document's width. The map and the header fields +--! are computed once per call, so the document is read once. +--! +--! **Gates.** An object member's ordinal is a function of its siblings' +--! keys, which the client does not know, so the client derives a token +--! only for a chain head and the array positions that directly follow it, +--! and walks the rest through one gate per further step: either the step's selector, +--! which the entry at that depth must carry in `s`, or `#n:`, an +--! array position, which is the one ordinal probed at that depth and whose +--! entry must carry the array elements' selector, so that an object member +--! ranked at ordinal `n` in a row whose node there is an object is not +--! taken for an element. A wildcard step's selector is every element's, so +--! it admits them all. Entries are returned from the depth of the last gate +--! down: the node each gate chain reaches, and everything beneath it. + +--! @brief The children of the node the gates reach from the node whose +--! token is `token`, as rows. --! --! What `jsonb_array_elements` is for an encrypted array: one row per ---! element, in ordinal order, each opening on its own. Each child's token ---! is unwrapped on the way, though not returned. +--! element, in the document's order, each opening on its own. With no +--! gates, the children of the token's own node; with gates, the walk +--! descends through them as `json_tree_subtree_at` does and returns one +--! level below the node they reach. --! --! @param e public.{{prefix}}_json_tree The stored document. ---! @param token text The parent's token, hex: derived by the client from the ---! path alone, and never stored. +--! @param token text The start node's token, base64: derived by the client +--! from the path alone, and never stored. +--! @param gates text[] One gate per step below the token, as +--! `json_tree_subtree_at` takes them. Default none. --! @return SETOF public.{{prefix}}_json_tree_entry The children, each with `d = 1`. -CREATE FUNCTION {{prefix}}.json_tree_children(e public.{{prefix}}_json_tree, token text) +CREATE FUNCTION {{prefix}}.json_tree_children(e public.{{prefix}}_json_tree, token text, gates text[] DEFAULT ARRAY[]::text[]) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - WITH + WITH RECURSIVE doc AS MATERIALIZED ( SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, - {{prefix}}.json_tree_meta(e::jsonb) AS meta + {{prefix}}.json_tree_meta(e::jsonb) AS meta, + COALESCE(array_length(gates, 1), 0) AS gated + ), + -- The walk of `json_tree_subtree_at`, cut one level below the gates. + walk(depth, pt, o, entry, t, ppass, pass) AS ( + SELECT 0, NULL::bytea, NULL::integer, NULL::jsonb, decode(token, 'base64'), true, true + UNION ALL + SELECT probe.depth, probe.pt, probe.o, hit.entry, + {{prefix}}_internal.json_tree_child_token(probe.pt, decode(hit.entry ->> 's', 'base64'), probe.o), + probe.ppass, + probe.ppass AND ( + probe.depth > doc.gated + OR hit.entry ->> 's' = CASE WHEN gates[probe.depth] LIKE '#%' + THEN split_part(gates[probe.depth], ':', 2) + ELSE gates[probe.depth] END) + FROM walk AS w + CROSS JOIN doc + CROSS JOIN LATERAL ( + SELECT w.depth + 1 AS depth, + w.t AS pt, + CASE WHEN COALESCE(gates[w.depth + 1], '') LIKE '#%' + THEN split_part(substr(gates[w.depth + 1], 2), ':', 1)::integer + ELSE 0 END AS o, + w.pass AS ppass + WHERE w.pass AND w.depth <= doc.gated + UNION ALL + SELECT w.depth, w.pt, w.o + 1, w.ppass + WHERE w.entry IS NOT NULL AND COALESCE(gates[w.depth], '') NOT LIKE '#%' + ) AS probe + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, probe.pt, probe.o) AS entry + ) AS hit + WHERE hit.entry IS NOT NULL ) - SELECT (doc.meta || c.entry || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry - FROM doc - CROSS JOIN generate_series(0, doc.fanout - 1) AS o - CROSS JOIN LATERAL ( - SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, decode(token, 'hex'), o) AS entry - ) AS c - WHERE c.entry IS NOT NULL - ORDER BY o + SELECT (doc.meta || w.entry || '{"d": 1}'::jsonb)::public.{{prefix}}_json_tree_entry + FROM walk AS w, doc + WHERE w.entry IS NOT NULL AND w.pass AND w.depth = doc.gated + 1 + ORDER BY (w.entry ->> 'j')::integer $$; ---! @brief How many children the node whose token is `token` has: the length ---! of an encrypted array, or the member count of an object. +--! @brief How many children the node the gates reach from `token` has: +--! the length of an encrypted array, or the member count of an +--! object. --! @param e public.{{prefix}}_json_tree The stored document. ---! @param token text The node's token, hex. +--! @param token text The start node's token, base64. +--! @param gates text[] One gate per step below the token. Default none. --! @return integer The count; `0` for a node with no children, a token that --! names no node included. -CREATE FUNCTION {{prefix}}.json_tree_child_count(e public.{{prefix}}_json_tree, token text) +CREATE FUNCTION {{prefix}}.json_tree_child_count(e public.{{prefix}}_json_tree, token text, gates text[] DEFAULT ARRAY[]::text[]) RETURNS integer LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT count(*)::integer FROM {{prefix}}.json_tree_children(e, token) + SELECT count(*)::integer FROM {{prefix}}.json_tree_children(e, token, gates) $$; ---! @brief Every entry under the node whose token is `token`, as rows: the ---! recursive walk of the chain, in core SQL. +--! @brief The entries under the node the gates reach from the node whose +--! token is `token`, as rows: the recursive walk of the chain, in +--! core SQL. --! ---! Starting from the token, each level finds its children by parent tag, ---! unwraps their tokens, and recurses. A token that names no node yields no ---! row, and nothing in a row is linkable to anything without the token. The ---! node itself is not returned: the walk addresses nodes by their parent, ---! and the client reads the node's kind from its children's labels, or ---! fetches the node by its selector. ---! The walk descends at most 127 levels, the deepest a sealed document ---! nests, and returns at most as many rows as the document has entries, ---! which a tree never exceeds. A stored document whose tags form a longer ---! chain or a loop therefore still ends it, after at most its entry count ---! times its fan-out in hashes. +--! Each level probes the ordinals under the node in hand until the first +--! miss, derives each hit's token, and recurses. A token that names no node +--! yields no row, and nothing in a row is linkable to anything without the +--! token. At depth `d` (1 below the token) the gate `gates[d]`, if there +--! is one, decides which hits the walk descends through: an entry whose +--! `s` is the gate's selector, or, for a gate `#n:`, the one hit +--! at ordinal `n` if it carries that selector. Rows are returned from the depth of the last gate down: the +--! node a gate chain reaches, with `d = 0`, and everything beneath it, with +--! `d` its depth below that node. The start node is never returned. With +--! no gates, every descendant of the start node, with `d` its depth below +--! it, from 1. --! ---! **The budget.** A walk costs the fan-out in hashes per node it visits, ---! and returns about four hundred characters per entry, where a sealed root ---! holds the whole document in one ciphertext of about the plaintext's ---! size. So when the root is sealed and the subtree has more than `budget` ---! entries, the walk stops and returns the root entry alone, with `d = 0`: ---! the client opens it to the whole document and takes the path itself, ---! which is cheaper on both sides than the walk past that point. The ---! default of 64 is past the crossover by bytes (about a dozen entries) by ---! a margin that keeps a small subtree arriving as the entries it is. A ---! `budget` of `0` never falls back. A token that names no node yields no ---! row under any budget, and a document whose root is not sealed is always ---! walked in full, since its root holds no document to return. +--! **The budget.** A walk returns about two hundred characters per entry, +--! where a sealed root holds the whole document in one ciphertext of about +--! the plaintext's size. So when the root is sealed and the walk would +--! return more than `budget` entries, it stops and returns the root entry +--! alone, with `d = 0`: the client opens it to the whole document and takes +--! the path itself, which is cheaper on both sides than the walk past that +--! point. The default of 64 is past the crossover by bytes (about a dozen +--! entries) by a margin that keeps a small subtree arriving as the entries +--! it is. A `budget` of `0` never falls back. A token that names no node +--! yields no row under any budget, and a document whose root is not sealed +--! is always walked in full, since its root holds no document to return. --! --! @param e public.{{prefix}}_json_tree The stored document. ---! @param token text The subtree's token, hex. +--! @param token text The start node's token, base64. +--! @param gates text[] One gate per step below the token: a selector, base64, +--! or `#n:` for the array position `n`, with the array +--! elements' selector. Empty for the whole subtree. --! @param budget integer The most entries to return before falling back to ---! the sealed root; `0` to always walk the subtree out. Default 64. ---! @return SETOF public.{{prefix}}_json_tree_entry The descendants, each with its ---! depth `d` below the node, in depth-first order; or the root ---! entry alone, with `d = 0`, past the budget. -CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, budget integer DEFAULT 64) +--! the sealed root; `0` to always walk out. Default 64. +--! @return SETOF public.{{prefix}}_json_tree_entry The entries, each with its +--! depth `d` below the node the gates reach, in document order; or +--! the root entry alone, with `j = 0` and `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, gates text[], budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ WITH RECURSIVE doc AS MATERIALIZED ( SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, {{prefix}}.json_tree_meta(e::jsonb) AS meta, e::jsonb -> 'sv' -> 0 AS root, - jsonb_array_length(e::jsonb -> 'sv') AS entries, - budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped + budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped, + COALESCE(array_length(gates, 1), 0) AS gated ), - walk(depth, t, entry) AS ( - SELECT 0, decode(token, 'hex'), NULL::jsonb + -- One row per hit, after the seed row for the start node. `pt` and `o` + -- are where the hit sits, its parent's token and its ordinal; `t` is its + -- own token; `ppass` whether every gate down to its parent admitted the + -- parent, `pass` whether this hit's gate admits it too. + walk(depth, pt, o, entry, t, ppass, pass) AS ( + SELECT 0, NULL::bytea, NULL::integer, NULL::jsonb, decode(token, 'base64'), true, true UNION ALL - SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), - c.entry + SELECT probe.depth, probe.pt, probe.o, hit.entry, + {{prefix}}_internal.json_tree_child_token(probe.pt, decode(hit.entry ->> 's', 'base64'), probe.o), + probe.ppass, + probe.ppass AND ( + probe.depth > doc.gated + OR hit.entry ->> 's' = CASE WHEN gates[probe.depth] LIKE '#%' + THEN split_part(gates[probe.depth], ':', 2) + ELSE gates[probe.depth] END) FROM walk AS w CROSS JOIN doc - CROSS JOIN generate_series(0, doc.fanout - 1) AS o CROSS JOIN LATERAL ( - SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry - ) AS c - -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a longer chain, which whoever holds a token can - -- write, stops here. A cycle that branches is stopped by `visited`. - WHERE c.entry IS NOT NULL AND w.depth < 127 - ), - -- A walk of a sealed document reaches each entry at most once, so it - -- has at most as many rows as the document has entries, the seed row - -- standing in for the root. A stored document can loop instead: `w` - -- is not authenticated, so whoever holds a token can wrap a child's - -- token to its parent's, and two such children double the rows at - -- every level the depth bound allows. PostgreSQL computes a recursive - -- query only as far as its reader pulls rows, so this limit stops the - -- recursion, and not only the output. - visited AS MATERIALIZED ( - SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) + -- Down: the first ordinal under this node, or the one position an + -- array gate names, from a node its gates admitted. + SELECT w.depth + 1 AS depth, + w.t AS pt, + CASE WHEN COALESCE(gates[w.depth + 1], '') LIKE '#%' + THEN split_part(substr(gates[w.depth + 1], 2), ':', 1)::integer + ELSE 0 END AS o, + w.pass AS ppass + WHERE w.pass + UNION ALL + -- Across: the next ordinal under the same parent, from any hit, + -- since a hit its gate refused still has siblings to find; not + -- when an array gate pins this depth to one position. + SELECT w.depth, w.pt, w.o + 1, w.ppass + WHERE w.entry IS NOT NULL AND COALESCE(gates[w.depth], '') NOT LIKE '#%' + ) AS probe + CROSS JOIN LATERAL ( + SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, probe.pt, probe.o) AS entry + ) AS hit + WHERE hit.entry IS NOT NULL ), -- One row past the budget is enough to know the walk went over; a walk -- under it is the subtree complete. Uncapped, the limit is NULL: all. found AS MATERIALIZED ( SELECT w.depth, w.entry - FROM visited AS w - WHERE w.entry IS NOT NULL + FROM walk AS w, doc + WHERE w.entry IS NOT NULL AND w.pass AND w.depth >= doc.gated LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) ), over AS MATERIALIZED ( @@ -521,97 +568,27 @@ AS $$ WHERE over.fallback UNION ALL SELECT (f.entry ->> 'j')::integer, - (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry + (doc.meta || f.entry || jsonb_build_object('d', f.depth - doc.gated))::public.{{prefix}}_json_tree_entry FROM found AS f, doc, over WHERE NOT over.fallback ) AS rows ORDER BY rows.j $$; ---! @brief The subtrees under the node whose token is `token` whose path ---! continues through a node with selector `selector`: how a path ---! with a wildcard below its last positional step is walked. ---! ---! The walk starts at the token and emits an entry once it has passed ---! through, or reached, an entry carrying the selector: for ---! `$.orders[*].items` it starts at `$.orders` and returns every element's ---! `items` subtree, each complete. The budget is `json_tree_subtree`'s, ---! counted over the entries emitted: past it, the sealed root alone, with ---! `d = 0`, and the client takes the wildcard path over the whole document. ---! +--! @brief Every entry under the node whose token is `token`: +--! `json_tree_subtree_at` with no gates. --! @param e public.{{prefix}}_json_tree The stored document. ---! @param token text The anchor's token, hex. ---! @param selector text The whole path's selector, hex. +--! @param token text The subtree's token, base64. --! @param budget integer The most entries to return before falling back to ---! the sealed root; `0` to always walk out. Default 64. ---! @return SETOF public.{{prefix}}_json_tree_entry The matching subtrees' entries, ---! each with its depth `d` below the anchor, in depth-first order; ---! or the root entry alone, with `d = 0`, past the budget. -CREATE FUNCTION {{prefix}}.json_tree_subtree_at(e public.{{prefix}}_json_tree, token text, selector text, budget integer DEFAULT 64) +--! the sealed root; `0` to always walk the subtree out. Default 64. +--! @return SETOF public.{{prefix}}_json_tree_entry The descendants, each with its +--! depth `d` below the node, in document order; or the root entry +--! alone, with `d = 0`, past the budget. +CREATE FUNCTION {{prefix}}.json_tree_subtree(e public.{{prefix}}_json_tree, token text, budget integer DEFAULT 64) RETURNS SETOF public.{{prefix}}_json_tree_entry LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - WITH RECURSIVE - doc AS MATERIALIZED ( - SELECT {{prefix}}_internal.json_tree_iv(e::jsonb ->> 'h') AS iv, - {{prefix}}_internal.json_tree_fanout(e::jsonb) AS fanout, - {{prefix}}_internal.json_tree_by_parent(e::jsonb) AS map, - {{prefix}}.json_tree_meta(e::jsonb) AS meta, - e::jsonb -> 'sv' -> 0 AS root, - jsonb_array_length(e::jsonb -> 'sv') AS entries, - budget > 0 AND {{prefix}}_internal.json_tree_sealed_root(e::jsonb ->> 'h') AS capped - ), - walk(depth, t, entry, hit) AS ( - SELECT 0, decode(token, 'hex'), NULL::jsonb, false - UNION ALL - SELECT w.depth + 1, - {{prefix}}_internal.json_tree_unwrap(doc.iv, w.t, c.entry), - c.entry, - w.hit OR c.entry ->> 's' = selector - FROM walk AS w - CROSS JOIN doc - CROSS JOIN generate_series(0, doc.fanout - 1) AS o - CROSS JOIN LATERAL ( - SELECT doc.map -> {{prefix}}_internal.json_tree_parent_tag(doc.iv, w.t, o) AS entry - ) AS c - -- No sealed document nests deeper than 127 levels. A stored one - -- whose tags form a longer chain, which whoever holds a token can - -- write, stops here. A cycle that branches is stopped by `visited`. - WHERE c.entry IS NOT NULL AND w.depth < 127 - ), - -- A walk of a sealed document reaches each entry at most once, so it - -- has at most as many rows as the document has entries, the seed row - -- standing in for the root. A stored document can loop instead: `w` - -- is not authenticated, so whoever holds a token can wrap a child's - -- token to its parent's, and two such children double the rows at - -- every level the depth bound allows. PostgreSQL computes a recursive - -- query only as far as its reader pulls rows, so this limit stops the - -- recursion, and not only the output. - visited AS MATERIALIZED ( - SELECT w.* FROM walk AS w LIMIT (SELECT doc.entries FROM doc) - ), - found AS MATERIALIZED ( - SELECT w.depth, w.entry - FROM visited AS w - WHERE w.entry IS NOT NULL AND w.hit - LIMIT (SELECT CASE WHEN doc.capped THEN budget + 1 END FROM doc) - ), - over AS MATERIALIZED ( - SELECT doc.capped AND (SELECT count(*) FROM found) > budget AS fallback FROM doc - ) - SELECT rows.entry - FROM ( - SELECT 0 AS j, - (doc.meta || doc.root || '{"j": 0, "d": 0}'::jsonb)::public.{{prefix}}_json_tree_entry AS entry - FROM doc, over - WHERE over.fallback - UNION ALL - SELECT (f.entry ->> 'j')::integer, - (doc.meta || f.entry || jsonb_build_object('d', f.depth))::public.{{prefix}}_json_tree_entry - FROM found AS f, doc, over - WHERE NOT over.fallback - ) AS rows - ORDER BY rows.j + SELECT * FROM {{prefix}}.json_tree_subtree_at(e, token, ARRAY[]::text[], budget) $$; ------------------------------------------------------------------------------ @@ -620,7 +597,7 @@ $$; --! @brief The path selector of an entry. --! @param entry public.{{prefix}}_json_tree_entry The entry. ---! @return text Its `s`, hex. +--! @return text Its `s`, base64. CREATE FUNCTION {{prefix}}.selector(entry public.{{prefix}}_json_tree_entry) RETURNS text LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE @@ -631,7 +608,7 @@ $$; --! @brief The leaf ordering term of an entry, for ordered comparison. --! --! The CLLW OPE term the index derived for a number or string leaf, keyed by ---! the column, hex-decoded to the bytea-backed `{{prefix}}_internal.ope_cllw` +--! the column, base64-decoded to the bytea-backed `{{prefix}}_internal.ope_cllw` --! domain: native byte order is plaintext order, so a functional btree on --! `{{prefix}}.ord_term(col -> 'selector')` engages with the DEFAULT opclass. NULL --! for an entry with no term (a container, a boolean, a null), which btree's @@ -648,23 +625,23 @@ CREATE FUNCTION {{prefix}}.ord_term(entry public.{{prefix}}_json_tree_entry) RETURNS {{prefix}}_internal.ope_cllw LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}_internal.ope_cllw(entry::jsonb) + SELECT {{prefix}}_internal.json_tree_ope_cllw(entry::jsonb) $$; --! @brief The leaf ordering term of an ordering operand. --! @param q {{prefix}}.query_json_tree_leaf The operand. ---! @return {{prefix}}_internal.ope_cllw Its `op`, hex-decoded. +--! @return {{prefix}}_internal.ope_cllw Its `op`, base64-decoded. CREATE FUNCTION {{prefix}}.ord_term(q {{prefix}}.query_json_tree_leaf) RETURNS {{prefix}}_internal.ope_cllw LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT {{prefix}}_internal.ope_cllw(q::jsonb) + SELECT {{prefix}}_internal.json_tree_ope_cllw(q::jsonb) $$; --! @brief The path selector of an ordering operand: what `->` takes to reach --! the leaf the operand compares against. --! @param q {{prefix}}.query_json_tree_leaf The operand. ---! @return text Its `s`, hex. +--! @return text Its `s`, base64. CREATE FUNCTION {{prefix}}.selector(q {{prefix}}.query_json_tree_leaf) RETURNS text LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE diff --git a/packages/eql/src/v3/json_tree/operators.sql b/packages/eql/src/v3/json_tree/operators.sql index 0d81b48b0..fbb30e1e4 100644 --- a/packages/eql/src/v3/json_tree/operators.sql +++ b/packages/eql/src/v3/json_tree/operators.sql @@ -24,7 +24,7 @@ --! returns native jsonb semantics, as it does on every EQL jsonb domain. --! --! @param e public.{{prefix}}_json_tree The stored document. ---! @param selector text The path selector, hex. +--! @param selector text The path selector, base64. --! @return public.{{prefix}}_json_tree_entry The entry, or NULL. CREATE FUNCTION {{prefix}}."->"(e public.{{prefix}}_json_tree, selector text) RETURNS public.{{prefix}}_json_tree_entry @@ -42,7 +42,7 @@ CREATE OPERATOR ->( --! @brief ->> operator with a text selector: the first entry at the path, as --! JSON text. It does not decrypt. --! @param e public.{{prefix}}_json_tree The stored document. ---! @param selector text The path selector, hex. +--! @param selector text The path selector, base64. --! @return text The entry as text, or NULL. CREATE FUNCTION {{prefix}}."->>"(e public.{{prefix}}_json_tree, selector text) RETURNS text diff --git a/packages/eql/src/v3/json_tree/types.sql b/packages/eql/src/v3/json_tree/types.sql index 108477f4d..a4b36e0d4 100644 --- a/packages/eql/src/v3/json_tree/types.sql +++ b/packages/eql/src/v3/json_tree/types.sql @@ -20,39 +20,41 @@ --! --! The design is `docs/plans/2026-10-17-json-tree-index.md` --! at the repository root. The document's per-entry fields: ---! `s` path selector (16 bytes, hex): every element of an array shares its +--! `s` path selector (16 bytes, base64): every element of an array shares its --! parent's, so a path reads `$.a[*].x`. ---! `vs` value selector (hex): the node's kind and, for a scalar, its value, ---! bound to `s`. Containment is a set comparison over `{s, vs}` pairs. ---! `c` the node's own ciphertext (base64): its label, its kind and, for a ---! scalar, its value. The root's is the whole document when the root is ---! sealed (the default). ---! `o` the node's ordinal under its parent, `p` the parent tag ---! `H("p" ‖ iv ‖ t(parent) ‖ LE32(o))` and `w` the node's own token ---! wrapped under its parent's: present below the chain's heads. ---! `n` the group tag of an array element that is a container, ---! `g` the group tag of the nearest enclosing such element. ---! `op` the leaf ordering term (CLLW OPE, hex) on number and string leaves. +--! `c` the node's own ciphertext (RFC 1924 base85, which no SQL function +--! decodes): its label, its kind and, for a scalar, its value. The +--! root's is the whole document when the root is sealed (the default). +--! `p` the parent tag, the first eight bytes of +--! `H("p" ‖ iv ‖ t(parent) ‖ LE32(o))` (base64), where `o` is the +--! node's ordinal under its parent: present below the chain's heads. +--! The ordinals under a node are dense from 0, so a walk holding +--! `t(parent)` probes them in turn until the first miss, and derives +--! each hit's own token as `H("t" ‖ t(parent) ‖ s ‖ LE32(o))` from +--! the hit's selector. Nothing stores a token or an ordinal. +--! `op` the leaf ordering term (CLLW OPE, base64) on number and string leaves. --! The header `h` (base64) carries the document's data key material once; the --! SQL reads only the sealed-root flag at byte offset 1 and the ZeroKMS `iv` ---! at byte offset 22 of it, the per-document salt of the chain tags, and +--! at byte offset 22 of it, the per-document salt of the parent tags, and --! otherwise carries it opaquely. `q` is the document's containment ---! projection, `{"sv": [{s, vs}, …], "el": [group, …]}`: its `{s, vs}` ---! pairs nested by array element, written by the client beside `sv` so the ---! server compares it with a needle by native `jsonb @>` and never ---! rebuilds it from the entries. The CHECK validates its shape, not that it ---! is `sv`'s: containment answers from `q` as written, so a row written by ---! anything but Stack Encrypt's JSON index can answer `@>` differently ---! from `->`, the walks and decryption, which answer from `sv`. +--! projection, `{"sv": [{s, vs}, …], "el": [group, …]}`: one `{s, vs}` pair +--! per node, `vs` being the value selector of the node's kind and, for a +--! scalar, its value, bound to `s`, nested by array element. The client +--! writes it beside `sv` and it is the only place the value selectors +--! live: the server compares it with a needle by native `jsonb @>`. +--! The CHECK validates its shape, not that it agrees with `sv`: containment +--! answers from `q` as written, so a row written by anything but Stack +--! Encrypt's JSON index can answer `@>` differently from `->`, the walks +--! and decryption, which answer from `sv`. --! @brief Validate one JSON tree entry. --! @internal --! @param val jsonb Candidate entry. ---! @return boolean True when `val` is an object with string `s`, `vs` and `c`, ---! an optional numeric `o`, optional string `p`, `w`, `n`, `g` and ---! `op`, and the metadata the extractors graft: string `h`, numeric ---! `j` (the entry's ordinal in the document) and `d` (its depth below ---! a walk's start), and the envelope's `v` and `i`. No other key. +--! @return boolean True when `val` is an object with string `s` and `c`, +--! optional string `p` and `op`, and the metadata the extractors +--! graft: string `h`, numeric `j` (the entry's ordinal in the +--! document) and `d` (its depth below a walk's start), and the +--! envelope's `v` and `i`. No other key. CREATE OR REPLACE FUNCTION public.{{prefix}}_is_valid_json_tree_entry_payload(val jsonb) RETURNS boolean LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE @@ -60,18 +62,13 @@ AS $$ SELECT COALESCE( jsonb_typeof(val) = 'object' AND jsonb_typeof(val -> 's') = 'string' - AND jsonb_typeof(val -> 'vs') = 'string' AND jsonb_typeof(val -> 'c') = 'string' - AND (NOT (val ? 'o') OR jsonb_typeof(val -> 'o') = 'number') AND (NOT (val ? 'p') OR jsonb_typeof(val -> 'p') = 'string') - AND (NOT (val ? 'w') OR jsonb_typeof(val -> 'w') = 'string') - AND (NOT (val ? 'n') OR jsonb_typeof(val -> 'n') = 'string') - AND (NOT (val ? 'g') OR jsonb_typeof(val -> 'g') = 'string') AND (NOT (val ? 'op') OR jsonb_typeof(val -> 'op') = 'string') AND (NOT (val ? 'h') OR jsonb_typeof(val -> 'h') = 'string') AND (NOT (val ? 'j') OR jsonb_typeof(val -> 'j') = 'number') AND (NOT (val ? 'd') OR jsonb_typeof(val -> 'd') = 'number') - AND val - ARRAY['s', 'vs', 'c', 'o', 'p', 'w', 'n', 'g', 'op', 'h', 'j', 'd', 'v', 'i']::text[] = '{}'::jsonb, + AND val - ARRAY['s', 'c', 'p', 'op', 'h', 'j', 'd', 'v', 'i']::text[] = '{}'::jsonb, false ) $$; diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index 0250b449d..5b499117f 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -11,12 +11,13 @@ mod common; use eql_bindings::v4::json_tree::{ - JsonTreeDocument, JsonTreeEntry, JsonTreeLeafQuery, JsonTreeQuery, JsonTreeSubtree, + JsonTree, JsonTreeDocument, JsonTreeEntry, JsonTreeLeafQuery, JsonTreeQuery, JsonTreeSubtree, }; use eql_bindings::Identifier; use serde_json::{json, Map, Value}; -use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath}; -use stack_encrypt::plan::{Typed, ValuePlan}; +use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath, JsonSubtreeKey}; +use stack_encrypt::plan::{Laid, ValuePlan}; +use stack_encrypt::target::{Json, Layout}; use stack_encrypt::{op, NonEmpty, Plan}; fn column() -> NonEmpty { @@ -29,10 +30,12 @@ fn fixture() -> Value { type Cipher = stack_encrypt::StackCipher; +/// Seal `value` as the EQL type under the identifier, laid out by the +/// `json_tree` layout with the same ordered paths as [`plan`]. async fn seal(cipher: &Cipher, value: &Value) -> JsonTreeDocument { cipher .default_keyset() - .encrypt_as::(value, column()) + .run(JsonTree::new(options()).encryption(), value, column()) .await .unwrap() } @@ -40,19 +43,30 @@ async fn seal(cipher: &Cipher, value: &Value) -> JsonTreeDocument { /// The plan a query derives through: the engine's own document target /// under the label `users/profile`, which is the same context as the /// identifier the EQL target stores (the same two parts, rendered -/// `users/profile`), so its selectors are the stored document's. An EQL -/// type takes its context as an identifier and runs through `encrypt_as`, -/// not through the chain. -fn plan() -> ValuePlan> { +/// `users/profile`), so its selectors are the stored document's, laid +/// out by the JSON index with `$.hello` and `$.number` ordered, the paths +/// the ordering queries below ask at. An EQL type takes its context as +/// an identifier and runs through `encrypt_as`, not through the chain. +fn plan() -> ValuePlan> { Plan::value::() .context("users/profile") - .encrypt_into::() + .encrypt_as(Json::new(options())) .build() .unwrap() } -fn hex(bytes: &[u8]) -> String { - bytes.iter().map(|b| format!("{b:02x}")).collect() +/// The index options the corpus is sealed under: `$.hello` and +/// `$.number` carry ordering terms. +fn options() -> JsonOptions { + JsonOptions::default() + .ordering(["$.hello", "$.number"]) + .unwrap() +} + +/// Padded standard base64, as the wire spells a selector. +fn base64(bytes: &[u8]) -> String { + use base64::Engine; + base64::engine::general_purpose::STANDARD.encode(bytes) } #[tokio::test] @@ -150,9 +164,13 @@ async fn the_query_operands_are_the_engines_terms_in_the_sql_contract() { let typed = seal(&cipher, &fixture()).await; assert_eq!(engine.entries().len(), typed.sv.len()); for (a, b) in engine.entries().iter().zip(&typed.sv) { - assert_eq!(hex(a.selector()), b.s.0); - assert_eq!(hex(a.value_selector()), b.vs.0); + assert_eq!(base64(a.selector()), b.s.0); } + assert_eq!( + serde_json::to_value(engine.projection()).unwrap(), + serde_json::to_value(&typed.q).unwrap(), + "and the same projection" + ); let opened: Value = cipher.open(engine).using(&plan).await.unwrap(); assert_eq!(opened, fixture()); } @@ -466,7 +484,7 @@ fn postgres_extracts_walks_and_orders_by_path() { &runtime, keyset.json_subtree_key( &JsonPath::parse(path).unwrap(), - JsonOptions::default(), + &JsonOptions::default(), column(), ), ) @@ -522,6 +540,7 @@ fn postgres_extracts_walks_and_orders_by_path() { // the path alone: six entries in row 1, reassembling the sub-document; // two in row 2. let nested = subtree_key("$.nested"); + assert!(nested.gates().is_empty(), "a chain head's token"); for (id, expected) in [ (1, fixture()["nested"].clone()), (2, rows[1].1["nested"].clone()), @@ -541,7 +560,7 @@ fn postgres_extracts_walks_and_orders_by_path() { .all(|e| e.d.is_some() && e.j.is_some() && e.h.is_some())); let value: Value = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::from_key(&nested, walked), column()), ) .unwrap(); assert_eq!(value, expected, "row {id}"); @@ -574,17 +593,11 @@ fn postgres_extracts_walks_and_orders_by_path() { let root = walked(&mut db, 5); assert_eq!(root.len(), 1, "past the budget, the root alone"); assert_eq!((root[0].j, root[0].d), (Some(0), Some(0))); - let refused = wait( - &runtime, - cipher.decrypt_as::(JsonTreeSubtree::new(root.clone()), column()), - ); - assert!(refused.is_err(), "the fallback needs the walked path"); + // The key that keyed the walk builds the rows, so the path the + // fallback is opened at is always there. let value: Value = wait( &runtime, - cipher.decrypt_as( - JsonTreeSubtree::new(root).below(JsonPath::parse("$.nested").unwrap()), - column(), - ), + cipher.decrypt_as(JsonTreeSubtree::from_key(&nested, root), column()), ) .unwrap(); assert_eq!(value, fixture()["nested"]); @@ -596,42 +609,56 @@ fn postgres_extracts_walks_and_orders_by_path() { .unwrap() .get(0); assert_eq!(garbage, 0); - // The fallback opens to what the walk opens to, as a list too: the - // list's elements, whether the server walked its four entries or - // returned the sealed root, so a caller's result does not change shape - // when a document grows past the budget. + // The fallback opens to what the walk opens to, as a list too, so a + // caller's result does not change shape when a document grows past the + // budget: `$.nested`'s member values from `json_tree_subtree`, whose + // rows are the head's descendants (its key has no gates), and the list + // itself from the gated walk to `$.nested.list`, whose rows are the + // node the gates reach. The key says which. + let sorted = |mut values: Vec| { + values.sort_by_key(ToString::to_string); + values + }; + let open_list = |rows: Vec, key: &JsonSubtreeKey| -> Vec { + wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(key, rows), column()), + ) + .unwrap() + }; + let walked_members = walked(&mut db, 0); + assert_eq!(walked_members.len(), 6); + let fallen_back = walked(&mut db, 5); + assert_eq!(fallen_back.len(), 1); + let members = sorted(vec![json!("constant"), json!([{"x": 1}, {"x": 2}])]); + assert_eq!(sorted(open_list(walked_members, &nested)), members); + assert_eq!(sorted(open_list(fallen_back, &nested)), members); let list = subtree_key("$.nested.list"); let list_rows = |db: &mut postgres::Client, budget: i32| -> Vec { db.query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree(doc, $1::text, $2::int) e WHERE id = 1", - &[&list.token(), &budget], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], $3::int) e WHERE id = 1", + &[&list.token(), &list.gates(), &budget], ) .unwrap() .iter() .map(entry) .collect() }; - let elements = |rows: Vec| -> Vec { - wait( - &runtime, - cipher.decrypt_as( - JsonTreeSubtree::new(rows).below(JsonPath::parse("$.nested.list").unwrap()), - column(), - ), - ) - .unwrap() - }; let walked_list = list_rows(&mut db, 0); - assert_eq!(walked_list.len(), 4); + assert_eq!( + walked_list.len(), + 5, + "the list, its two elements and their x" + ); let fallen_back = list_rows(&mut db, 3); assert_eq!(fallen_back.len(), 1); assert_eq!( - elements(walked_list), - vec![json!({"x": 1}), json!({"x": 2})] + open_list(walked_list, &list), + vec![fixture()["nested"]["list"].clone()] ); assert_eq!( - elements(fallen_back), - vec![json!({"x": 1}), json!({"x": 2})] + open_list(fallen_back, &list), + vec![fixture()["nested"]["list"].clone()] ); // The budget is gated on the header's sealed-root flag, the bit // `JsonHeader` writes: with the flags byte cleared, the same document @@ -649,13 +676,17 @@ fn postgres_extracts_walks_and_orders_by_path() { .get(0); assert_eq!(unflagged, 6, "an unsealed root is walked out"); - // A wildcard below the anchor: every element's `x`, each a complete - // subtree, from the list's token gated on the whole path's selector. + // A path below the head walks from the head's token through one gate + // per step: `$.nested.list[*].x` is the list's selector, then every + // element's (the wildcard's), then x's; each `x` comes back as a + // complete subtree. let xs = subtree_key("$.nested.list[*].x"); + assert_eq!(xs.token(), nested.token(), "anchored at the head"); + assert_eq!(xs.gates().len(), 3); let walked: Vec = db .query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text) e WHERE id = 1", - &[&xs.token(), &xs.selector().unwrap().to_string()], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[]) e WHERE id = 1", + &[&xs.token(), &xs.gates()], ) .unwrap() .iter() @@ -663,7 +694,7 @@ fn postgres_extracts_walks_and_orders_by_path() { .collect(); let values: Vec = wait( &runtime, - cipher.decrypt_as(JsonTreeSubtree::new(walked), column()), + cipher.decrypt_as(JsonTreeSubtree::from_key(&xs, walked), column()), ) .unwrap(); assert_eq!(values, vec![json!(1), json!(2)]); @@ -671,8 +702,8 @@ fn postgres_extracts_walks_and_orders_by_path() { // wildcard path selects every subtree over the whole document. let root: Vec = db .query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text, 1) e WHERE id = 1", - &[&xs.token(), &xs.selector().unwrap().to_string()], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 1) e WHERE id = 1", + &[&xs.token(), &xs.gates()], ) .unwrap() .iter() @@ -682,20 +713,42 @@ fn postgres_extracts_walks_and_orders_by_path() { assert_eq!(root[0].d, Some(0)); let values: Vec = wait( &runtime, - cipher.decrypt_as( - JsonTreeSubtree::new(root).at(JsonPath::parse("$.nested.list[*].x").unwrap()), - column(), - ), + cipher.decrypt_as(JsonTreeSubtree::from_key(&xs, root), column()), ) .unwrap(); assert_eq!(values, vec![json!(1), json!(2)]); - // The array functions, by the list's token. + // A positional path below a key step: the second element's subtree, + // through the list's selector and the position `#1`, with the selector + // every element of the list carries. + let second = subtree_key("$.nested.list[1]"); + assert_eq!( + second.gates()[1], + format!("#1:{}", selector("$.nested.list[*]")) + ); + let walked: Vec = db + .query( + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[]) e WHERE id = 1", + &[&second.token(), &second.gates()], + ) + .unwrap() + .iter() + .map(entry) + .collect(); + let value: Value = wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(&second, walked), column()), + ) + .unwrap(); + assert_eq!(value, json!({"x": 2})); + + // The array functions, by the head's token and the list's gate. let list = subtree_key("$.nested.list"); + assert_eq!(list.gates().len(), 1); let counts: Vec<(i32, i32)> = db .query( - "SELECT id, eql_v4.json_tree_child_count(doc, $1::text) FROM json_tree_rows ORDER BY id", - &[&list.token()], + "SELECT id, eql_v4.json_tree_child_count(doc, $1::text, $2::text[]) FROM json_tree_rows ORDER BY id", + &[&list.token(), &list.gates()], ) .unwrap() .iter() @@ -704,8 +757,8 @@ fn postgres_extracts_walks_and_orders_by_path() { assert_eq!(counts, [(1, 2), (2, 0), (3, 0)]); let elements: Vec = db .query( - "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_children(doc, $1::text) e WHERE id = 1", - &[&list.token()], + "SELECT e::jsonb::text FROM json_tree_rows, eql_v4.json_tree_children(doc, $1::text, $2::text[]) e WHERE id = 1", + &[&list.token(), &list.gates()], ) .unwrap() .iter() @@ -713,9 +766,18 @@ fn postgres_extracts_walks_and_orders_by_path() { .collect(); assert_eq!(elements.len(), 2); assert_eq!( - elements.iter().map(|e| e.o).collect::>(), - [Some(0), Some(1)] + elements.iter().map(|e| (e.j, e.d)).collect::>(), + [(Some(5), Some(1)), (Some(7), Some(1))] ); + // The head's own children, with no gate: deep and list. + let members: i32 = db + .query_one( + "SELECT eql_v4.json_tree_child_count(doc, $1::text) FROM json_tree_rows WHERE id = 1", + &[&nested.token()], + ) + .unwrap() + .get(0); + assert_eq!(members, 2); // Orderings at a path: the rows whose `$.number` is in range, and // sorted by the leaf's term. @@ -818,103 +880,167 @@ fn postgres_extracts_walks_and_orders_by_path() { } } +/// Needs an EQL v4 installation. A position names an array element and +/// nothing else, in every row of the column, as a strict SQL/JSON path does +/// (`jsonb_path_query_first(plain, 'strict $.a[1]')`): where a +/// row's node at the path is an object, an object member ranked at that +/// ordinal is not returned, whether the client derived the position's +/// token (`$.a[1]`, directly after the head) or the walk took it as a gate +/// (`$.h.a[0]`, below a key step). Each walk's rows open to the plaintext's +/// value at the strict path, or come back empty where it has none. (`#>` +/// would not do as the oracle: it reads `1` as a key of an object.) +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_a_position_reaches_array_elements_only() { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = wait(&runtime, common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = connect(); + db.batch_execute( + "CREATE TEMP TABLE json_tree_positions (id int PRIMARY KEY, plain jsonb NOT NULL, doc public.eql_v4_json_tree NOT NULL)", + ) + .unwrap(); + let rows = [ + json!({"a": [{"x": 1}, {"x": 2}], "h": {"a": [{"x": 3}, {"x": 4}]}}), + json!({"a": {"k": {"x": 9}, "m": {"x": 6}}, "h": {"a": {"k": {"x": 8}, "m": {"x": 5}}}}), + json!({"a": {"0": {"x": 7}, "1": {"x": 7}}, "h": {"a": {"0": {"x": 7}}}}), + ]; + for (id, plain) in (1..).zip(&rows) { + let stored = wait(&runtime, seal(&cipher, plain)); + let stored = serde_json::to_string(&stored).unwrap(); + db.execute( + "INSERT INTO json_tree_positions VALUES ($1, $2::text::jsonb, $3::text::jsonb::public.eql_v4_json_tree)", + &[&id, &plain.to_string(), &stored], + ) + .unwrap(); + } + for path in ["$.a[1]", "$.h.a[0]"] { + let key = wait( + &runtime, + keyset.json_subtree_key(&JsonPath::parse(path).unwrap(), &options(), column()), + ) + .unwrap(); + for id in 1..=3 { + let walked: Vec = db + .query( + "SELECT e::jsonb::text FROM json_tree_positions, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 0) e WHERE id = $3", + &[&key.token(), &key.gates(), &id], + ) + .unwrap() + .iter() + .map(|r| serde_json::from_str(&r.get::<_, String>(0)).unwrap()) + .collect(); + let expected: Option = db + .query_one( + "SELECT jsonb_path_query_first(plain, ('strict ' || $1::text)::jsonpath, '{}', true)::text FROM json_tree_positions WHERE id = $2", + &[&path, &id], + ) + .unwrap() + .get(0); + match expected { + None => assert!(walked.is_empty(), "{path}, row {id}: jsonb has no node"), + Some(expected) => { + let value: Value = wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(&key, walked), column()), + ) + .unwrap(); + assert_eq!( + value, + serde_json::from_str::(&expected).unwrap(), + "{path}, row {id}" + ); + } + } + } + } + // A position gate that does not name the elements' selector admits + // nothing: an ordinal alone cannot tell an element from a member. + let key = wait( + &runtime, + keyset.json_subtree_key(&JsonPath::parse("$.h.a[0]").unwrap(), &options(), column()), + ) + .unwrap(); + let bare = [key.gates()[0].clone(), "#0".to_owned()]; + let none: i64 = db + .query_one( + "SELECT count(*) FROM json_tree_positions, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 0) e", + &[&key.token(), &bare.as_slice()], + ) + .unwrap() + .get(0); + assert_eq!(none, 0); + let counts: Vec<(i32, i32)> = db + .query( + "SELECT id, eql_v4.json_tree_child_count(doc, $1::text, $2::text[]) FROM json_tree_positions ORDER BY id", + &[&key.token(), &key.gates()], + ) + .unwrap() + .iter() + .map(|r| (r.get(0), r.get(1))) + .collect(); + assert_eq!( + counts, + [(1, 1), (2, 0), (3, 0)], + "one member, x, under $.h.a[0] in row 1 only" + ); +} + /// Needs an EQL v4 installation. A stored document is untrusted input: its -/// chain fields are not authenticated, so whoever holds a token can write -/// children whose tokens loop back to their parent's, or an ordinal far -/// past the document's size. Every walk still ends, in time bounded by the -/// document's size, under any budget; `statement_timeout` turns a walk -/// that would not end into a failure instead of a hung suite. +/// parent tags are not authenticated, so whoever holds a token can write a +/// chain of them deeper than any sealed document. A child's token is a +/// hash of its parent's, so the chain cannot loop back, and the walk, which +/// carries no depth bound, still ends after one row per entry: here a chain +/// of 300, past the 127 levels a sealed document can nest. +/// `statement_timeout` turns a walk that would not end into a failure +/// instead of a hung suite. #[test] #[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] fn postgres_walks_end_on_hostile_documents() { let mut db = connect(); db.batch_execute("SET statement_timeout = '10s'").unwrap(); - // A root and two children at ordinals 0 and 1 under the all-zero - // token, each wrapping that same token as its own: unbounded, the walk - // doubles at every one of its 127 levels. `flags` is the header's flags - // byte, so the same document is built with its root sealed and not. - let looping = |flags: i32| { + // A root, then 300 entries each the only child of the one before, under + // the all-zero token. `flags` is the header's flags byte, so the same + // document is built with its root sealed and not. + let chain = |flags: i32| { format!( - "WITH k AS ( + "WITH RECURSIVE k AS ( SELECT set_byte(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 1, {flags}) AS h, - decode(repeat('00', 16), 'hex') AS iv, - decode(repeat('00', 16), 'hex') AS t - ), e AS ( - SELECT o, c, - encode(eql_v4_internal.json_tree_h16('p'::bytea || k.iv || k.t || eql_v4_internal.json_tree_le32(o)), 'hex') AS p, - encode(eql_v4_internal.json_tree_xor16(k.t, eql_v4_internal.json_tree_h16('w'::bytea || k.iv || k.t || decode(c, 'base64'))), 'hex') AS w - FROM k, (VALUES (0, 'AA=='), (1, 'AQ==')) AS v(o, c) + decode(repeat('00', 16), 'hex') AS iv + ), link(n, t) AS ( + SELECT 0, decode(repeat('00', 16), 'hex') + UNION ALL + SELECT n + 1, eql_v4_internal.json_tree_child_token(t, decode(repeat('00', 16), 'hex'), 0) FROM link WHERE n < 299 ) SELECT (jsonb_build_object( 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), 'h', (SELECT encode(h, 'base64') FROM k), - 'sv', jsonb_build_array(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==')) - || (SELECT jsonb_agg(jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', c, 'o', o, 'p', p, 'w', w) ORDER BY o) FROM e), + 'sv', jsonb_build_array(jsonb_build_object('s', 'AAAAAAAAAAAAAAAAAAAAAA==', 'c', '00')) + || (SELECT jsonb_agg(jsonb_build_object('s', 'AAAAAAAAAAAAAAAAAAAAAA==', 'c', '00', 'p', eql_v4_internal.json_tree_parent_tag(k.iv, link.t, 0)) ORDER BY n) FROM link, k), 'q', '{{\"sv\": [], \"el\": []}}'::jsonb))::public.eql_v4_json_tree" ) }; - let token = "00".repeat(16); - for flags in [0, 1] { - let doc = looping(flags); - let count = |db: &mut postgres::Client, walk: &str| -> i64 { - db.query_one( - &format!("SELECT count(*) FROM ({doc}) AS d(doc), {walk} e"), + let token = "AAAAAAAAAAAAAAAAAAAAAA==".to_owned(); + for (flags, budget, expected) in [(0, 0, 300), (0, 64, 300), (1, 0, 300), (1, 64, 1)] { + let doc = chain(flags); + let walked: i64 = db + .query_one( + &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, {budget}) e"), &[&token], ) - .unwrap_or_else(|error| panic!("{walk} (flags {flags}): {error}")) - .get(0) - }; - assert_eq!( - count(&mut db, "eql_v4.json_tree_children(d.doc, $1::text)"), - 2 - ); - for budget in [0, 64] { - let walked = count( - &mut db, - &format!("eql_v4.json_tree_subtree(d.doc, $1::text, {budget})"), - ); - assert!( - walked <= 3, - "a walk returns at most as many rows as the document has entries: {walked} (flags {flags}, budget {budget})" - ); - // No entry carries the selector, so nothing is emitted and the - // budget never trips: only the walk's own bound ends it. - assert_eq!( - count( - &mut db, - &format!("eql_v4.json_tree_subtree_at(d.doc, $1::text, 'ff', {budget})"), - ), - 0 - ); - } + .unwrap_or_else(|error| panic!("flags {flags}, budget {budget}: {error}")) + .get(0); + assert_eq!(walked, expected, "flags {flags}, budget {budget}"); + let children: i64 = db + .query_one( + &format!("SELECT count(*) FROM ({doc}) AS d(doc), eql_v4.json_tree_children(d.doc, $1::text) e"), + &[&token], + ) + .unwrap() + .get(0); + assert_eq!(children, 1); } - - // One entry claiming the largest ordinal an integer holds: the fan-out - // is capped at the document's entry count, so the probe is two tags, - // not two billion. - let wide = "SELECT (jsonb_build_object( - 'v', 4, 'i', jsonb_build_object('t', 'users', 'c', 'profile'), - 'h', encode(set_byte(decode(repeat('00', 40), 'hex'), 0, 1), 'base64'), - 'sv', jsonb_build_array( - jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA=='), - jsonb_build_object('s', 'aa', 'vs', 'bb', 'c', 'AA==', 'o', 2147483646, 'p', '00', 'w', '00')), - 'q', '{\"sv\": [], \"el\": []}'::jsonb))::public.eql_v4_json_tree"; - let children: i32 = db - .query_one( - &format!( - "SELECT eql_v4.json_tree_child_count(d.doc, $1::text) FROM ({wide}) AS d(doc)" - ), - &[&token], - ) - .unwrap() - .get(0); - assert_eq!(children, 0); - let walked: i64 = db - .query_one( - &format!("SELECT count(*) FROM ({wide}) AS d(doc), eql_v4.json_tree_subtree(d.doc, $1::text, 0) e"), - &[&token], - ) - .unwrap() - .get(0); - assert_eq!(walked, 0); } diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index 173780f49..298e1e6a0 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -12,30 +12,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **A JSON index, `target::Json`** (`docs/plans/2026-10-17-json-tree-index.md`, ADR-0009). An `Index` whose stored form, `json::JsonDocument`, is one entry per node under one data key per - document: a path selector and a value selector, the node's own - ciphertext, a token chain the server walks from one client-derived - token, group tags that make containment agree with `jsonb @>` across - array elements, and an ordering term on number and string leaves. It is - what EQL v4's `json_tree` family stores. `target::json_document` seals - one, `JsonDocument` is an `EncryptFrom` / `DecryptInto` - target, and it answers `op::Contains`, `op::ContainedBy`, - `op::AtPath(op::Eq)` (a scalar at a path) and the orderings at a path; - `KeysetCipher::json_selector` and `json_subtree_key` derive the - extraction keys, and `JsonNode` and `JsonSubtree` open what the server - returns. `JsonOptions::sealed_root`, on by default, makes the root entry - the whole document so no token for the root exists. A document is at - most 127 levels deep. New `TermError` variants: `JsonPath`, - `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, `JsonLeafNotOrdered` and - `JsonValueNotScalar`. A data plan reads and writes the index (`"json"`, - or `{"json": {"sealed_root": bool}}`) and refuses to run it with - `Reason::JsonIndexPending`. A `JsonDocument` writes its own containment - projection to the wire as `q`, beside `sv`, for the server to compare - with a needle, and ignores a stored one, deriving it from the entries - again. A walk past its budget returns the sealed root in place of the - subtree: `JsonSubtree::below(path)` (for `json_tree_subtree`) or - `JsonSubtree::at(path)` (for `json_tree_subtree_at`) attaches the walked - path, and the rows then open to the same value either way, over - `JsonPath::select`, which evaluates a path over a value. + document: a path selector, the node's own ciphertext, a chain of tokens + derived by ordinal that the server walks from one client-derived token, + a stored containment projection that agrees with `jsonb @>` across array + elements, and, at the paths `JsonOptions::ordered` names, an ordering + term on number and string leaves. It is what EQL v4's `json_tree` family + stores. `target::json_document` seals one, `JsonDocument` is an + `EncryptFrom` / `DecryptInto` target, and it answers + `op::Contains`, `op::ContainedBy`, `op::AtPath(op::Eq)` (a scalar at a + path) and the orderings at an ordered path; `KeysetCipher::json_selector` + and `json_subtree_key` derive the extraction keys, and `JsonNode` and + `JsonSubtree` open what the server returns. `JsonOptions::sealed_root`, + on by default, makes the root entry the whole document so no token for + the root exists. A document is at most 127 levels deep. New `TermError` + variants: `JsonPath`, `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, + `JsonLeafNotOrdered`, `JsonPathNotOrdered`, `JsonOrderingAtPosition` + and `JsonValueNotScalar`. `JsonOptions::ordering([..])` names the + ordered paths, `JsonPath::covers` is the rule that matches them, and a + path naming an array position is refused there and in an ordering + query, since the comparison extracts by selector and reads a position as + the wildcard. A data plan reads and writes the index (`"json"`, or + `{"json": {"sealed_root": bool, "ordered": [path, …]}}`) and refuses to + run it with `Reason::JsonIndexPending`. A subtree key is a token and + `JsonSubtreeKey::gates()`, one gate per path step below it, a selector + or `#n:` for an array position. A walk past its budget returns + the sealed root in place of the subtree. The key carries the path it + was derived from (`JsonSubtreeKey::path`) and builds the rows' wrapper, + `key.open(rows) -> JsonSubtree`, the wrapper's only constructor; from + its gates it knows whether the rows are below the path's node + (`json_tree_subtree`) or at the nodes it reaches (`json_tree_subtree_at`), + so the rows open to the same value either way, over `JsonPath::select`, + which evaluates a path over a value. On the wire every field the SQL reads (`s`, the pairs + of `q`, `p`, `op`, `h`, tokens and gates) is padded standard base64, and + the ciphertext `c` is RFC 1924 base85 (`base85-simd`). The document's + plaintext is held under `Protected` throughout, as the scalar paths hold + theirs: each node's plaintext is encoded straight from the caller's value + into a `Protected>` for the cipher, and what an entry opens to + (a key, a scalar, a sealed root's document) stays wrapped until it is + handed to the caller, so the sealed-root fallback's whole document is + wiped once the walked path is selected from it. + +- **A layout chosen by value, `target::Layout`, and + `Plan::value().encrypt_as(layout)`.** `EncryptFrom` is a type's own + layout, fixed where the type is written; a `Layout` is a value with + options that seals the same target under them and names the index a + query answers through. `Json::new(options)` is one. `encrypt_as` builds + a `ValuePlan>` that seals under the layout and + resolves every query through its index, so a query does not name the + options again; `ValuePlan::specs` and `ValuePlan::layout` report them. + `Error` gains `From`. - **A block ORE index, `OreBlock`** (ADR-0002 and ADR-0009, amended 2026-10-19). It derives ore-rs 0.9.0's 6-bit block ORE under its own key diff --git a/packages/stack-encrypt/Cargo.toml b/packages/stack-encrypt/Cargo.toml index be631aedb..065609938 100644 --- a/packages/stack-encrypt/Cargo.toml +++ b/packages/stack-encrypt/Cargo.toml @@ -47,8 +47,14 @@ vitaminc-protected = { workspace = true } vitaminc-aead-frame = { workspace = true, optional = true, features = ["serde"] } serde = { workspace = true } # `Descriptor`: the base64 rendering of a non-textual context for ZeroKMS, -# and the JSON index's ciphertext and header fields on the wire. +# and the JSON index's chain fields, header and ordering term on the wire: +# padded standard base64, the one form PostgreSQL's `decode(x, 'base64')` +# reads and `encode(x, 'base64')` writes. base64ct = { version = "1.7", features = ["alloc"] } +# The JSON index's ciphertext field on the wire, which nothing in SQL +# decodes: RFC 1924 base85, five characters per four bytes. SIMD on +# aarch64 and x86_64, portable scalar code elsewhere (wasm32 included). +base85-simd = "0.1.0" # The JSON index's token chain: SHA-256 is what links a node to its # children, because core PostgreSQL computes `sha256(bytea)` and can walk # the chain server-side (`json` module). The same version vitaminc-hmac diff --git a/packages/stack-encrypt/fuzz/Cargo.lock b/packages/stack-encrypt/fuzz/Cargo.lock index d942dea07..052083929 100644 --- a/packages/stack-encrypt/fuzz/Cargo.lock +++ b/packages/stack-encrypt/fuzz/Cargo.lock @@ -177,6 +177,12 @@ version = "1.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" +[[package]] +name = "base85-simd" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbd16ef96137ca89cc38b0acb5bee36fada9707c044bc9c753f55c816bfa719" + [[package]] name = "bitflags" version = "2.10.0" @@ -1487,7 +1493,7 @@ dependencies = [ "num", "orderable-bytes", "rand 0.8.6", - "rand_chacha 0.3.1", + "rand_chacha", "subtle-ng", "thiserror 1.0.69", "vitaminc-hmac", @@ -2079,6 +2085,7 @@ name = "stack-encrypt" version = "0.3.0" dependencies = [ "base64ct", + "base85-simd", "cllw-ore", "miette", "ore-rs", @@ -2115,6 +2122,7 @@ version = "0.0.0" dependencies = [ "arbitrary", "libfuzzer-sys", + "serde_json", "stack-encrypt", "uuid", "vitaminc-aead 0.6.1", diff --git a/packages/stack-encrypt/fuzz/Cargo.toml b/packages/stack-encrypt/fuzz/Cargo.toml index 90a6eaebc..223338291 100644 --- a/packages/stack-encrypt/fuzz/Cargo.toml +++ b/packages/stack-encrypt/fuzz/Cargo.toml @@ -26,6 +26,8 @@ libfuzzer-sys = "0.4" arbitrary = { version = "1", features = ["derive"] } # The fixed leaf the record harness fills its sealed nodes with. uuid = "1.8" +# The JSON index's wire form is JSON: the `json_wire_decode` harness reads it. +serde_json = "1" [dependencies.stack-encrypt] path = ".." @@ -63,5 +65,12 @@ test = false doc = false bench = false +[[bin]] +name = "json_wire_decode" +path = "fuzz_targets/json_wire_decode.rs" +test = false +doc = false +bench = false + [workspace] resolver = "2" diff --git a/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-open-root b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-open-root new file mode 100644 index 000000000..7f6fdbc5a --- /dev/null +++ b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-open-root @@ -0,0 +1 @@ +{"h":"AQAAAAAAAAAAAAAAAAAAAAAAAAAAANIkbh1SlApdiok/mH5A0y0gAOQA5EoyZTbrzBqVUIdV4xg/x1IurKpjtYcdkC3075ES","sv":[{"s":"GUG6+MOjeHGk8vJS5CJOxA==","c":"0z_z&4uZ@=$H@nfJ;TLc2!WvRB^)e4)~BCS@KxODY~VN"},{"s":"ExJSKX+WLXlRo47xPs+Pjw==","c":"0-Hyc){m-585eT>A%Ic${%f*gb&pQecVdC5C2t5g>oDx}hfnAi#J5%","p":"E2rwHzdyogU="},{"s":"ShAx/UGS67/ex/VpZHT5YA==","c":"0=HCNaNC0;x_5~{kj59-YDV^ypfcBUOTx8%H-x$IaWhH==A5ix","p":"RQv+PlaQGuI="},{"s":"YZINpBKKE2Pe4VuD40e6pQ==","c":"0vIUgnZMq7p)P+V%6nYK5j|htqqfK&yaub>9$on1oIAs7`y%6)7AB!D","p":"bBWX/L/rx/E="},{"s":"YZINpBKKE2Pe4VuD40e6pQ==","c":"0PC$Ka-Q6uX)YZMONS)XDKjmy~5JVD#o8)FslVOm8","p":"geMZYPBpVSU="},{"s":"ExJSKX+WLXlRo47xPs+Pjw==","c":"0@^HM$R|__1yG(mL4%}WLIoAol$|)kXNI=t9AAC","p":"nAFCVRKssP4="},{"s":"ExJSKX+WLXlRo47xPs+Pjw==","c":"0)zmeW|-=APNVqgRac|F_*IiJiWhC8bJ!7eQ0C`p_2t4_2mB3%w7sZ9","p":"rZKzWt2q2JA="}],"q":{"sv":[{"s":"GUG6+MOjeHGk8vJS5CJOxA==","vs":"zAI3S2R1ohpJhfgWmnRmxg=="},{"s":"ExJSKX+WLXlRo47xPs+Pjw==","vs":"eGqsMOSc2W0Bx4C7O8SneQ=="},{"s":"ExJSKX+WLXlRo47xPs+Pjw==","vs":"MvrShvASjRgE5vrU2PBgoA=="}],"el":[{"sv":[{"s":"ExJSKX+WLXlRo47xPs+Pjw==","vs":"O1k8IHoig1PAQJ3gPeclfQ=="},{"s":"ShAx/UGS67/ex/VpZHT5YA==","vs":"hUbQp5DbczWtT2O/9ubZtA=="},{"s":"YZINpBKKE2Pe4VuD40e6pQ==","vs":"Lh/z3wdOvqoTt0lQUcDDjw=="},{"s":"YZINpBKKE2Pe4VuD40e6pQ==","vs":"aGBns3vFOnXOivJuSRJDXQ=="}],"el":[]}]}} \ No newline at end of file diff --git a/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-sealed-root b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-sealed-root new file mode 100644 index 000000000..71937f12c --- /dev/null +++ b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-document-sealed-root @@ -0,0 +1 @@ +{"h":"AQEAAAAAAAAAAAAAAAAAAAAAAAAAAModWByqFwzHFmXW6zBOgBIgALdJJWr10FWvJW0c+enHwFzJNgKih/2xZckLO87P5bPA","sv":[{"s":"GUG6+MOjeHGk8vJS5CJOxA==","c":"0t1}o=XqADD1eMsxs4DtiFO4)PT&RZ!28}Hi;-w&S1-hb4@^;EEm$(Toj4284Jqu=>ag&T4>|x$U)_I?2^3vyd#JRP+T_3`zeL)SU#bj#f23ew$<;Z?-DqE)nv=R"},{"s":"Tm5hoSb1R2y2K1ileozKQQ==","c":"0$d}`UG9!!2zc|70dbbXQT*l>3XCn|avEj^q13yc$n~FYb6~5qH}s~Nu-QViqt(#"},{"s":"lihJXUx9+YPMvJXQ97UpgA==","c":"0zBNN!Rd^LxAecmwyi?Z8on|q_JlQjGl^PqrDNSannTJ7b}&!hH=h1RwE"},{"s":"mwxTGtOaM0Q8dwFeVMXzBA==","c":"0%?9A501BMDh5#JJ&PCWXKCCTdaSCMvk2kXf+}{^^L>^llxYVWG7(BPYkJSk90s@","p":"t980GLGUA+w="},{"s":"9nWXlzomXDcM6TjK3prsEQ==","c":"0+jHnFB2n_4P!_S^`TcBs)r{(Hr3>p?pqz&cZ8{nlHZA=e;o*DM-Y+","p":"T1Z1ZgBOtes="},{"s":"x6fT5bkR9wVtuRnupRk3lg==","c":"0$Xju@DoU4*%E4&sMnf`M&e0()eH`sd#7NAG4;zEU5D0sX~T=>y#6%","p":"a7mxfLHzS9k="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","c":"0$r1?vQ(9$?($NHXdP^l`0eN!qO;;zMU4Ar$so8V;UrYt1`{M0%>cN9N3SSFSS)1D","p":"xHMEagMG00I="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","c":"0uPz!h(9!yqN>{O#@uXGMJb<0xKyaqrdk1{&tku@_rE?}HohCNVg","p":"eriAuFTQ3sk="},{"s":"guOaofIleFSaqgIbtgP9BA==","c":"0>bADC+pE&wsaPW)#?6&TQzE}Ce;ANg8^g`Bhgkpxv0rM%d&Cz@!7Qt(Z~","op":"AXCYcmjGlPA9o88wtKHqszfNGZu5WS/7jkkQP0S/ijn14DbmuAaN9OyLuYmFzBvH/sbekttZj1Rq+SleI2caoiM="}],"q":{"sv":[{"s":"GUG6+MOjeHGk8vJS5CJOxA==","vs":"MUinEHjmPaxhTO0p+J6gNw=="},{"s":"Tm5hoSb1R2y2K1ileozKQQ==","vs":"1nnluVkGLHmNyEBbXNpLFw=="},{"s":"lihJXUx9+YPMvJXQ97UpgA==","vs":"o2ktqHlsWUKHWNjBs+cWBQ=="},{"s":"mwxTGtOaM0Q8dwFeVMXzBA==","vs":"0k1uvXj0q1y00BoSeO5uZA=="},{"s":"9nWXlzomXDcM6TjK3prsEQ==","vs":"LUR7qeHmg+k3cJgbmnUJnQ=="},{"s":"guOaofIleFSaqgIbtgP9BA==","vs":"KJQFi9ZscpCAnh7U2HOgLQ=="}],"el":[{"sv":[{"s":"x6fT5bkR9wVtuRnupRk3lg==","vs":"vzVi6dl1WYdpruf8GpBrVg=="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","vs":"baRZIwaJ+CSTK/GflygMAg=="}],"el":[]},{"sv":[{"s":"x6fT5bkR9wVtuRnupRk3lg==","vs":"vzVi6dl1WYdpruf8GpBrVg=="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","vs":"wYWQ5BRfSrEVO+HQeX/bxQ=="}],"el":[]}]}} \ No newline at end of file diff --git a/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-header b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-header new file mode 100644 index 000000000..dafb181a6 Binary files /dev/null and b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-header differ diff --git a/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-node b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-node new file mode 100644 index 000000000..e455632da --- /dev/null +++ b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-node @@ -0,0 +1 @@ +{"s":"guOaofIleFSaqgIbtgP9BA==","c":"0>bADC+pE&wsaPW)#?6&TQzE}Ce;ANg8^g`Bhgkpxv0rM%d&Cz@!7Qt(Z~","op":"AXCYcmjGlPA9o88wtKHqszfNGZu5WS/7jkkQP0S/ijn14DbmuAaN9OyLuYmFzBvH/sbekttZj1Rq+SleI2caoiM=","h":"AQEAAAAAAAAAAAAAAAAAAAAAAAAAAModWByqFwzHFmXW6zBOgBIgALdJJWr10FWvJW0c+enHwFzJNgKih/2xZckLO87P5bPA","j":9} \ No newline at end of file diff --git a/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-projection b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-projection new file mode 100644 index 000000000..dc5843aab --- /dev/null +++ b/packages/stack-encrypt/fuzz/corpus/json_wire_decode/valid-projection @@ -0,0 +1 @@ +{"sv":[{"s":"GUG6+MOjeHGk8vJS5CJOxA==","vs":"MUinEHjmPaxhTO0p+J6gNw=="},{"s":"Tm5hoSb1R2y2K1ileozKQQ==","vs":"1nnluVkGLHmNyEBbXNpLFw=="},{"s":"lihJXUx9+YPMvJXQ97UpgA==","vs":"o2ktqHlsWUKHWNjBs+cWBQ=="},{"s":"mwxTGtOaM0Q8dwFeVMXzBA==","vs":"0k1uvXj0q1y00BoSeO5uZA=="},{"s":"9nWXlzomXDcM6TjK3prsEQ==","vs":"LUR7qeHmg+k3cJgbmnUJnQ=="},{"s":"guOaofIleFSaqgIbtgP9BA==","vs":"KJQFi9ZscpCAnh7U2HOgLQ=="}],"el":[{"sv":[{"s":"x6fT5bkR9wVtuRnupRk3lg==","vs":"vzVi6dl1WYdpruf8GpBrVg=="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","vs":"baRZIwaJ+CSTK/GflygMAg=="}],"el":[]},{"sv":[{"s":"x6fT5bkR9wVtuRnupRk3lg==","vs":"vzVi6dl1WYdpruf8GpBrVg=="},{"s":"2LHfpdQzfmw63dqQ9eP0Ww==","vs":"wYWQ5BRfSrEVO+HQeX/bxQ=="}],"el":[]}]} \ No newline at end of file diff --git a/packages/stack-encrypt/fuzz/fuzz_targets/json_wire_decode.rs b/packages/stack-encrypt/fuzz/fuzz_targets/json_wire_decode.rs new file mode 100644 index 000000000..eb5c0ea67 --- /dev/null +++ b/packages/stack-encrypt/fuzz/fuzz_targets/json_wire_decode.rs @@ -0,0 +1,34 @@ +#![no_main] + +use libfuzzer_sys::fuzz_target; +use stack_encrypt::json::{JsonDocument, JsonHeader, JsonLeafQuery, JsonNode, JsonProjection}; + +// Fuzz the JSON index's wire decoders. A stored `json_tree` document, and +// every entry `->` or a walk returns, come back from the database through +// these before anything is authenticated, so the text is attacker-controlled: +// the base64 of the selectors, parent tags, ordering terms and header, the +// base85 of the ciphertexts (decoded with SIMD code on aarch64 and x86_64), +// and the header's frozen byte layout. Decoding must never panic, only +// return `Err`. +// +// An accepted value must also survive a round trip through its own wire +// form unchanged: a decoder that accepts text its encoder cannot reproduce +// would let two stored forms alias one document. +fuzz_target!(|bytes: &[u8]| { + let _ = JsonHeader::from_bytes(bytes); + let Ok(text) = std::str::from_utf8(bytes) else { + return; + }; + if let Ok(doc) = serde_json::from_str::(text) { + let wire = serde_json::to_value(&doc).expect("a document serializes"); + let back: JsonDocument = serde_json::from_value(wire).expect("its wire form reads back"); + assert_eq!(back, doc, "JsonDocument wire form is not lossless"); + } + if let Ok(node) = serde_json::from_str::(text) { + let wire = serde_json::to_value(&node).expect("a node serializes"); + let back: JsonNode = serde_json::from_value(wire).expect("its wire form reads back"); + assert_eq!(back, node, "JsonNode wire form is not lossless"); + } + let _ = serde_json::from_str::(text); + let _ = serde_json::from_str::(text); +}); diff --git a/packages/stack-encrypt/src/cipher.rs b/packages/stack-encrypt/src/cipher.rs index 092dcf737..62c79fa50 100644 --- a/packages/stack-encrypt/src/cipher.rs +++ b/packages/stack-encrypt/src/cipher.rs @@ -390,6 +390,15 @@ impl From for Error { } } +/// A text that is not a JSON path is a term error, as it is when an +/// accessor's path is parsed: a caller building a JSON index's options +/// (`JsonOptions::ordering`) can `?` it. +impl From for Error { + fn from(error: crate::json::JsonPathError) -> Self { + Error::Term(error.into()) + } +} + /// The CipherStash cipher, scoped to one client: a ZeroKMS client (a /// [`DataKeySource`] — production: [`kms::StackKms`](crate::kms::StackKms); tests: /// `stack_encrypt::kms::FakeDataKeySource`) and the keysets that client uses. diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index 491581033..970182d9e 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -97,6 +97,8 @@ mod tests { TermError::JsonTooWide => TermError::JsonTooWide, TermError::JsonSealedRoot => TermError::JsonSealedRoot, TermError::JsonLeafNotOrdered => TermError::JsonLeafNotOrdered, + TermError::JsonPathNotOrdered => TermError::JsonPathNotOrdered, + TermError::JsonOrderingAtPosition => TermError::JsonOrderingAtPosition, TermError::JsonValueNotScalar => TermError::JsonValueNotScalar, ])); errors.extend(boxed(variants![ diff --git a/packages/stack-encrypt/src/dynamic/term.rs b/packages/stack-encrypt/src/dynamic/term.rs index 7df0ed7e9..dc9e1676c 100644 --- a/packages/stack-encrypt/src/dynamic/term.rs +++ b/packages/stack-encrypt/src/dynamic/term.rs @@ -77,10 +77,14 @@ use crate::{Encrypt, IntoPrfContext, KeysetCipher, NonEmpty}; /// A JSON index is `"json"`, or an object mapping `"json"` to its options: /// /// ```text -/// { "json": { "sealed_root": } } +/// { "json": { "sealed_root": , "ordered": [, …] } } /// ``` /// -/// `sealed_root` defaults to `true` ([`JsonOptions::default`]). A data plan +/// `sealed_root` defaults to `true` and `ordered` to none +/// ([`JsonOptions::default`]); each path in `ordered` is text +/// [`JsonPath::parse`](crate::json::JsonPath::parse) takes, such as +/// `"$.age"` or `"$.orders[*].total"`, and one it refuses, or one that +/// names an array position (`"$.scores[2]"`), refuses the index. A data plan /// reads and writes the JSON index and refuses to run it /// ([`Reason::JsonIndexPending`]): the engine's typed path produces a JSON /// document, and the binding's half is not written yet. The other four @@ -210,6 +214,7 @@ impl Serialize for IndexSpec { #[derive(Serialize)] struct JsonOut { sealed_root: bool, + ordered: Vec, } #[derive(Serialize)] struct MatchOut { @@ -242,6 +247,7 @@ impl Serialize for IndexSpec { IndexSpec::Json(options) if *options == JsonOptions::default() => Out::DefaultJson, IndexSpec::Json(options) => Out::Json(JsonOut { sealed_root: options.sealed_root, + ordered: options.ordered.iter().map(ToString::to_string).collect(), }), } .serialize(s) @@ -279,29 +285,36 @@ impl From for IndexSpec { } /// A JSON index's options object: each option optional, defaulting to -/// [`JsonOptions::default`]'s. +/// [`JsonOptions::default`]'s, and each ordered path parsed. #[derive(Deserialize)] -#[serde(from = "JsonOptionsWire")] +#[serde(try_from = "JsonOptionsWire")] pub(super) struct JsonWire(pub(super) JsonOptions); #[derive(Deserialize)] #[serde(deny_unknown_fields, default)] struct JsonOptionsWire { sealed_root: bool, + ordered: Vec, } impl Default for JsonOptionsWire { fn default() -> Self { - let JsonOptions { sealed_root } = JsonOptions::default(); - Self { sealed_root } + Self { + sealed_root: JsonOptions::default().sealed_root, + ordered: Vec::new(), + } } } -impl From for JsonWire { - fn from(wire: JsonOptionsWire) -> Self { - JsonWire(JsonOptions { +impl TryFrom for JsonWire { + type Error = crate::sem::TermError; + fn try_from(wire: JsonOptionsWire) -> Result { + let options = JsonOptions { sealed_root: wire.sealed_root, - }) + ..JsonOptions::new() + } + .ordering(wire.ordered)?; + Ok(JsonWire(options)) } } diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index c1f82297d..db1ff502c 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -19,7 +19,10 @@ use vitaminc_prf::{BlockVisitor, IntoPrfContext, PrfValue}; use vitaminc_protected::{Controlled, Protected}; use super::path::{JsonPath, Segment}; -use super::query::{JsonSelector, JsonSubtreeKey}; +use super::plain::{ + insert_member, protected_array, protected_object, push_element, read_protected, PlainJson, +}; +use super::query::{Gate, JsonSelector, JsonSubtreeKey}; use super::{JsonDocument, JsonEntry, JsonHeader, JsonNode, JsonOptions, JsonSubtree, Tag}; use crate::sem::TermError; use crate::target::{AeadContext, CallerContext, Decryption, Pending, Request}; @@ -34,11 +37,9 @@ const VALUE_DOMAIN: &[u8] = b"stack-encrypt/json/value/v1"; const TOKEN_DOMAIN: &[u8] = b"stack-encrypt/json/token/v1"; /// PAE domain of an entry's AAD. const ENTRY_AAD_DOMAIN: &[u8] = b"stack-encrypt/json/entry/v1"; -/// PAE domain of a token step: a child's token, from its parent's token and -/// its label. Keyed, so a token does not let its holder test key names. -const STEP_DOMAIN: &[u8] = b"stack-encrypt/json/token-step/v1"; /// PAE domain of an object member's rank under its parent's token, which -/// orders the members' ordinals. Keyed for the same reason. +/// orders the members' ordinals. Keyed, so a token does not let its holder +/// test key names. const RANK_DOMAIN: &[u8] = b"stack-encrypt/json/rank/v1"; /// Deeper than this is refused rather than sealed: the walk recurses, and a @@ -124,11 +125,23 @@ impl Kind { } } -/// A node's label under its parent. -#[derive(Clone, Debug, PartialEq, Eq)] +/// An opened node's label under its parent. A key is plaintext, so it is +/// held in a `Protected` until it is moved into the value the caller asked +/// for. +#[derive(Clone, Debug)] pub(crate) enum Label { Root, - Key(String), + Key(Protected), + Index(u64), +} + +/// A node's label as the walk reads it from the caller's value, borrowed: +/// the walk encodes it straight into the node's plaintext and keeps no +/// copy. +#[derive(Clone, Copy)] +enum LabelRef<'a> { + Root, + Key(&'a str), Index(u64), } @@ -153,14 +166,6 @@ pub(crate) struct TermNode { pub(crate) container_element: bool, } -/// One node's plaintext, as its ciphertext encodes it. -struct NodePlain { - label: Label, - kind: Kind, - scalar: Option, - children: u32, -} - /// The one-way terms of a document: one term node per node, in the /// walk's order, with the root first. The term of the /// [`Json`](struct@crate::target::Json) index, and what every query term is built @@ -265,50 +270,43 @@ fn h16(parts: &[&[u8]]) -> Tag { truncate(&hasher.finalize()) } -/// A child's token: `PRF(json/token-step ‖ column, PAE(t(parent), 0x01, -/// key))` for an object member, `PAE(t(parent), 0x02, LE64(index))` for an -/// array element, the first sixteen bytes. +/// `H("t" ‖ t(parent) ‖ s(child) ‖ LE32(ordinal))`: the token of the child +/// at `ordinal` under a node, whose selector is `selector`. Column-wide like +/// every token (no document salt), so the client derives an array +/// element's token from the path alone, and the server derives every hit's +/// from its parent's and the hit's stored `s` as it walks; no token is ever +/// stored. /// -/// Keyed with the index key (the design's decision 11). An unkeyed hash of -/// the parent's token and the label would let anyone holding a token test a -/// guessed key name against every child token the walk unwraps below it, in -/// every row of the column. The server never derives a child token from a -/// label (it unwraps `w`), so only the client computes this step, and the -/// client holds the key. -fn child_token( - prf: &HmacSha256Prf, - context: &Context<'_>, - parent: &Tag, - label: &Label, -) -> Result { - let framed = match label { - Label::Key(key) => Context::pae(&[parent, &[0x01], key.as_bytes()]), - Label::Index(index) => Context::pae(&[parent, &[0x02], &index.to_le_bytes()]), - Label::Root => return Ok(*parent), - }; - prf16(prf, STEP_DOMAIN, context, framed.as_bytes()) -} - -/// `H("p" ‖ iv ‖ t(parent) ‖ LE32(ordinal))`: position-keyed, so siblings -/// are unlinkable at rest. -pub(super) fn parent_tag(iv: &[u8], parent: &Tag, ordinal: u32) -> Tag { - h16(&[b"p", iv, parent, &ordinal.to_le_bytes()]) -} - -/// `H("w" ‖ iv ‖ t(parent) ‖ c)`: the mask a child's token is wrapped under. -pub(super) fn mask(iv: &[u8], parent: &Tag, c: &[u8]) -> Tag { - h16(&[b"w", iv, parent, c]) +/// The selector is what keeps an element apart from an object member. An +/// ordinal alone does not: a member's is its rank among its siblings, and +/// rows of one column need not agree on a node's kind, so without it the +/// token of `$.a[0]` would be the token of whichever member of an object +/// `$.a` ranks first, in every row whose `$.a` is an object. An element's +/// selector is the wildcard's and a member's names its key, so the two +/// never share a token. +pub(super) fn child_token(parent: &Tag, selector: &Tag, ordinal: u32) -> Tag { + h16(&[b"t", parent, selector, &ordinal.to_le_bytes()]) } -/// `H("g" ‖ iv ‖ t(element))`: an array element's group tag. -fn group_tag(iv: &[u8], token: &Tag) -> Tag { - h16(&[b"g", iv, token]) +/// The first eight bytes of `H("p" ‖ iv ‖ t(parent) ‖ LE32(ordinal))`: +/// position-keyed, so siblings are unlinkable at rest, and salted by the +/// document, so the same position in two rows is too. Eight bytes, since a +/// parent tag only has to be distinct within its document +/// ([`ParentTag`](super::ParentTag)). +pub(super) fn parent_tag(iv: &[u8], parent: &Tag, ordinal: u32) -> super::ParentTag { + let full = h16(&[b"p", iv, parent, &ordinal.to_le_bytes()]); + let mut tag = super::ParentTag::default(); + tag.copy_from_slice(&full[..8]); + tag } /// `PRF(json/rank ‖ column, PAE(t(parent), key))`: what an object's members /// are ranked by for their ordinals, so an ordinal does not rank key names. -/// Keyed, as [`child_token`] is, so a token's holder cannot rank guessed -/// key names and compare the order with the ordinals it sees. +/// Keyed with the index key (the design's decision 11). The chain below a +/// token is public to whoever holds it (the server derives each child's +/// token from its ordinal), so an unkeyed rank would let a token's holder +/// rank guessed key names and compare the order with the selector it finds +/// at each ordinal, in every row. Only the client ranks, when it seals. fn rank( prf: &HmacSha256Prf, context: &Context<'_>, @@ -319,14 +317,6 @@ fn rank( prf16(prf, RANK_DOMAIN, context, framed.as_bytes()) } -pub(super) fn xor(a: &Tag, b: &Tag) -> Tag { - let mut out = [0; 16]; - for (i, byte) in out.iter_mut().enumerate() { - *byte = a[i] ^ b[i]; - } - out -} - /// A number's canonical bytes for its value selector: an integral value, /// whichever JSON spelling it had, is its `i128`; anything else is its /// `f64`. So `2` and `2.0` are one value, as `jsonb` compares them, and @@ -380,7 +370,8 @@ fn value_selector( /// The leaf ordering term: the OPE term of a number as an `f64` or of a /// string, under the column's context, as the scalar `Ope` index derives /// it. Nothing for any other kind. An integer wider than 53 bits orders by -/// the `f64` it rounds to. +/// the `f64` it rounds to. The walk asks for it only at a path the +/// options order. fn leaf_op( prf: &HmacSha256Prf, context: &Context<'_>, @@ -409,24 +400,28 @@ fn leaf_op( struct Walked { terms: JsonTerms, - plain: Vec, + /// Each node's plaintext as its ciphertext encodes it, in the walk's + /// order, built under `Protected` from the caller's value and handed to + /// the cipher as it is. + plain: Vec>>, } struct Walker<'p> { prf: &'p HmacSha256Prf, context: &'p Context<'p>, - options: JsonOptions, + options: &'p JsonOptions, nodes: Vec, - plain: Vec, + plain: Vec>>, } /// Flatten `value` into its terms and its nodes' plaintexts. fn walk( value: &Value, - options: JsonOptions, + options: &JsonOptions, prf: &HmacSha256Prf, context: &Context<'_>, ) -> Result { + options.check()?; let mut walker = Walker { prf, context, @@ -438,7 +433,7 @@ fn walk( value, &JsonPath::root(), &JsonPath::root(), - Label::Root, + LabelRef::Root, None, None, None, @@ -469,7 +464,7 @@ impl Walker<'_> { value: &Value, positional: &JsonPath, wild: &JsonPath, - label: Label, + label: LabelRef<'_>, parent: Option, parent_token: Option, ordinal: Option, @@ -496,11 +491,21 @@ impl Walker<'_> { &positional.token_bytes().unwrap_or_default(), )?), _ => parent_token - .map(|p| child_token(self.prf, self.context, &p, &label)) - .transpose()?, + .zip(ordinal) + .map(|(p, o)| child_token(&p, &s, o)), + }; + let op = if self.options.orders(positional) { + leaf_op(self.prf, self.context, value)? + } else { + None + }; + let container_element = matches!(label, LabelRef::Index(_)) && kind.is_container(); + let children = match value { + Value::Object(map) => map.len(), + Value::Array(items) => items.len(), + _ => 0, }; - let op = leaf_op(self.prf, self.context, value)?; - let container_element = matches!(label, Label::Index(_)) && kind.is_container(); + let children = u32::try_from(children).map_err(|_| TermError::JsonTooWide)?; let index = self.nodes.len(); self.nodes.push(TermNode { positional: positional.clone(), @@ -513,18 +518,12 @@ impl Walker<'_> { element, container_element, }); - self.plain.push(NodePlain { - label, - kind, - scalar: (!kind.is_container()).then(|| value.clone()), - children: 0, - }); + self.plain.push(encode_node(kind, children, label, value)); let child_element = if container_element { Some(index) } else { element }; - let mut count: u32 = 0; match value { Value::Object(map) => { let ordinals: Option> = token @@ -536,14 +535,13 @@ impl Walker<'_> { child, &positional.then(Segment::Key(key.clone())), &wild.then(Segment::Key(key.clone())), - Label::Key(key.clone()), + LabelRef::Key(key), Some(index), token, ordinal, child_element, depth + 1, )?; - count += 1; } } Value::Array(items) => { @@ -557,19 +555,17 @@ impl Walker<'_> { child, &positional.then(Segment::Index(i)), &wild.then(Segment::Wildcard), - Label::Index(i), + LabelRef::Index(i), Some(index), token, ordinal, child_element, depth + 1, )?; - count += 1; } } _ => {} } - self.plain[index].children = count; Ok(index) } } @@ -603,7 +599,7 @@ fn shuffled_ordinals( /// derives them: no key is requested. pub(crate) fn terms_pending<'a, K>( value: &Value, - options: JsonOptions, + options: &JsonOptions, cipher: &'a KeysetCipher<'_, K>, context: NonEmpty, ) -> Pending<'a, JsonTerms, K> { @@ -620,7 +616,7 @@ pub(crate) fn terms_pending<'a, K>( /// column and its ordinal. pub(crate) fn seal_pending<'a, K: 'static>( value: Value, - options: JsonOptions, + options: &JsonOptions, cipher: &'a KeysetCipher<'_, K>, context: NonEmpty, ) -> Pending<'a, JsonDocument, K> { @@ -636,7 +632,7 @@ pub(crate) fn seal_pending<'a, K: 'static>( Err(error) => return Pending::failed(cipher, error.into()), }; let whole = if options.sealed_root { - match serde_json::to_vec(&value) { + match sealed_root_plaintext(&value) { Ok(bytes) => Some(bytes), Err(error) => return Pending::failed(cipher, Error::Other(Box::new(error))), } @@ -644,12 +640,13 @@ pub(crate) fn seal_pending<'a, K: 'static>( None }; let keyset_id = cipher.keyset_id(); + let sealed_root = options.sealed_root; Pending::request( cipher, vec![Request::generate_under(descriptor)], move |responses| { let key = responses.next_generated_key()?; - seal(walked, whole, keyset_id, key, &aad, options) + seal(walked, whole, keyset_id, key, &aad, sealed_root) }, ) } @@ -672,38 +669,98 @@ fn leaf_cipher(key: &stack_kms::DataKey) -> Result { } /// A node's plaintext: its kind, its child count, its label and, for a -/// scalar, its value. -fn encode_node(plain: &NodePlain) -> Vec { - let mut out = vec![plain.kind.byte()]; - out.extend_from_slice(&plain.children.to_le_bytes()); - match &plain.label { - Label::Root => out.push(0x00), - Label::Key(key) => { - out.push(0x01); - out.extend_from_slice(&(key.len() as u32).to_le_bytes()); - out.extend_from_slice(key.as_bytes()); +/// scalar, its value. Built under `Protected` straight from the caller's +/// value, sized before it is written so it never reallocates and leaves a +/// copy behind, and handed to the cipher as it is. +fn encode_node( + kind: Kind, + children: u32, + label: LabelRef<'_>, + value: &Value, +) -> Protected> { + // A number's text is a new allocation: it is made wrapped too. + let number = match value { + Value::Number(n) => Some(Protected::new(n.to_string())), + _ => None, + }; + let size = encoded_len(label, value, number.as_ref()); + Protected::generate(|| { + let mut out = Vec::with_capacity(size); + out.push(kind.byte()); + out.extend_from_slice(&children.to_le_bytes()); + match label { + LabelRef::Root => out.push(0x00), + LabelRef::Key(key) => { + out.push(0x01); + out.extend_from_slice(&(key.len() as u32).to_le_bytes()); + out.extend_from_slice(key.as_bytes()); + } + LabelRef::Index(index) => { + out.push(0x02); + out.extend_from_slice(&index.to_le_bytes()); + } } - Label::Index(index) => { - out.push(0x02); - out.extend_from_slice(&index.to_le_bytes()); + match value { + Value::String(text) => out.extend_from_slice(text.as_bytes()), + Value::Bool(flag) => out.push(u8::from(*flag)), + _ => { + if let Some(text) = &number { + out.extend_from_slice(text.risky_ref().as_bytes()); + } + } + } + out + }) +} + +/// How many bytes [`encode_node`] writes: the kind, the child count, the +/// label and the scalar, `number` being a number's text. +fn encoded_len(label: LabelRef<'_>, value: &Value, number: Option<&Protected>) -> usize { + let label_len = match label { + LabelRef::Root => 1, + LabelRef::Key(key) => 1 + 4 + key.len(), + LabelRef::Index(_) => 1 + 8, + }; + let scalar_len = match value { + Value::String(text) => text.len(), + Value::Bool(_) => 1, + _ => number.map_or(0, |text| text.risky_ref().len()), + }; + 1 + 4 + label_len + scalar_len +} + +/// A sealed root's plaintext: [`SEALED_ROOT`], then the whole document as +/// JSON, built under `Protected`. The JSON is measured first, so the buffer +/// is allocated once at its final size and no reallocation leaves a copy of +/// the document behind. +fn sealed_root_plaintext(value: &Value) -> Result>, serde_json::Error> { + struct Measure(usize); + impl std::io::Write for Measure { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + self.0 += bytes.len(); + Ok(bytes.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) } } - match &plain.scalar { - Some(Value::String(s)) => out.extend_from_slice(s.as_bytes()), - Some(Value::Number(n)) => out.extend_from_slice(n.to_string().as_bytes()), - Some(Value::Bool(b)) => out.push(u8::from(*b)), - _ => {} - } - out + let mut measure = Measure(0); + serde_json::to_writer(&mut measure, value)?; + Protected::generate_ok(|| { + let mut out = Vec::with_capacity(1 + measure.0); + out.push(SEALED_ROOT); + serde_json::to_writer(&mut out, value)?; + Ok(out) + }) } fn seal( walked: Walked, - whole: Option>, + mut whole: Option>>, keyset_id: Uuid, key: stack_kms::DataKeyWithTag, aad: &Context<'_>, - options: JsonOptions, + sealed_root: bool, ) -> Result { if key.tag.len() > usize::from(u16::MAX) { return Err(Error::Other( @@ -713,58 +770,37 @@ fn seal( let cipher = leaf_cipher(&key.key)?; let iv = key.key.iv; let tag = key.tag; - let nodes = &walked.terms.nodes; + let Walked { terms, plain } = walked; + let nodes = &terms.nodes; let mut entries = Vec::with_capacity(nodes.len()); - for (j, (node, plain)) in nodes.iter().zip(&walked.plain).enumerate() { - let plaintext = match (&whole, j) { - (Some(whole), 0) => { - let mut out = Vec::with_capacity(whole.len() + 1); - out.push(SEALED_ROOT); - out.extend_from_slice(whole); - out - } - _ => encode_node(plain), - }; + for (j, (node, plain)) in nodes.iter().zip(plain).enumerate() { + // The root's plaintext, the first, is the whole document when the + // root is sealed (`whole` is taken there, and is `None` after); the + // node encoding it replaces is wiped as it drops. + let plaintext = whole.take().unwrap_or(plain); let c = cipher .seal( - Protected::new(plaintext), + plaintext, &entry_aad(keyset_id, aad, &tag, &node.s, j as u64), ) .map_err(|_| Error::Aead)?; let c = c.as_ref().to_vec(); let parent_token = node.parent.and_then(|p| nodes[p].token); - let (o, p, w) = match (parent_token, node.ordinal, node.token) { - (Some(parent), Some(o), Some(token)) => ( - Some(o), - Some(parent_tag(&iv, &parent, o)), - Some(xor(&token, &mask(&iv, &parent, &c))), - ), - _ => (None, None, None), - }; - let n = if node.container_element { - node.token.map(|t| group_tag(&iv, &t)) - } else { - None - }; - let g = node - .element - .and_then(|e| nodes[e].token) - .map(|t| group_tag(&iv, &t)); + let p = parent_token + .zip(node.ordinal) + .map(|(parent, o)| parent_tag(&iv, &parent, o)); entries.push(JsonEntry { s: node.s, - vs: node.vs, c, - o, p, - w, - n, - g, op: node.op.clone(), }); } + let projection = super::query::group_of(nodes, None); Ok(JsonDocument::new( - JsonHeader::new(keyset_id, iv, tag, options.sealed_root), + JsonHeader::new(keyset_id, iv, tag, sealed_root), entries, + projection, )) } @@ -772,28 +808,41 @@ fn seal( // Opening // ============================================================================= -/// An opened node. +/// An opened node. Every part that holds plaintext (a key, a scalar, a +/// sealed root's document) is under `Protected`; the kind and child count +/// are the node's framing. struct Opened { label: Label, kind: Kind, - scalar: Option, + scalar: Option>, children: u32, /// The whole document, from a sealed root. - whole: Option, + whole: Option>, } fn shape(expected: &'static str) -> Error { Error::OpenedType { expected } } -fn decode_node(bytes: &[u8]) -> Result { +/// Decode an opened entry's plaintext. The bytes stay in the cipher's +/// `Protected`: the framing (kind, child count, label tag, key length) is +/// read in place, and every part that carries plaintext is read out +/// through [`read_protected`], wrapped as it is made. +fn decode_node(plaintext: &Protected>) -> Result { let refused = || shape("a JSON index node"); + let bytes: &[u8] = plaintext.risky_ref(); let (&first, rest) = bytes.split_first().ok_or_else(refused)?; if first == SEALED_ROOT { - let whole: Value = serde_json::from_slice(rest).map_err(|_| refused())?; + let mut kind = None; + let whole = read_protected(plaintext, |bytes: &Vec| { + let whole = PlainJson::parse(bytes.get(1..)?)?; + kind = Some(Kind::of(whole.value())); + Some(whole) + }) + .ok_or_else(refused)?; return Ok(Opened { label: Label::Root, - kind: Kind::of(&whole), + kind: kind.ok_or_else(refused)?, scalar: None, children: 0, whole: Some(whole), @@ -806,43 +855,51 @@ fn decode_node(bytes: &[u8]) -> Result { let (count, rest) = rest.split_at(4); let children = u32::from_le_bytes([count[0], count[1], count[2], count[3]]); let (&label_tag, rest) = rest.split_first().ok_or_else(refused)?; - let (label, rest) = match label_tag { - 0x00 => (Label::Root, rest), + // Offsets into `bytes`, so the plaintext parts are read out of the + // `Protected` itself. + let at = bytes.len() - rest.len(); + let (label, scalar_at) = match label_tag { + 0x00 => (Label::Root, at), 0x01 => { if rest.len() < 4 { return Err(refused()); } - let (len, rest) = rest.split_at(4); - let len = u32::from_le_bytes([len[0], len[1], len[2], len[3]]) as usize; - if rest.len() < len { - return Err(refused()); - } - let (key, rest) = rest.split_at(len); - let key = std::str::from_utf8(key).map_err(|_| refused())?; - (Label::Key(key.to_owned()), rest) + let len = u32::from_le_bytes([rest[0], rest[1], rest[2], rest[3]]) as usize; + let start = at + 4; + let end = start.checked_add(len).filter(|end| *end <= bytes.len()); + let end = end.ok_or_else(refused)?; + let key = read_protected(plaintext, |bytes: &Vec| { + std::str::from_utf8(bytes.get(start..end)?) + .ok() + .map(str::to_owned) + }) + .ok_or_else(refused)?; + (Label::Key(key), end) } 0x02 => { if rest.len() < 8 { return Err(refused()); } - let (index, rest) = rest.split_at(8); - let mut bytes = [0; 8]; - bytes.copy_from_slice(index); - (Label::Index(u64::from_le_bytes(bytes)), rest) + let mut index = [0; 8]; + index.copy_from_slice(&rest[..8]); + (Label::Index(u64::from_le_bytes(index)), at + 8) } _ => return Err(refused()), }; let scalar = match kind { - Kind::Null => Some(Value::Null), - Kind::Bool => Some(Value::Bool(rest.first().is_some_and(|&b| b != 0))), - Kind::Number => { - let text = std::str::from_utf8(rest).map_err(|_| refused())?; - Some(Value::Number(text.parse().map_err(|_| refused())?)) - } - Kind::String => Some(Value::String( - std::str::from_utf8(rest).map_err(|_| refused())?.to_owned(), - )), Kind::Object | Kind::Array => None, + _ => Some( + read_protected(plaintext, |bytes: &Vec| { + let rest = bytes.get(scalar_at..)?; + Some(PlainJson::from_value(match kind { + Kind::Null => Value::Null, + Kind::Bool => Value::Bool(rest.first().is_some_and(|&b| b != 0)), + Kind::Number => Value::Number(std::str::from_utf8(rest).ok()?.parse().ok()?), + _ => Value::String(std::str::from_utf8(rest).ok()?.to_owned()), + })) + }) + .ok_or_else(refused)?, + ), }; Ok(Opened { label, @@ -853,6 +910,8 @@ fn decode_node(bytes: &[u8]) -> Result { }) } +/// Open one entry. The plaintext the cipher returns stays in its +/// `Protected` and is wiped when decoding is done. fn open_entry( cipher: &Aes256Cipher, header: &JsonHeader, @@ -866,12 +925,19 @@ fn open_entry( &entry_aad(header.keyset_id, aad, &header.tag, &entry.s, j), ) .map_err(|_| Error::Aead)?; - decode_node(&plaintext.risky_unwrap()) + decode_node(&plaintext) +} + +/// Take a node's label out of it, so the rebuild can consume the node. +fn take_label(mut node: Opened) -> (Label, Opened) { + (std::mem::replace(&mut node.label, Label::Root), node) } /// Rebuild the value of `node`, taking its descendants from `rest`, the /// pre-order sequence after it. Every child is taken with `next`, so each -/// step consumes an entry and the rebuild ends on any sequence. +/// step consumes an entry and the rebuild ends on any sequence. The value +/// is built under `Protected`: each member and element is moved in from +/// its own, so a rebuild that fails part-way wipes what it had. /// /// `depth` is `node`'s depth below the rebuild's start. Each entry is /// authentic, but which entries the server returned, and so how deep the @@ -879,49 +945,55 @@ fn open_entry( /// list only the containers of a wide document, would otherwise recurse as /// deep as there are rows. Nothing the index seals nests deeper than /// [`MAX_DEPTH`], so deeper is refused as a shape mismatch. -fn rebuild<'a>( - node: &'a Opened, - rest: &mut impl Iterator, +fn rebuild( + node: Opened, + rest: &mut impl Iterator, depth: usize, -) -> Result { - if let Some(whole) = &node.whole { - return Ok(whole.clone()); +) -> Result, Error> { + if let Some(whole) = node.whole { + return Ok(whole); } if depth > MAX_DEPTH { return Err(shape("a JSON document no deeper than the index seals")); } match node.kind { Kind::Object => { - let mut map = Map::new(); + let mut object = protected_object(); for _ in 0..node.children { - let member = next_child(rest)?; - let Label::Key(key) = &member.label else { + let (label, member) = take_label(next_child(rest)?); + let Label::Key(key) = label else { return Err(shape("an object member")); }; let value = rebuild(member, rest, depth + 1)?; - let _ = map.insert(key.clone(), value); + insert_member(&mut object, key, value); } - Ok(Value::Object(map)) + Ok(object) } Kind::Array => { - let mut items = Vec::with_capacity(node.children as usize); + let mut array = protected_array(node.children as usize); for i in 0..node.children { let element = next_child(rest)?; - if element.label != Label::Index(u64::from(i)) { + if !matches!(element.label, Label::Index(at) if at == u64::from(i)) { return Err(shape("an array element in order")); } - items.push(rebuild(element, rest, depth + 1)?); + push_element(&mut array, rebuild(element, rest, depth + 1)?); } - Ok(Value::Array(items)) + Ok(array) } - _ => node.scalar.clone().ok_or_else(|| shape("a JSON scalar")), + _ => node.scalar.ok_or_else(|| shape("a JSON scalar")), } } -fn next_child<'a>(rest: &mut impl Iterator) -> Result<&'a Opened, Error> { +fn next_child(rest: &mut impl Iterator) -> Result { rest.next().ok_or_else(|| shape("a complete JSON subtree")) } +/// The value the caller asked for, out of the `Protected` that held it: +/// the one place an opened JSON plaintext is unwrapped. +fn hand_over(value: Protected) -> Value { + value.risky_unwrap().into_value() +} + /// The retrieve request for a document's one data key, and the AAD its /// entries open under. fn retrieval( @@ -958,6 +1030,7 @@ fn open_document<'a, K>( .ok_or_else(|| shape("a document with a root entry"))?; return open_entry(&aes, &doc.header, &aad, root, 0)? .whole + .map(hand_over) .ok_or_else(|| shape("a sealed root")); } let opened = doc @@ -966,7 +1039,7 @@ fn open_document<'a, K>( .enumerate() .map(|(j, entry)| open_entry(&aes, &doc.header, &aad, entry, j as u64)) .collect::, _>>()?; - let mut rest = opened.iter(); + let mut rest = opened.into_iter(); let root = rest .next() .ok_or_else(|| shape("a document with a root entry"))?; @@ -974,7 +1047,7 @@ fn open_document<'a, K>( if rest.next().is_some() { return Err(shape("a document with no entries past its root")); } - Ok(value) + Ok(hand_over(value)) }) } @@ -1004,6 +1077,7 @@ impl DecryptInto for JsonNode { opened .whole .or(opened.scalar) + .map(hand_over) .ok_or_else(|| shape("a JSON scalar, or a sealed root")) }) }) @@ -1012,15 +1086,16 @@ impl DecryptInto for JsonNode { /// Open the rows of a walk: each complete subtree the rows hold, as a /// value, in order. What `json_tree_subtree_at` returns is one subtree per -/// node whose selector matched; what `json_tree_subtree` returns is the -/// children of one node, which [`DecryptInto`] assembles into that -/// node's value, or this opens into its children's values. The sealed root -/// a walk fell back to opens whole, and the attached path selects over it -/// to the same values the walked rows open to: one per node the path -/// reaches, for a path attached with [`JsonSubtree::at`]; the children of -/// the one node at the path, for [`JsonSubtree::below`]. An object's -/// members come in an order neither form fixes, and without their keys; -/// open it as a [`Value`] to keep them. +/// node its gates reached, each rooted at depth `0`; what +/// `json_tree_subtree` returns is the children of one node, which +/// [`DecryptInto`] assembles into that node's value, or this opens +/// into its children's values. The sealed root a walk fell back to opens +/// whole, and the key's path selects over it to the same values the walked +/// rows open to: one per node the path reaches, for a key with gates +/// (`json_tree_subtree_at`); the children of the one node at the path, for +/// a key without (`json_tree_subtree`). An object's members come in an +/// order neither form fixes, and without their keys; open it as a +/// [`Value`] to keep them. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { @@ -1028,29 +1103,46 @@ impl DecryptInto> for JsonSubtree { if self.is_sealed_root() { let below = self.below; return open_fallback(self, cipher, context).try_map(move |(path, whole)| { - let found = path.select(&whole); - if !below { - return Ok(found.into_iter().cloned().collect()); - } - match found.as_slice() { - [node] => Ok(children(node)), - _ => Err(shape("one node at the walked path")), - } + // The selection is made inside the document's + // `Protected` and comes out wrapped; the document is + // wiped when it drops at the end of this closure. + let selected = read_protected(&whole, |doc: &PlainJson| { + let found = path.select(doc.value()); + let selected = if below { + match found.as_slice() { + [node] => children(node), + _ => return None, + } + } else { + found.into_iter().cloned().collect() + }; + Some(PlainJson::from_value(Value::Array(selected))) + }) + .ok_or_else(|| shape("one node at the walked path"))?; + Ok(match hand_over(selected) { + Value::Array(items) => items, + other => vec![other], + }) }); } open_subtree(self, cipher, context) - .map(|forest| forest.into_iter().map(|(_, v)| v).collect()) + .map(|forest| forest.into_iter().map(|(_, v)| hand_over(v)).collect()) }) } } -/// Open the rows of `json_tree_subtree(doc, token)` into the value under -/// that token: an array when the rows are elements, an object when they are -/// members. No rows is no value ([`Error::OpenedType`]): an empty container -/// and a scalar both have no descendants, and `->` on the node's selector -/// returns the node itself. The sealed root a walk fell back to opens -/// whole, and the path attached with [`JsonSubtree::below`] (or -/// [`JsonSubtree::at`]) selects the one value under it; a path that +/// Open the rows of a walk into the one value they hold. For +/// `json_tree_subtree(doc, token)`, the rows are the descendants of the +/// token's node and the value is that node's: an array when the rows are +/// elements, an object when they are members. For a gated +/// `json_tree_subtree_at`, the rows are the node the gates reached, at +/// depth `0`, and its descendants, and the value is that node's; more than +/// one reached node (a wildcard) is [`Error::OpenedType`], and opens +/// through [`DecryptInto>`] instead. No rows is no value +/// ([`Error::OpenedType`]): an empty container and a scalar both have no +/// descendants, and `->` on the node's selector returns the node itself. +/// The sealed root a walk fell back to opens whole, and the key's path +/// ([`JsonSubtreeKey::open`]) selects the one value under it; a path that /// reaches no node, or more than one, is [`Error::OpenedType`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; @@ -1058,46 +1150,68 @@ impl DecryptInto for JsonSubtree { Decryption::from_open(move |cipher| { if self.is_sealed_root() { return open_fallback(self, cipher, context).try_map(|(path, whole)| { - let mut found = path.select(&whole); - match found.len() { - 1 => Ok(found.remove(0).clone()), - 0 => Err(shape("a node at the walked path")), - _ => Err(shape("one node at the walked path")), + let mut reached = 0; + let selected = read_protected(&whole, |doc: &PlainJson| { + let found = path.select(doc.value()); + reached = found.len(); + match found.as_slice() { + [node] => Some(PlainJson::from_value((*node).clone())), + _ => None, + } + }); + match (selected, reached) { + (Some(value), _) => Ok(hand_over(value)), + (None, 0) => Err(shape("a node at the walked path")), + (None, _) => Err(shape("one node at the walked path")), } }); } - open_subtree(self, cipher, context).try_map(|forest| { - let mut members = Map::new(); - let mut elements: Vec<(u64, Value)> = Vec::new(); + let reached = self.nodes.iter().any(|n| n.depth == Some(0)); + open_subtree(self, cipher, context).try_map(move |forest| { + if reached { + let mut values: Vec> = + forest.into_iter().map(|(_, v)| v).collect(); + return match values.len() { + 1 => Ok(hand_over(values.remove(0))), + _ => Err(shape("one node at the walked path")), + }; + } + let mut members = None; + let mut elements: Vec<(u64, Protected)> = Vec::new(); for (label, value) in forest { match label { Label::Key(key) => { - let _ = members.insert(key, value); + insert_member(members.get_or_insert_with(protected_object), key, value) } Label::Index(i) => elements.push((i, value)), Label::Root => return Err(shape("a subtree below the root")), } } - match (members.is_empty(), elements.is_empty()) { - (false, true) => Ok(Value::Object(members)), - (true, false) => { + match (members, elements.is_empty()) { + (Some(members), true) => Ok(hand_over(members)), + (None, false) => { elements.sort_by_key(|(i, _)| *i); - Ok(Value::Array(elements.into_iter().map(|(_, v)| v).collect())) + let mut array = protected_array(elements.len()); + for (_, element) in elements { + push_element(&mut array, element); + } + Ok(hand_over(array)) } - (true, true) => Err(shape("a node with children")), - (false, false) => Err(shape("the children of one node")), + (None, true) => Err(shape("a node with children")), + (Some(_), false) => Err(shape("the children of one node")), } }) }) } } -/// Open every row and rebuild each complete subtree, labelled. +/// Open every row and rebuild each complete subtree, labelled, each still +/// under its `Protected`. fn open_subtree<'a, K>( subtree: JsonSubtree, cipher: &'a StackCipher, context: AeadContext, -) -> Pending<'a, Vec<(Label, Value)>, K> { +) -> Pending<'a, Vec<(Label, Protected)>, K> { let mut nodes = subtree.nodes; nodes.sort_by_key(|n| n.j); // A walk returns each entry once. Two rows at one ordinal are the @@ -1125,10 +1239,11 @@ fn open_subtree<'a, K>( .map(|n| open_entry(&aes, &header, &aad, &n.entry, n.j)) .collect::, _>>()?; let mut forest = Vec::new(); - let mut rest = opened.iter(); + let mut rest = opened.into_iter(); while let Some(top) = rest.next() { + let (label, top) = take_label(top); let value = rebuild(top, &mut rest, 0)?; - forest.push((top.label.clone(), value)); + forest.push((label, value)); } Ok(forest) }) @@ -1146,20 +1261,16 @@ fn children(node: &Value) -> Vec { } } -/// Open the sealed root a walk fell back to: the whole document, with the -/// path the rows were keyed from, which the caller selects over. Refused -/// without the path, since nothing in the rows records it. +/// Open the sealed root a walk fell back to: the whole document, under its +/// `Protected`, with the path the rows were keyed from, which the caller +/// selects over. The key that built the rows attached the path, so it is +/// always there. fn open_fallback<'a, K>( subtree: JsonSubtree, cipher: &'a StackCipher, context: AeadContext, -) -> Pending<'a, (JsonPath, Value), K> { - let Some(path) = subtree.path().cloned() else { - return Pending::failed( - cipher, - shape("the walked path, attached with JsonSubtree::below or JsonSubtree::at, to select over the sealed root the walk returned"), - ); - }; +) -> Pending<'a, (JsonPath, Protected), K> { + let path = subtree.path().clone(); let Some(root) = subtree.nodes().first().cloned() else { return Pending::failed(cipher, shape("the sealed root")); }; @@ -1180,14 +1291,25 @@ fn open_fallback<'a, K>( // Extraction keys, on the keyset cipher // ============================================================================= -/// The token of the node at `path`, which must be positional: a chain head -/// by PRF, then one hash per step below it. -fn token_for( +/// The key that walks `path`: the token of the deepest node on it the +/// client can derive, and one gate per step below that. +/// +/// A chain head's token is a PRF of its path. An array element's is one +/// hash under its parent's, by its position and the element selector, so +/// the client follows array steps as far as they run from the head. An +/// object member's ordinal is a rank among its siblings' keys, which the +/// client does not know, so from the first key step on the client cannot +/// derive tokens and the server walks instead, through gates: the step's +/// selector for a key or a wildcard (every element's selector is the +/// wildcard's, so a wildcard admits them all), and for an array step the +/// position with the element selector, so a member that happens to sit at +/// that ordinal under an object is not taken for an element. +fn subtree_key_for( prf: &HmacSha256Prf, context: &Context<'_>, - options: JsonOptions, + options: &JsonOptions, path: &JsonPath, -) -> Result { +) -> Result { let segments = path.segments(); let (head, rest) = if options.sealed_root { let Some((first, rest)) = segments.split_first() else { @@ -1198,22 +1320,51 @@ fn token_for( } else { (JsonPath::root(), segments) }; - let mut token = prf16( - prf, - TOKEN_DOMAIN, - context, - &head.token_bytes().ok_or(TermError::JsonSealedRoot)?, - )?; + // A wildcard head names no one node. + let head_bytes = head.token_bytes().ok_or(TermError::JsonSealedRoot)?; + let mut token = prf16(prf, TOKEN_DOMAIN, context, &head_bytes)?; + let mut prefix = head; + let mut rest = rest.iter(); + let mut gates = Vec::new(); + for segment in rest.by_ref() { + prefix = prefix.then(segment.clone()); + match segment { + Segment::Index(index) => { + let o = u32::try_from(*index).map_err(|_| TermError::JsonTooWide)?; + let element = prf16(prf, SELECTOR_DOMAIN, context, &prefix.selector_bytes())?; + token = child_token(&token, &element, o); + } + Segment::Key(_) | Segment::Wildcard => { + gates.push(Gate::Selector(prf16( + prf, + SELECTOR_DOMAIN, + context, + &prefix.selector_bytes(), + )?)); + break; + } + } + } for segment in rest { - let label = match segment { - Segment::Key(key) => Label::Key(key.clone()), - Segment::Index(index) => Label::Index(*index), - // `anchor` cut the path before any wildcard. - Segment::Wildcard => return Err(TermError::JsonSealedRoot), - }; - token = child_token(prf, context, &token, &label)?; + prefix = prefix.then(segment.clone()); + gates.push(match segment { + Segment::Index(index) => Gate::Slot( + u32::try_from(*index).map_err(|_| TermError::JsonTooWide)?, + prf16(prf, SELECTOR_DOMAIN, context, &prefix.selector_bytes())?, + ), + Segment::Key(_) | Segment::Wildcard => Gate::Selector(prf16( + prf, + SELECTOR_DOMAIN, + context, + &prefix.selector_bytes(), + )?), + }); } - Ok(token) + Ok(JsonSubtreeKey { + token, + gates, + path: path.clone(), + }) } impl KeysetCipher<'_, K> { @@ -1240,9 +1391,9 @@ impl KeysetCipher<'_, K> { /// The key that walks the subtree under `path`, under the field /// `descriptor` and the index's `options`: the token of the deepest - /// positional node on the path, and, when the path continues with a - /// wildcard below it, the selector of the whole path, which the walk - /// gates on. What `json_tree_subtree` and `json_tree_subtree_at` take. + /// node on the path the client can derive (a chain head, then array + /// positions), and one gate per step below it, which the server walks + /// through. What `json_tree_subtree` and `json_tree_subtree_at` take. /// /// # Errors /// @@ -1252,30 +1403,18 @@ impl KeysetCipher<'_, K> { pub fn json_subtree_key<'c>( &self, path: &JsonPath, - options: JsonOptions, + options: &JsonOptions, descriptor: NonEmpty>, ) -> Pending<'_, JsonSubtreeKey, K> { let context = descriptor.into_prf_context().into_owned(); - let (anchor, cut) = path.anchor(); - let key = token_for(self.prf(), &context, options, &anchor).and_then(|token| { - let selector = cut - .then(|| { - prf16( - self.prf(), - SELECTOR_DOMAIN, - &context, - &path.selector_bytes(), - ) - }) - .transpose()?; - Ok(JsonSubtreeKey { token, selector }) - }); + let key = subtree_key_for(self.prf(), &context, options, path); Pending::ready(self, key.map_err(Error::from)) } } #[cfg(test)] mod tests { + use super::super::base64; use super::*; use serde_json::json; use vitaminc_prf::PrfKeyInit; @@ -1284,7 +1423,14 @@ mod tests { HmacSha256Prf::new(Protected::new([7; 32])) } - fn terms(value: &Value, options: JsonOptions) -> JsonTerms { + fn open_root() -> JsonOptions { + JsonOptions { + sealed_root: false, + ..JsonOptions::new() + } + } + + fn terms(value: &Value, options: &JsonOptions) -> JsonTerms { walk( value, options, @@ -1316,7 +1462,7 @@ mod tests { #[test] fn the_terms_name_a_node_by_its_path() { - let t = terms(&json!({"a": [1, {"b": 2}]}), JsonOptions::default()); + let t = terms(&json!({"a": [1, {"b": 2}]}), &JsonOptions::default()); // $, $.a, $.a[0], $.a[1], $.a[1].b assert_eq!(t.len(), 5); assert!(!t.is_empty()); @@ -1332,18 +1478,17 @@ mod tests { fn the_terms_print_neither_key_names_nor_tokens() { let t = terms( &json!({"alice@example.com": {"ssn": "123"}}), - JsonOptions::default(), + &JsonOptions::default(), ); + assert!(t.nodes()[1].token.is_some(), "the terms hold a token"); let printed = format!("{t:?}"); assert_eq!(printed, "JsonTerms { nodes: 3, .. }"); - let token = t.nodes()[1].token.unwrap(); assert!(!printed.contains("alice")); - assert!(!printed.contains(&super::super::hex(&token))); } #[test] fn every_element_of_an_array_shares_a_selector_and_values_do_not() { - let t = terms(&json!({"a": [1, 2, 1]}), JsonOptions::default()); + let t = terms(&json!({"a": [1, 2, 1]}), &JsonOptions::default()); let nodes = t.nodes(); assert_eq!(nodes.len(), 5); assert_eq!(nodes[2].s, nodes[3].s); @@ -1354,12 +1499,12 @@ mod tests { #[test] fn the_sealed_root_has_no_token_and_its_fields_are_heads() { - let sealed = terms(&json!({"a": {"b": 1}}), JsonOptions::default()); + let sealed = terms(&json!({"a": {"b": 1}}), &JsonOptions::default()); assert!(sealed.nodes()[0].token.is_none()); assert!(sealed.nodes()[1].token.is_some()); assert!(sealed.nodes()[1].ordinal.is_none(), "a head has no ordinal"); assert!(sealed.nodes()[2].ordinal.is_some()); - let open = terms(&json!({"a": {"b": 1}}), JsonOptions { sealed_root: false }); + let open = terms(&json!({"a": {"b": 1}}), &open_root()); assert!(open.nodes()[0].token.is_some()); assert!(open.nodes()[1].ordinal.is_some()); // The chain below a head is the same hash chain either way, but @@ -1369,44 +1514,92 @@ mod tests { } #[test] - fn a_head_token_is_what_the_subtree_key_derives() { + fn the_subtree_key_derives_what_it_can_and_gates_the_rest() { let context = Context::from_encoded(b"users/profile"); - let value = json!({"a": {"b": [1, {"c": 2}]}}); - for options in [JsonOptions::default(), JsonOptions { sealed_root: false }] { - let t = terms(&value, options); - for node in t.nodes().iter().filter(|n| n.token.is_some()) { - let derived = token_for(&prf(), &context, options, &node.positional).unwrap(); - assert_eq!(Some(derived), node.token, "{}", node.positional); - } + let value = json!({"a": {"b": [1, {"c": 2}]}, "l": [[0], 1]}); + let key = |options: &JsonOptions, path: &str| { + subtree_key_for(&prf(), &context, options, &JsonPath::parse(path).unwrap()) + }; + let at = |t: &JsonTerms, path: &str| { + t.nodes() + .iter() + .find(|n| n.positional == JsonPath::parse(path).unwrap()) + .unwrap() + .clone() + }; + let options = JsonOptions::default(); + let t = terms(&value, &options); + // A head's token, and the positions under it, derive exactly. + for path in ["$.a", "$.l", "$.l[0]", "$.l[0][0]", "$.l[1]"] { + let k = key(&options, path).unwrap(); + assert_eq!(Some(*k.token_bytes()), at(&t, path).token, "{path}"); + assert!(k.gates().is_empty(), "{path}"); } + // The server derives an object member's token the same way, from + // its parent's, its selector and its ordinal. + let a = at(&t, "$.a"); + let b = at(&t, "$.a.b"); + assert_eq!( + b.token, + Some(child_token(&a.token.unwrap(), &b.s, b.ordinal.unwrap())) + ); + // The selector is bound: the same parent and ordinal under another + // selector (an element where the row holds a member) is another + // token. + assert_ne!( + b.token, + Some(child_token(&a.token.unwrap(), &a.s, b.ordinal.unwrap())) + ); + // Below a key step, gates: the step's selector, a position, the + // wildcard's selector (every element's). + let k = key(&options, "$.a.b[1].c").unwrap(); + assert_eq!(Some(*k.token_bytes()), a.token); + assert_eq!( + k.gates(), + [ + base64(&b.s), + format!("#1:{}", base64(&at(&t, "$.a.b[1]").s)), + base64(&at(&t, "$.a.b[1].c").s) + ] + ); + let k = key(&options, "$.a.b[*].c").unwrap(); + assert_eq!( + k.gates(), + [ + base64(&b.s), + base64(&at(&t, "$.a.b[0]").s), + base64(&at(&t, "$.a.b[1].c").s) + ] + ); + // Under the sealed root the root has no token; under an open root + // it has, and `$.a` is then one gate below it. assert!(matches!( - token_for(&prf(), &context, JsonOptions::default(), &JsonPath::root()), + key(&JsonOptions::default(), "$"), Err(TermError::JsonSealedRoot) )); - assert!(token_for( - &prf(), - &context, - JsonOptions { sealed_root: false }, - &JsonPath::root() - ) - .is_ok()); + let open = open_root(); + let root = key(&open, "$").unwrap(); + assert!(root.gates().is_empty()); + let a = key(&open, "$.a").unwrap(); + assert_eq!(a.token_bytes(), root.token_bytes()); + assert_eq!(a.gates().len(), 1); } #[test] fn a_root_scalar_carries_its_element_selectors_as_alt() { - let scalar = terms(&json!("a"), JsonOptions::default()); - let array = terms(&json!(["a"]), JsonOptions::default()); + let scalar = terms(&json!("a"), &JsonOptions::default()); + let array = terms(&json!(["a"]), &JsonOptions::default()); assert_eq!( scalar.alt(), Some((array.nodes()[1].s, array.nodes()[1].vs)), "the alt pair is the stored array element's" ); - assert!(terms(&json!({}), JsonOptions::default()).alt().is_none()); + assert!(terms(&json!({}), &JsonOptions::default()).alt().is_none()); } #[test] fn container_elements_group_what_sits_inside_them() { - let t = terms(&json!({"a": [{"x": 1}, 2, [3]]}), JsonOptions::default()); + let t = terms(&json!({"a": [{"x": 1}, 2, [3]]}), &JsonOptions::default()); let nodes = t.nodes(); // $, $.a, $.a[0], $.a[0].x, $.a[1], $.a[2], $.a[2][0] assert!(nodes[2].container_element); @@ -1438,103 +1631,133 @@ mod tests { ); } - /// A token's holder cannot test key names: the step from a parent's - /// token to a child's, and an object member's rank, are keyed with the - /// index key, so the same parent token and label give unrelated bytes - /// under another key, and nothing public reproduces them. + /// A token's holder cannot rank guessed key names: an object member's + /// rank is keyed with the index key, so the same parent token and key + /// give unrelated bytes under another key. #[test] - fn the_token_step_and_the_rank_are_keyed() { + fn the_rank_is_keyed() { let context = Context::from_encoded(b"users/profile"); let other = HmacSha256Prf::new(Protected::new([8; 32])); let parent = [5; 16]; - let key = Label::Key("email@example.com".into()); - let step = |prf: &HmacSha256Prf, label: &Label| { - child_token(prf, &context, &parent, label).unwrap() - }; - assert_eq!(step(&prf(), &key), step(&prf(), &key), "deterministic"); - assert_ne!(step(&prf(), &key), step(&other, &key), "keyed"); - assert_ne!( - step(&prf(), &key), - h16(&[b"t", &parent, &[0x01], b"email@example.com"]), - "not the unkeyed hash" - ); - assert_ne!( - step(&prf(), &Label::Index(0)), - step(&prf(), &Label::Key("0".into())), - "an element is not a member" - ); - assert_ne!( - rank(&prf(), &context, &parent, "a").unwrap(), - rank(&other, &context, &parent, "a").unwrap(), - "the rank is keyed" - ); + let rank_under = |prf: &HmacSha256Prf| rank(prf, &context, &parent, "a").unwrap(); + assert_eq!(rank_under(&prf()), rank_under(&prf()), "deterministic"); + assert_ne!(rank_under(&prf()), rank_under(&other), "keyed"); + } + + /// The plain bytes of a node's encoding, for the test to inspect. + fn bytes(plain: &Protected>) -> Vec { + plain.risky_ref().clone() + } + + fn key_of(label: &Label) -> Option { + match label { + Label::Key(key) => Some(key.risky_ref().clone()), + _ => None, + } } #[test] fn a_node_encodes_and_decodes() { - for (label, scalar, kind) in [ - (Label::Root, None, Kind::Object), - (Label::Key("k".into()), Some(json!("v")), Kind::String), - (Label::Index(3), Some(json!(1.5)), Kind::Number), - (Label::Index(0), Some(json!(true)), Kind::Bool), - (Label::Key("n".into()), Some(Value::Null), Kind::Null), - (Label::Key("a".into()), None, Kind::Array), + for (label, value, kind) in [ + (LabelRef::Root, json!({}), Kind::Object), + (LabelRef::Key("k"), json!("v"), Kind::String), + (LabelRef::Index(3), json!(1.5), Kind::Number), + (LabelRef::Index(0), json!(true), Kind::Bool), + (LabelRef::Key("n"), Value::Null, Kind::Null), + (LabelRef::Key("a"), json!([]), Kind::Array), ] { - let plain = NodePlain { - label: label.clone(), - kind, - scalar: scalar.clone(), - children: 2, - }; - let opened = decode_node(&encode_node(&plain)).unwrap(); - assert_eq!(opened.label, label); + let opened = decode_node(&encode_node(kind, 2, label, &value)).unwrap(); + match (label, &opened.label) { + (LabelRef::Root, Label::Root) => {} + (LabelRef::Index(a), Label::Index(b)) => assert_eq!(a, *b), + (LabelRef::Key(a), Label::Key(_)) => { + assert_eq!(key_of(&opened.label).as_deref(), Some(a)) + } + _ => panic!("{kind:?}: the label changed kind"), + } assert_eq!(opened.kind, kind); - assert_eq!(opened.scalar, scalar); + let scalar = opened.scalar.map(|s| s.risky_unwrap().into_value()); + assert_eq!(scalar, (!kind.is_container()).then_some(value)); assert_eq!(opened.children, 2); } - assert!(decode_node(&[]).is_err()); - assert!(decode_node(&[9, 0, 0, 0, 0, 0]).is_err()); + assert!(decode_node(&Protected::new(vec![])).is_err()); + assert!(decode_node(&Protected::new(vec![9, 0, 0, 0, 0, 0])).is_err()); // The empty key is a key: its label is the tag and a zero length, // and nothing after it. - let empty_key = NodePlain { - label: Label::Key(String::new()), - kind: Kind::Object, - scalar: None, - children: 0, - }; - let encoded = encode_node(&empty_key); - assert_eq!(encoded.len(), 1 + 4 + 1 + 4); + let encoded = encode_node(Kind::Object, 0, LabelRef::Key(""), &json!({})); + assert_eq!(encoded.risky_ref().len(), 1 + 4 + 1 + 4); assert_eq!( - decode_node(&encoded).unwrap().label, - Label::Key(String::new()) + key_of(&decode_node(&encoded).unwrap().label).as_deref(), + Some("") ); + // The encoding is allocated once at its size: nothing reallocated + // and left a copy behind. + for (label, value) in [ + (LabelRef::Key("key"), json!("a string value")), + (LabelRef::Index(9), json!(-12.25)), + (LabelRef::Root, json!(false)), + (LabelRef::Key("z"), Value::Null), + (LabelRef::Root, json!({"a": 1})), + ] { + let plain = encode_node(Kind::of(&value), 0, label, &value); + let number = match &value { + Value::Number(n) => Some(Protected::new(n.to_string())), + _ => None, + }; + assert_eq!( + encoded_len(label, &value, number.as_ref()), + plain.risky_ref().len(), + "measured before it is written" + ); + assert_eq!(plain.risky_ref().len(), plain.risky_ref().capacity()); + } // Every truncation of a labelled node is refused, not read past. - let keyed = encode_node(&NodePlain { - label: Label::Key("key".into()), - kind: Kind::Array, - scalar: None, - children: 1, - }); - let indexed = encode_node(&NodePlain { - label: Label::Index(7), - kind: Kind::Object, - scalar: None, - children: 1, - }); - for bytes in [&keyed, &indexed, &encoded] { - for len in 0..bytes.len() { - assert!(decode_node(&bytes[..len]).is_err(), "{len} of {bytes:?}"); + let keyed = bytes(&encode_node( + Kind::Array, + 1, + LabelRef::Key("key"), + &json!([]), + )); + let indexed = bytes(&encode_node( + Kind::Object, + 1, + LabelRef::Index(7), + &json!({}), + )); + let encoded = bytes(&encoded); + for whole in [&keyed, &indexed, &encoded] { + for len in 0..whole.len() { + assert!( + decode_node(&Protected::new(whole[..len].to_vec())).is_err(), + "{len} of {whole:?}" + ); } } } + #[test] + fn a_sealed_root_plaintext_is_the_document_allocated_once() { + let value = json!({"a": [1, "two", {"three": null}], "b": "x".repeat(300)}); + let plain = sealed_root_plaintext(&value).unwrap(); + assert_eq!(plain.risky_ref()[0], SEALED_ROOT); + assert_eq!( + serde_json::from_slice::(&plain.risky_ref()[1..]).unwrap(), + value + ); + assert_eq!(plain.risky_ref().len(), plain.risky_ref().capacity()); + let opened = decode_node(&plain).unwrap(); + assert!(matches!(opened.label, Label::Root)); + assert_eq!(opened.kind, Kind::Object); + assert_eq!(opened.whole.map(hand_over), Some(value)); + } + #[test] fn a_key_tag_must_fit_the_header() { let value = json!({"a": 1}); let context = Context::from_encoded(b"users/profile"); let sealed = |tag_len: usize| { - let walked = walk(&value, JsonOptions::default(), &prf(), &context).unwrap(); + let walked = walk(&value, &JsonOptions::default(), &prf(), &context).unwrap(); let key = stack_kms::DataKeyWithTag { key: stack_kms::DataKey { iv: [1; 16], @@ -1545,11 +1768,11 @@ mod tests { }; seal( walked, - Some(serde_json::to_vec(&value).unwrap()), + Some(sealed_root_plaintext(&value).unwrap()), Uuid::from_u128(1), key, &context, - JsonOptions::default(), + true, ) }; let longest = usize::from(u16::MAX); @@ -1575,7 +1798,7 @@ mod tests { }; assert!(walk( &nested(MAX_DEPTH), - JsonOptions::default(), + &JsonOptions::default(), &prf(), &Context::empty() ) @@ -1583,7 +1806,7 @@ mod tests { assert!(matches!( walk( &nested(MAX_DEPTH + 1), - JsonOptions::default(), + &JsonOptions::default(), &prf(), &Context::empty() ), diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index 33c47da7f..ff59c71f0 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -7,22 +7,28 @@ //! //! - a **selector** `s`, a PRF of the node's path with every array position //! folded into the wildcard, so every element of an array shares one; -//! - a **value selector** `vs`, a PRF of the node's kind and, for a scalar, -//! its canonical value, bound to `s`; containment is a set comparison -//! over these two; //! - the node's own **ciphertext** `c`, sealed under one ZeroKMS data key //! per document: the node's label, its kind and, for a scalar, its //! value, and for a container its child count. Nothing beneath a node is //! encrypted in that node's entry, so a document's ciphertext is linear //! in its size; -//! - the **token chain** fields `p` and `w`, which let the server walk the -//! subtree under a node from one client-derived token and nothing else, -//! and the **group tags** `n` and `g`, which tell the server which -//! entries sit inside the same array element, so containment agrees with -//! PostgreSQL's own `jsonb @>` across elements; -//! - a leaf ordering term `op` on number and string leaves, the same OPE -//! term the scalar [`Ope`](crate::target::Ope) index derives, keyed by -//! the column. +//! - the **parent tag** `p`, the first eight bytes of +//! `H("p" ‖ iv ‖ t(parent) ‖ LE32(o))`, which lets the server walk the subtree under a node from one client-derived +//! token and nothing else: the ordinals `o` under a node are dense from +//! zero, so a walk probes them in turn until the first miss, and derives +//! each hit's own token as `H("t" ‖ t(parent) ‖ s ‖ LE32(o))` from the +//! hit's own selector `s`, which keeps an array element's token apart +//! from an object member's at the same ordinal; +//! - a leaf ordering term `op` on the number and string leaves at the +//! paths the index declares ordered ([`JsonOptions::ordered`]), the +//! same OPE term the scalar [`Ope`](crate::target::Ope) index derives, +//! keyed by the column. No path is ordered by default. +//! +//! Beside the entries the document stores its containment **projection** +//! `q`: one `(s, vs)` pair per node, `vs` being a PRF of the node's kind +//! and, for a scalar, its canonical value, bound to `s`, nested by array +//! element so that PostgreSQL's own `jsonb @>` over two projections is +//! exact containment. The value selectors live there and nowhere else. //! //! By default the root is **sealed** ([`JsonOptions::sealed_root`]): the root //! entry's ciphertext is the whole document, the top-level fields are the @@ -60,18 +66,38 @@ //! `N² / 2^97`, negligible for any document. The ordinal still binds the //! entry to its place through the AAD, so a moved or reordered entry fails //! to open as the design requires. The cost is twelve bytes per entry. -//! - **The ordinal field is `o`, not `i`**, because `i` is the EQL -//! envelope's identifier, which the SQL grafts onto an extracted entry. +//! - **Tokens chain by ordinal, not by label, and nothing stores a token.** +//! The design wraps each child's label-derived token under its parent's +//! (`w`) and stores the ordinal (`o`) so the server can find and unmask +//! it. Here a child's token is a hash of its parent's token, its own +//! selector and its ordinal, so the server derives it as it walks and +//! both fields go. The +//! ordinals under a node are dense from zero, so the walk needs no +//! fan-out bound either: it probes until the first miss. The price is +//! that the client cannot derive an object member's token (its ordinal +//! is a rank among sibling keys it does not know), so it walks from the +//! nearest token it can derive through one gate per step +//! ([`JsonSubtreeKey`]). //! //! # Byte encodings //! -//! The document's wire form is JSON, as the EQL domain stores it: -//! sixteen-byte fields (`s`, `vs`, `p`, `w`, `n`, `g`) and the OPE term `op` -//! in lowercase hex, the ciphertext `c` and the header `h` in standard -//! base64. The header's bytes are a frozen layout, [`JsonHeader`]. The +//! The document's wire form is JSON, as the EQL domain stores it. Every +//! field the SQL compares, derives or decodes is standard base64 with +//! padding, which is what PostgreSQL's `encode(x, 'base64')` writes and +//! `decode(x, 'base64')` reads: the selectors `s` and the pairs of `q` +//! (sixteen bytes, 24 characters), the parent tag `p` (eight bytes, 12 +//! characters), the ordering term `op`, the tokens and gates a walk +//! takes, and the header `h`, which the SQL reads the `iv` and the +//! sealed-root flag from. The ciphertext `c`, which nothing in SQL +//! decodes, is RFC 1924 base85 (`base85-simd`): five characters per four +//! bytes against base64's eight per six, on the one field that is most of +//! a document. The alphabet has no `"` or `\`, so a JSON string carries it +//! unescaped. The header's bytes are a frozen layout, [`JsonHeader`]. The //! derivations are versioned by their PAE domain labels, as every other //! term kind's are ([`sem`](crate::sem)): a change to any of them is a new //! version, never an edit in place. +use std::collections::BTreeSet; + use serde::{Deserialize, Serialize}; use uuid::Uuid; use vitaminc_protected::OpaqueDebug; @@ -80,6 +106,7 @@ use crate::target::{DecryptField, DecryptInto, Decryptable, Decryption}; mod derive; mod path; +mod plain; mod query; pub use derive::JsonTerms; @@ -90,14 +117,23 @@ pub use query::{ pub(crate) use derive::{seal_pending, terms_pending}; -/// A sixteen-byte tag: a selector, a token, a parent tag or a group tag. +/// A sixteen-byte tag: a selector or a token. pub type Tag = [u8; 16]; -/// How a JSON index lays a document out. Part of the index's identity: a -/// stored document and the query probing it must agree on it, so it rides -/// in the data plan beside the index and is recorded in every document's -/// header. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +/// An eight-byte parent tag: the first eight bytes of the parent-tag hash. +/// A parent tag is a lookup key for the server's walk and nothing else (an +/// entry's AAD binds its selector and ordinal, not its parent tag), so it +/// needs only to be distinct among the entries of one document; eight +/// bytes put a collision in a document of ten thousand nodes below one in +/// 10^11, and save eight bytes an entry over the full hash. +pub type ParentTag = [u8; 8]; + +/// How a JSON index lays a document out, and which of its leaves it +/// orders. Part of the index's identity: a stored document and the query +/// probing it must agree on it, so it rides in the data plan beside the +/// index. The layout is recorded in every document's header; the ordered +/// paths are not, since a stored leaf shows whether it carries a term. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] pub struct JsonOptions { /// Seal the root: the root entry's ciphertext is the whole document and /// the top-level fields are the heads of the token chain, so no token @@ -105,11 +141,77 @@ pub struct JsonOptions { /// default. Off, the chain starts at the root and a query may walk the /// whole document from the root's token. pub sealed_root: bool, + /// The paths whose number and string leaves carry an ordering term + /// `op`, and so answer `op::AtPath(op::Lt)` and the other orderings. + /// None by default: an ordering term reveals the order of the leaf + /// across every row that holds one, and costs about eighty bytes a + /// leaf, so a path pays for it only when asked. A wildcard step covers + /// every element of the array (`$.orders[*].total`). A position + /// (`$.scores[2]`) is refused, here and in an ordering query + /// ([`TermError::JsonOrderingAtPosition`](crate::sem::TermError::JsonOrderingAtPosition)): + /// an ordering compares the leaf `->` extracts by the path's selector, + /// which reads every position as the wildcard, so it reaches the first + /// element at the path and never the one a position names. An ordering + /// asked at a path no entry here covers is refused + /// ([`TermError::JsonPathNotOrdered`](crate::sem::TermError::JsonPathNotOrdered)). + pub ordered: BTreeSet, +} + +impl JsonOptions { + /// The defaults: the root sealed, no leaf ordered. + pub const fn new() -> Self { + Self { + sealed_root: true, + ordered: BTreeSet::new(), + } + } + + /// These options with the leaves at `paths` ordered, each path parsed + /// as [`JsonPath::parse`] does. + /// + /// # Errors + /// + /// [`TermError::JsonPath`](crate::sem::TermError::JsonPath) for a path + /// that is not one, and + /// [`TermError::JsonOrderingAtPosition`](crate::sem::TermError::JsonOrderingAtPosition) + /// for a path that names an array position, which no ordering can + /// reach (see [`ordered`](Self::ordered)). + pub fn ordering(mut self, paths: I) -> Result + where + I: IntoIterator, + I::Item: AsRef, + { + for path in paths { + let path = JsonPath::parse(path.as_ref())?; + if path.names_a_position() { + return Err(crate::sem::TermError::JsonOrderingAtPosition); + } + let _ = self.ordered.insert(path); + } + Ok(self) + } + + /// Refuse options whose ordered paths name a position, which the + /// field admits but [`ordering`](Self::ordering) does not: sealing + /// under them would put a term no ordering can reach on the leaf. + pub(crate) fn check(&self) -> Result<(), crate::sem::TermError> { + if self.ordered.iter().any(JsonPath::names_a_position) { + Err(crate::sem::TermError::JsonOrderingAtPosition) + } else { + Ok(()) + } + } + + /// Whether the leaf at `path` carries an ordering term: some ordered + /// path [covers](JsonPath::covers) it. + pub fn orders(&self, path: &JsonPath) -> bool { + self.ordered.iter().any(|ordered| ordered.covers(path)) + } } impl Default for JsonOptions { fn default() -> Self { - Self { sealed_root: true } + Self::new() } } @@ -128,8 +230,9 @@ impl Default for JsonOptions { /// | 38 | `tag_len` | 2 | `u16` little-endian | /// | 40 | ZeroKMS key `tag` | `tag_len` | required to retrieve the key | /// -/// The SQL reads the `iv` at offset 22 to compute parent and group tags; it -/// parses nothing else. The epoch is reserved for bounding a revealed token +/// The SQL reads the `iv` at offset 22 to compute parent tags, and the +/// flags at offset 1 to know whether a walk may fall back to the sealed +/// root; it parses nothing else. The epoch is reserved for bounding a revealed token /// in time (the design's "Epochs"); writing it now keeps that a new value /// rather than a new layout. /// @@ -189,13 +292,6 @@ impl JsonHeader { self.sealed_root } - /// The layout this document was laid out under. - pub fn options(&self) -> JsonOptions { - JsonOptions { - sealed_root: self.sealed_root, - } - } - /// The frozen byte layout; see the type docs. pub fn to_bytes(&self) -> Vec { let mut out = Vec::with_capacity(Self::FIXED_LEN + self.tag.len()); @@ -262,21 +358,14 @@ impl JsonHeader { /// One entry of a stored document: one JSON node. /// -/// Every field but `s`, `vs` and `c` is optional, and which are present -/// says what the node is: `p`, `w` and `o` on a node whose parent has a -/// token (everything below the chain's heads); `n` on an array element -/// that is a container; `g` on every entry inside such an element; `op` on -/// a number or string leaf. +/// `s` and `c` are always present. `p` is present on a node whose parent +/// has a token (everything below the chain's heads); `op` on a number or +/// string leaf at a path the index orders ([`JsonOptions::ordered`]). #[derive(Clone, Debug, PartialEq, Eq)] pub struct JsonEntry { s: Tag, - vs: Tag, c: Vec, - o: Option, - p: Option, - w: Option, - n: Option, - g: Option, + p: Option, op: Option>, } @@ -285,37 +374,16 @@ impl JsonEntry { pub fn selector(&self) -> &Tag { &self.s } - /// The value selector. - pub fn value_selector(&self) -> &Tag { - &self.vs - } /// The node's ciphertext, as sealed. pub fn ciphertext(&self) -> &[u8] { &self.c } - /// The node's ordinal under its parent, when its parent has a token. - pub fn ordinal(&self) -> Option { - self.o - } /// The parent tag, when the parent has a token. - pub fn parent_tag(&self) -> Option<&Tag> { + pub fn parent_tag(&self) -> Option<&ParentTag> { self.p.as_ref() } - /// The wrapped token, when the parent has a token. - pub fn wrapped_token(&self) -> Option<&Tag> { - self.w.as_ref() - } - /// This node's group tag, when it is an array element that is a - /// container. - pub fn group_tag(&self) -> Option<&Tag> { - self.n.as_ref() - } - /// The enclosing array element's group tag, when this node sits inside - /// an element that is a container. - pub fn enclosing_group_tag(&self) -> Option<&Tag> { - self.g.as_ref() - } - /// The leaf ordering term, on a number or string leaf. + /// The leaf ordering term, on a number or string leaf at an ordered + /// path. pub fn ordering_term(&self) -> Option<&[u8]> { self.op.as_deref() } @@ -327,15 +395,15 @@ impl JsonEntry { /// [`DecryptInto`](crate::DecryptInto) opens. /// /// Serializes to the wire form the EQL domain stores: `{"h": "…", "sv": -/// [{"s": "…", "vs": "…", "c": "…", …}, …], "q": {"sv": [{"s", "vs"}, …], -/// "el": […]}}`, where `q` is the document's own containment -/// [projection](Self::projection), written once so the server never -/// rebuilds it; see the [module docs](self#byte-encodings). +/// [{"s": "…", "c": "…", …}, …], "q": {"sv": [{"s", "vs"}, …], "el": +/// […]}}`, where `q` is the document's containment +/// [projection](Self::projection); see the [module docs](self#byte-encodings). #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(try_from = "DocumentWire", into = "DocumentWire")] pub struct JsonDocument { header: JsonHeader, entries: Vec, + projection: JsonProjection, } impl JsonDocument { @@ -349,32 +417,40 @@ impl JsonDocument { &self.entries } + /// The document's containment projection: what a [`JsonQuery`] is + /// matched against, stored as `q`. Computed from the plaintext when the + /// document is sealed and read back as stored: like every search field + /// it is unauthenticated (only `c` is), and a tampered `q` changes + /// which rows a containment query returns and nothing else. + pub fn projection(&self) -> &JsonProjection { + &self.projection + } + /// The children of the node whose token is `token`, as the server - /// finds them: every entry whose parent tag is the tag of `token` at - /// the entry's own ordinal, with its own token unwrapped, in ordinal - /// order. The SQL walk finds the same entries by trying the ordinals - /// `0..F`, where `F` is one more than the largest ordinal in the - /// document; checking each entry at its own ordinal gives the same - /// answer without a loop whose length a stored document chooses. A + /// finds them: the entries at the ordinals `0, 1, 2, …` under it, + /// probed until the first miss, each with its own token derived. A /// token that names no node yields nothing. /// - /// What a `jsonb_array_elements(doc, token)` returns; one level of the + /// What `json_tree_children(doc, token)` returns; one level of the /// walk [`subtree`](Self::subtree) runs to the bottom. pub fn children(&self, token: &Tag) -> Vec<(usize, &JsonEntry, Tag)> { let iv = &self.header.iv; - let mut out: Vec<(usize, &JsonEntry, Tag)> = self - .entries - .iter() - .enumerate() - .filter_map(|(j, e)| { - let (p, w, o) = (e.p.as_ref()?, e.w.as_ref()?, e.o?); - (derive::parent_tag(iv, token, o) == *p).then(|| { - let child = derive::xor(w, &derive::mask(iv, token, &e.c)); - (j, e, child) - }) - }) - .collect(); - out.sort_by_key(|(_, e, _)| e.o); + let mut out = Vec::new(); + // A node has at most as many children as the document has entries, + // so the probe loop is bounded whatever the entries say. + let most = u32::try_from(self.entries.len()).unwrap_or(u32::MAX); + for o in 0..most { + let tag = derive::parent_tag(iv, token, o); + let Some((j, entry)) = self + .entries + .iter() + .enumerate() + .find(|(_, e)| e.p.as_ref() == Some(&tag)) + else { + break; + }; + out.push((j, entry, derive::child_token(token, &entry.s, o))); + } out } @@ -396,7 +472,12 @@ impl JsonDocument { } fn walk<'a>(&'a self, token: &Tag, depth: u32, out: &mut Vec<(usize, &'a JsonEntry, u32)>) { - if depth as usize > derive::MAX_DEPTH { + // A walk finds each entry at most once, so a document whose tags do + // not form a tree (which no sealing produces) ends the walk once it + // has returned as many entries as the document holds, instead of + // looping; and no sealed document nests deeper than MAX_DEPTH, so a + // chain of tags longer than that stops there, in bounded stack. + if out.len() >= self.entries.len() || depth as usize > derive::MAX_DEPTH { return; } for (j, entry, child) in self.children(token) { @@ -419,8 +500,16 @@ impl JsonDocument { }) } - pub(crate) fn new(header: JsonHeader, entries: Vec) -> Self { - Self { header, entries } + pub(crate) fn new( + header: JsonHeader, + entries: Vec, + projection: JsonProjection, + ) -> Self { + Self { + header, + entries, + projection, + } } } @@ -486,69 +575,39 @@ where } } -/// The entries the server's walk returned for one token: what -/// `json_tree_subtree(doc, token)` yields, collected, and what +/// The entries the server's walk returned for one key, ready to open: +/// what `json_tree_subtree(doc, token)` or `json_tree_subtree_at(doc, +/// token, gates)` yields, collected, and what /// [`DecryptInto`](crate::DecryptInto) reassembles into the value -/// under that node. +/// under the walked path. Built only by the key that keyed the walk +/// ([`JsonSubtreeKey::open`]), so it always carries the walked path. /// -/// The walk returns the node's descendants and not the node itself, so the -/// node's kind is read from its children's labels: elements make an array, -/// members an object. A node with no descendants, an empty container or a -/// scalar, yields no rows and so no value; `->` on its selector returns the -/// node itself. +/// The ungated walk returns the node's descendants and not the node +/// itself, so the node's kind is read from its children's labels: elements +/// make an array, members an object. A node with no descendants, an empty +/// container or a scalar, yields no rows and so no value; `->` on its +/// selector returns the node itself. A gated walk also returns each node +/// its gates reach, at depth `0`, with that node's descendants beneath it. /// /// Past its budget, a walk over a document with a sealed root returns the /// root entry alone, at depth `0`, instead of a subtree it would be /// cheaper to open whole: the client opens the document and takes the -/// walked path over it. Opening such rows needs that path and which walk -/// it keyed, attached with [`below`](Self::below) for `json_tree_subtree` -/// or [`at`](Self::at) for `json_tree_subtree_at`, so that the rows open -/// to the same value whichever form the server returned; without a path -/// the opening fails with [`Error::OpenedType`](crate::Error::OpenedType). -#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(transparent)] +/// walked path over it, to the same value the walked rows open to. Whether +/// the path names the node walked below (`json_tree_subtree`) or the nodes +/// reached (`json_tree_subtree_at`) is the key's to say, by whether it has +/// gates, so no caller attaches it. +#[derive(Clone, Debug, PartialEq, Eq)] pub struct JsonSubtree { nodes: Vec, - #[serde(skip)] - path: Option, - #[serde(skip)] + path: JsonPath, below: bool, } impl JsonSubtree { - /// The rows, as returned. - pub fn new(nodes: Vec) -> Self { - Self { - nodes, - path: None, - below: false, - } - } - - /// The rows of `json_tree_subtree_at`, with the whole path its key was - /// derived from: what the opening takes over the whole document when - /// the rows are the sealed root the server fell back to. They open as - /// the walked rows do, to one value per node the path reaches. - pub fn at(self, path: JsonPath) -> Self { - Self { - path: Some(path), - below: false, - ..self - } - } - - /// The rows of `json_tree_subtree`, with the path of the node whose - /// token keyed the walk. They open as the walked rows do, which are - /// that node's descendants: to the node's value, or, as a - /// [`Vec`], to its children's values. Attach the path this way and not - /// with [`at`](Self::at), which would open the fallback to the node - /// itself where the walk opens to its children. - pub fn below(self, path: JsonPath) -> Self { - Self { - path: Some(path), - below: true, - ..self - } + /// The rows of a walk keyed from `path`: below the path's node when + /// `below` (`json_tree_subtree`), at the nodes it reaches otherwise. + pub(crate) fn keyed(nodes: Vec, path: JsonPath, below: bool) -> Self { + Self { nodes, path, below } } /// The rows. @@ -556,10 +615,9 @@ impl JsonSubtree { &self.nodes } - /// The path attached with [`at`](Self::at) or [`below`](Self::below), - /// if any. - pub fn path(&self) -> Option<&JsonPath> { - self.path.as_ref() + /// The path the walk was keyed from. + pub fn path(&self) -> &JsonPath { + &self.path } /// Whether the rows are the sealed root the server returned in place @@ -589,102 +647,74 @@ where // Wire form // ============================================================================= -/// Lowercase hex, as the SQL decodes with `decode(x, 'hex')`. -pub(crate) fn hex(bytes: &[u8]) -> String { - const DIGITS: &[u8; 16] = b"0123456789abcdef"; - let mut out = String::with_capacity(bytes.len() * 2); - for &b in bytes { - out.push(DIGITS[usize::from(b >> 4)] as char); - out.push(DIGITS[usize::from(b & 0xf)] as char); - } - out -} - -/// The inverse of [`hex`]; either case. -pub(crate) fn unhex(text: &str) -> Result, WireError> { - fn nibble(c: u8) -> Result { - match c { - b'0'..=b'9' => Ok(c - b'0'), - b'a'..=b'f' => Ok(c - b'a' + 10), - b'A'..=b'F' => Ok(c - b'A' + 10), - _ => Err(WireError("not hex")), - } - } - let bytes = text.as_bytes(); - if !bytes.len().is_multiple_of(2) { - return Err(WireError("odd hex length")); - } - bytes - .chunks_exact(2) - .map(|pair| Ok(nibble(pair[0])? << 4 | nibble(pair[1])?)) - .collect() -} - -fn unhex16(text: &str) -> Result { - Tag::try_from(unhex(text)?).map_err(|_| WireError("not 16 bytes")) -} - -fn base64(bytes: &[u8]) -> String { +/// Standard base64 with padding, as PostgreSQL's `encode(x, 'base64')` +/// writes it and `decode(x, 'base64')` reads it (which refuses the unpadded +/// form): every field the SQL compares, derives or decodes. +pub(crate) fn base64(bytes: &[u8]) -> String { use base64ct::{Base64, Encoding}; Base64::encode_string(bytes) } -fn unbase64(text: &str) -> Result, WireError> { +/// The inverse of [`base64`]: padded, canonical. +pub(crate) fn unbase64(text: &str) -> Result, WireError> { use base64ct::{Base64, Encoding}; Base64::decode_vec(text).map_err(|_| WireError("not base64")) } +fn unbase64_16(text: &str) -> Result { + Tag::try_from(unbase64(text)?).map_err(|_| WireError("not 16 bytes")) +} + +fn unbase64_8(text: &str) -> Result { + ParentTag::try_from(unbase64(text)?).map_err(|_| WireError("not 8 bytes")) +} + +/// RFC 1924 base85: the ciphertext, which nothing in SQL decodes. Five +/// characters per four bytes; the alphabet has no `"` or `\`. +fn base85(bytes: &[u8]) -> String { + base85_simd::encode(bytes) +} + +fn unbase85(text: &str) -> Result, WireError> { + base85_simd::decode(text).map_err(|_| WireError("not base85")) +} + /// A wire field that does not decode. Names the field's fault and never its /// text. #[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] #[error("a JSON index wire field does not decode: {0}")] pub struct WireError(&'static str); -/// `h`, `sv` and `q`, as stored. `q` is written from the entries and not -/// read back: a document deserialized from the wire derives its projection -/// from `sv` again, so the field is accepted and ignored on the way in. +/// `h`, `sv` and `q`, as stored. All three are required: the projection +/// `q` is the only place the value selectors live. #[derive(Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct DocumentWire { h: String, sv: Vec, - #[serde(default, skip_serializing_if = "Option::is_none")] - q: Option, + q: query::GroupWire, } -/// One entry, as stored: tags in hex, the ciphertext in base64. +/// One entry, as stored: the tags and the ordering term in base64, the +/// ciphertext in base85. #[derive(Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct EntryWire { s: String, - vs: String, c: String, #[serde(default, skip_serializing_if = "Option::is_none")] - o: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] p: Option, #[serde(default, skip_serializing_if = "Option::is_none")] - w: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - n: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - g: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] op: Option, } impl From<&JsonEntry> for EntryWire { fn from(e: &JsonEntry) -> Self { Self { - s: hex(&e.s), - vs: hex(&e.vs), - c: base64(&e.c), - o: e.o, - p: e.p.as_ref().map(|t| hex(t)), - w: e.w.as_ref().map(|t| hex(t)), - n: e.n.as_ref().map(|t| hex(t)), - g: e.g.as_ref().map(|t| hex(t)), - op: e.op.as_deref().map(hex), + s: base64(&e.s), + c: base85(&e.c), + p: e.p.as_ref().map(|t| base64(t)), + op: e.op.as_deref().map(base64), } } } @@ -694,15 +724,10 @@ impl TryFrom for JsonEntry { fn try_from(w: EntryWire) -> Result { Ok(Self { - s: unhex16(&w.s)?, - vs: unhex16(&w.vs)?, - c: unbase64(&w.c)?, - o: w.o, - p: w.p.as_deref().map(unhex16).transpose()?, - w: w.w.as_deref().map(unhex16).transpose()?, - n: w.n.as_deref().map(unhex16).transpose()?, - g: w.g.as_deref().map(unhex16).transpose()?, - op: w.op.as_deref().map(unhex).transpose()?, + s: unbase64_16(&w.s)?, + c: unbase85(&w.c)?, + p: w.p.as_deref().map(unbase64_8).transpose()?, + op: w.op.as_deref().map(unbase64).transpose()?, }) } } @@ -712,7 +737,7 @@ impl From for DocumentWire { Self { h: base64(&doc.header.to_bytes()), sv: doc.entries.iter().map(EntryWire::from).collect(), - q: Some(query::GroupWire::from(&doc.projection())), + q: query::GroupWire::from(&doc.projection), } } } @@ -727,7 +752,12 @@ impl TryFrom for JsonDocument { w.sv.into_iter() .map(JsonEntry::try_from) .collect::>()?; - Ok(Self { header, entries }) + let projection = JsonProjection::try_from(w.q)?; + Ok(Self { + header, + entries, + projection, + }) } } @@ -813,9 +843,8 @@ mod tests { assert_eq!(header.iv(), &[9; 16]); assert_eq!(header.tag(), &[1, 2, 3]); assert!(!header.sealed_root()); - assert_eq!(header.options(), JsonOptions { sealed_root: false }); let sealed = JsonHeader::new(Uuid::from_u128(7), [9; 16], vec![], true); - assert_eq!(sealed.options(), JsonOptions { sealed_root: true }); + assert!(sealed.sealed_root()); } #[test] @@ -851,84 +880,39 @@ mod tests { ); } - /// A chained entry that unwraps to `child` under `parent` at ordinal `o`, - /// as whoever holds `parent` can write one. - fn chained(iv: &[u8], parent: &Tag, o: u32, child: &Tag, c: Vec) -> JsonEntry { + /// An entry at ordinal `o` under `parent`, as whoever holds `parent` can + /// write one: its parent tag is public given the token. + fn chained(iv: &[u8], parent: &Tag, o: u32) -> JsonEntry { JsonEntry { s: [0; 16], - vs: [0; 16], - o: Some(o), + c: o.to_le_bytes().to_vec(), p: Some(derive::parent_tag(iv, parent, o)), - w: Some(derive::xor(child, &derive::mask(iv, parent, &c))), - n: None, - g: None, op: None, - c, } } - fn root() -> JsonEntry { - JsonEntry { + #[test] + fn a_walk_ends_on_a_chain_deeper_than_any_sealed_document() { + let iv = [9; 16]; + let token = [1; 16]; + // A chain longer than any document the index seals: the first child + // of each link's token, a thousand times. + let mut entries = vec![JsonEntry { s: [0; 16], - vs: [0; 16], c: vec![0], - o: None, p: None, - w: None, - n: None, - g: None, op: None, + }]; + let mut parent = token; + for _ in 0..1000 { + entries.push(chained(&iv, &parent, 0)); + parent = derive::child_token(&parent, &[0; 16], 0); } - } - - fn document(entries: Vec) -> JsonDocument { - JsonDocument::new( - JsonHeader::new(Uuid::from_u128(7), [9; 16], vec![], true), + let chain = JsonDocument::new( + JsonHeader::new(Uuid::from_u128(7), iv, vec![], true), entries, - ) - } - - #[test] - fn an_ordinal_at_the_top_of_its_range_is_an_entry_like_any_other() { - let token = [1; 16]; - let mut entry = chained(&[9; 16], &token, u32::MAX, &[2; 16], vec![3; 20]); - let doc = document(vec![root(), entry.clone()]); - let children = doc.children(&token); - assert_eq!(children.len(), 1, "found at its own ordinal, no fan-out"); - assert_eq!(children[0].2, [2; 16]); - assert!(doc.children(&[7; 16]).is_empty()); - entry.p = Some([0; 16]); - assert!(document(vec![root(), entry]).children(&token).is_empty()); - } - - #[test] - fn a_walk_ends_on_tags_that_do_not_form_a_tree() { - let iv = [9; 16]; - let token = [1; 16]; - // Two children of the token that each unwrap to the token itself: - // a cycle that doubles at every level. - let cycle = document(vec![ - root(), - chained(&iv, &token, 0, &token, vec![4; 20]), - chained(&iv, &token, 1, &token, vec![5; 20]), - ]); - assert_eq!( - cycle.subtree(&token).len(), - cycle.entries().len(), - "no more entries than the document holds" + JsonProjection::default(), ); - - // A chain longer than any document the index seals. - let mut entries = vec![root()]; - let mut parent = token; - for i in 0..1000u32 { - let mut child = [0; 16]; - child[..4].copy_from_slice(&i.to_le_bytes()); - child[15] = 0xee; - entries.push(chained(&iv, &parent, 0, &child, i.to_le_bytes().to_vec())); - parent = child; - } - let chain = document(entries); let walked = chain.subtree(&token); assert_eq!(walked.len(), derive::MAX_DEPTH, "cut at the deepest level"); assert_eq!( @@ -938,10 +922,35 @@ mod tests { } #[test] - fn hex_round_trips_and_refuses_what_is_not_hex() { - assert_eq!(hex(&[0, 15, 255]), "000fff"); - assert_eq!(unhex("000FfF").unwrap(), vec![0, 15, 255]); - assert!(unhex("abc").is_err()); - assert!(unhex("zz").is_err()); + fn the_encodings_round_trip_and_refuse_what_they_are_not() { + assert_eq!(base64(&[0, 15, 255]), "AA//"); + assert_eq!(unbase64("AA//").unwrap(), vec![0, 15, 255]); + assert_eq!(base64(&[1; 16]).len(), 24, "a sixteen-byte tag, padded"); + assert_eq!( + base64(&[1; 8]).len(), + 12, + "an eight-byte parent tag, padded" + ); + assert!( + unbase64("AA/").is_err(), + "unpadded, which the SQL refuses too" + ); + assert!(unbase64("AA//=").is_err()); + assert!(unbase64("zz").is_err()); + assert!(unbase64_16("AA//").is_err()); + assert!(unbase64_8("AA//").is_err()); + assert_eq!(unbase64_8(&base64(&[7; 8])).unwrap(), [7; 8]); + + assert_eq!(base85(&[0, 15, 255]), "01y8"); + assert_eq!(unbase85("01y8").unwrap(), vec![0, 15, 255]); + let bytes: Vec = (0..=255).collect(); + assert_eq!(unbase85(&base85(&bytes)).unwrap(), bytes); + assert_eq!(base85(&bytes).len(), 320, "five characters per four bytes"); + assert!( + !base85(&bytes).contains(['"', '\\']), + "unescaped in a JSON string" + ); + assert!(unbase85("\"").is_err()); + assert!(unbase85("z").is_err(), "a one-character tail is no byte"); } } diff --git a/packages/stack-encrypt/src/json/path.rs b/packages/stack-encrypt/src/json/path.rs index 87ce2c960..6f42bfbaf 100644 --- a/packages/stack-encrypt/src/json/path.rs +++ b/packages/stack-encrypt/src/json/path.rs @@ -17,7 +17,7 @@ use std::fmt; use std::str::FromStr; /// One step of a [`JsonPath`]. -#[derive(Clone, Debug, PartialEq, Eq, Hash)] +#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] pub enum Segment { /// An object member, by key. Key(String), @@ -28,7 +28,7 @@ pub enum Segment { } /// A path into a JSON document, from the root (`$`) down. -#[derive(Clone, Debug, PartialEq, Eq, Hash, Default)] +#[derive(Clone, Debug, PartialEq, Eq, Hash, Default, PartialOrd, Ord)] pub struct JsonPath { segments: Vec, } @@ -147,11 +147,52 @@ impl JsonPath { &self.segments } + /// Whether some step is an array position (`[2]`), which a selector + /// reads as the wildcard. + pub(crate) fn names_a_position(&self) -> bool { + self.segments.iter().any(|s| matches!(s, Segment::Index(_))) + } + /// Whether this path names one node: no step is the wildcard. pub fn is_positional(&self) -> bool { !self.segments.iter().any(|s| matches!(s, Segment::Wildcard)) } + /// Whether this path, read as a pattern, covers `path`: the same + /// steps, except that a wildcard here covers any array step there, a + /// position or the wildcard. A position here covers that position + /// alone, not the wildcard, since the wildcard names elements the + /// position does not. How the index's ordered paths + /// ([`JsonOptions::ordered`](super::JsonOptions::ordered)) are matched + /// against a node's path when a document is sealed, and against a + /// query's path when an ordering is asked. + /// + /// ``` + /// use stack_encrypt::json::JsonPath; + /// + /// let wild = JsonPath::parse("$.orders[*].total").unwrap(); + /// assert!(wild.covers(&JsonPath::parse("$.orders[3].total").unwrap())); + /// assert!(wild.covers(&JsonPath::parse("$.orders[*].total").unwrap())); + /// let third = JsonPath::parse("$.orders[3].total").unwrap(); + /// assert!(third.covers(&JsonPath::parse("$.orders[3].total").unwrap())); + /// assert!(!third.covers(&JsonPath::parse("$.orders[2].total").unwrap())); + /// assert!(!third.covers(&JsonPath::parse("$.orders[*].total").unwrap())); + /// assert!(!wild.covers(&JsonPath::parse("$.orders[*]").unwrap())); + /// ``` + pub fn covers(&self, path: &JsonPath) -> bool { + self.segments.len() == path.segments.len() + && self + .segments + .iter() + .zip(&path.segments) + .all(|(pattern, step)| match (pattern, step) { + (Segment::Key(a), Segment::Key(b)) => a == b, + (Segment::Wildcard, Segment::Index(_) | Segment::Wildcard) => true, + (Segment::Index(a), Segment::Index(b)) => a == b, + _ => false, + }) + } + /// The values at this path in `value`, in document order: at most one /// for a positional path, one per element a wildcard reaches, none when /// the path names nothing. A key step into an array, or a position @@ -186,21 +227,6 @@ impl JsonPath { Self { segments } } - /// The longest prefix with no wildcard, and whether anything was cut. - pub(crate) fn anchor(&self) -> (Self, bool) { - let end = self - .segments - .iter() - .position(|s| matches!(s, Segment::Wildcard)) - .unwrap_or(self.segments.len()); - ( - Self { - segments: self.segments[..end].to_vec(), - }, - end < self.segments.len(), - ) - } - /// The canonical bytes a selector is derived over: every array step, /// positional or wildcard, is the one array tag, so every element of an /// array shares its parent's selector. A key is length-prefixed, so two @@ -457,14 +483,17 @@ mod tests { ); } + /// `covers` is public and matches positions though an ordered path may + /// not name one: a position covers that position alone. #[test] - fn the_anchor_is_the_prefix_before_the_first_wildcard() { - let (anchor, cut) = JsonPath::parse("$.a[*].x").unwrap().anchor(); - assert_eq!(anchor, JsonPath::parse("$.a").unwrap()); - assert!(cut); - let (anchor, cut) = JsonPath::parse("$.a[1].x").unwrap().anchor(); - assert_eq!(anchor.to_string(), "$.a[1].x"); - assert!(!cut); + fn a_position_covers_its_own_position_only() { + let path = |text: &str| JsonPath::parse(text).unwrap(); + let third = path("$.orders[3].total"); + assert!(third.covers(&path("$.orders[3].total"))); + assert!(!third.covers(&path("$.orders[2].total"))); + assert!(!third.covers(&path("$.orders[*].total"))); + assert!(path("$.orders[*].total").covers(&third)); + assert!(!path("$.orders[3]").covers(&third), "the same steps only"); } #[test] diff --git a/packages/stack-encrypt/src/json/plain.rs b/packages/stack-encrypt/src/json/plain.rs new file mode 100644 index 000000000..9795ce623 --- /dev/null +++ b/packages/stack-encrypt/src/json/plain.rs @@ -0,0 +1,215 @@ +//! JSON plaintext under vitaminc's controls. +//! +//! A JSON document's plaintext lives in `Protected` here as it does on the +//! scalar paths: a node's plaintext is built into a `Protected>` +//! before it is sealed, and an opened entry's bytes stay in the +//! `Protected>` the cipher returns, read in place while they are +//! parsed. A parsed value is a [`PlainJson`] inside a `Protected`, which +//! wipes it on drop. It leaves the wrapper only where the caller receives +//! it: the value of the document, the node or the subtree it asked for. +use serde_json::{Map, Number, Value}; +use vitaminc_protected::{Controlled, Protected}; +use zeroize::Zeroize; + +/// A JSON value's plaintext, for holding in a `Protected`. +/// +/// `serde_json::Value` has no `Zeroize`, so this newtype gives it one: +/// [`zeroize`](Zeroize::zeroize) wipes every string in the tree, values +/// and object keys alike (a key is taken out of its map by ownership to +/// reach it), sets every number to zero and every boolean to `false`, and +/// leaves `null`. Strings are wiped through `zeroize`, which writes +/// volatilely over the whole buffer. Numbers and booleans sit inline in +/// their parent's array or map node and are overwritten in place; with +/// unsafe code denied in this crate there is no volatile write for them, +/// so that overwrite is best-effort against the optimiser. +/// +/// It has no `Debug`, `Clone`, `PartialEq`, `Default` or `Deref`, so +/// nothing reads it but through its `Protected`. +pub(crate) struct PlainJson(Value); + +impl PlainJson { + /// Wrap `value`, which the caller has just built or parsed. + pub(crate) fn protect(value: Value) -> Protected { + Protected::new(Self(value)) + } + + /// A value just built, to be wrapped by the caller. + pub(crate) fn from_value(value: Value) -> Self { + Self(value) + } + + /// Parse JSON text into a value to be wrapped by the caller. + pub(crate) fn parse(bytes: &[u8]) -> Option { + serde_json::from_slice(bytes).ok().map(Self) + } + + /// The value, for the caller that asked for it. Only at that + /// boundary, after `risky_unwrap` on the `Protected` that held it. + pub(crate) fn into_value(self) -> Value { + self.0 + } + + /// The value, read in place. + pub(crate) fn value(&self) -> &Value { + &self.0 + } + + /// The value, changed in place, inside its `Protected`. + pub(crate) fn value_mut(&mut self) -> &mut Value { + &mut self.0 + } +} + +impl Zeroize for PlainJson { + fn zeroize(&mut self) { + wipe(&mut self.0); + self.0 = Value::Null; + } +} + +/// Overwrite every scalar and key in `value` in place, then empty its +/// containers. +fn wipe(value: &mut Value) { + match value { + Value::String(text) => text.zeroize(), + Value::Number(number) => *number = Number::from(0u8), + Value::Bool(flag) => *flag = false, + Value::Null => {} + Value::Array(items) => { + items.iter_mut().for_each(wipe); + items.clear(); + } + Value::Object(members) => { + members.values_mut().for_each(wipe); + for (mut key, _) in std::mem::take(members) { + key.zeroize(); + } + } + } + // Keep the in-place overwrites above from being dropped as dead stores + // before the memory is freed. + let _ = std::hint::black_box(&*value); +} + +/// Read a `T` out of `source` without `source` leaving its `Protected`: +/// `read` borrows the inner value through vitaminc (`zip_ref`), and what it +/// returns is wrapped as it is made. `None` when `read` refuses. +pub(crate) fn read_protected( + source: &Protected, + read: impl FnOnce(&S) -> Option, +) -> Option> +where + S: Zeroize, + T: Zeroize, +{ + Protected::new(None::) + .zip_ref(source, |_, inner: &S| read(inner)) + .transpose() +} + +/// An object under construction inside a `Protected`, so a member already +/// moved in is wiped if the build fails before the object is handed over. +pub(crate) fn protected_object() -> Protected { + PlainJson::protect(Value::Object(Map::new())) +} + +/// An array under construction inside a `Protected`; see +/// [`protected_object`]. +pub(crate) fn protected_array(capacity: usize) -> Protected { + PlainJson::protect(Value::Array(Vec::with_capacity(capacity))) +} + +/// Move `member` into the object `into` under `key`. Both arrive wrapped +/// and are unwrapped only to be moved into `into`'s own `Protected`. +pub(crate) fn insert_member( + into: &mut Protected, + key: Protected, + member: Protected, +) { + let mut entry = Some((key.risky_unwrap(), member.risky_unwrap().into_value())); + into.update(|object| { + if let (Value::Object(members), Some((key, member))) = (object.value_mut(), entry.take()) { + let _ = members.insert(key, member); + } + }); +} + +/// Move `element` onto the end of the array `into`; see [`insert_member`]. +pub(crate) fn push_element(into: &mut Protected, element: Protected) { + let mut element = Some(element.risky_unwrap().into_value()); + into.update(|array| { + if let (Value::Array(items), Some(element)) = (array.value_mut(), element.take()) { + items.push(element); + } + }); +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn zeroize_leaves_null_whatever_the_tree_held() { + let mut plain = PlainJson(json!({ + "secret": "hunter2", + "nested": {"pin": 1234, "ok": true, "list": ["a", 2.5, null]}, + })); + plain.zeroize(); + assert_eq!(plain.value(), &Value::Null); + // And through `Protected`, whose drop runs the same `zeroize`. + let mut protected = PlainJson::protect(json!(["x"])); + protected.zeroize(); + assert_eq!(protected.risky_ref().value(), &Value::Null); + } + + #[test] + fn a_wiped_string_is_zero_before_it_is_freed() { + // `wipe` zeroizes strings in place; observe it on a value the test + // still owns. + let mut value = json!(["hunter2", {"k": "v", "n": 7, "b": true}]); + if let Value::Array(items) = &mut value { + wipe(&mut items[0]); + assert_eq!(items[0], json!(""), "a zeroized String is emptied"); + if let Value::Object(members) = &mut items[1] { + let mut number = members.get("n").cloned().unwrap(); + wipe(&mut number); + assert_eq!(number, json!(0)); + let mut flag = members.get("b").cloned().unwrap(); + wipe(&mut flag); + assert_eq!(flag, json!(false)); + } + } + wipe(&mut value); + assert_eq!(value, json!([]), "containers are emptied"); + let mut object = json!({"key": "value"}); + wipe(&mut object); + assert_eq!(object, json!({}), "keys are taken out and wiped"); + } + + #[test] + fn read_protected_wraps_what_it_reads_and_refuses_with_none() { + let bytes = Protected::new(b"\"hello\"".to_vec()); + let parsed = read_protected(&bytes, |b: &Vec| { + serde_json::from_slice(b).ok().map(PlainJson) + }) + .unwrap(); + assert_eq!(parsed.risky_ref().value(), &json!("hello")); + assert!(read_protected(&bytes, |_: &Vec| None::).is_none()); + // The source is only borrowed: it is still whole. + assert_eq!(bytes.risky_ref(), b"\"hello\""); + } + + #[test] + fn members_and_elements_move_in_under_protection() { + let mut object = protected_object(); + insert_member( + &mut object, + Protected::new("k".to_owned()), + PlainJson::protect(json!(1)), + ); + let mut array = protected_array(1); + push_element(&mut array, object); + assert_eq!(array.risky_unwrap().into_value(), json!([{"k": 1}])); + } +} diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index 351065929..e515723ce 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -2,9 +2,9 @@ //! //! A containment needle is the **element-nested projection** of the //! needle's terms: the selector pairs `{s, vs}` of every node, grouped by -//! the array element they sit in. The server projects a stored document the -//! same way from its group tags, and native `jsonb @>` over the two -//! projections is exact containment: each needle element has to fit inside +//! the array element they sit in. A stored document carries its own +//! projection, built the same way, as `q`, which the server reads, and +//! native `jsonb @>` over the two projections is exact containment: each needle element has to fit inside //! one stored element, which is PostgreSQL's own rule for arrays. The //! design's "The preferred form" records why no re-check is needed. use std::fmt; @@ -14,7 +14,7 @@ use serde_json::Value; use super::derive::{JsonTerms, TermNode}; use super::path::JsonPath; -use super::{hex, JsonDocument, JsonEntry, JsonOptions, Tag}; +use super::{base64, JsonDocument, JsonNode, JsonOptions, JsonSubtree, Tag}; use crate::op::{ Answers, AtPath, Between, ContainedBy, Contains, DefaultIndex, Eq, Gt, Gte, Input, LabelledTarget, Lt, Lte, One, Pair, Pathed, QueryTerm, QueryToken, Supports, @@ -32,8 +32,8 @@ struct Pair16 { /// The element-nested projection of a document: the selector pairs that /// sit outside any array element, and one nested projection per array -/// element that is a container. What the server computes from a stored -/// document's group tags (`json_tree_query`), and what a containment +/// element that is a container. What a stored document carries as `q`, +/// which the server reads (`json_tree_query`), and what a containment /// needle is. #[derive(Clone, Debug, PartialEq, Eq, Default)] pub struct JsonProjection { @@ -106,8 +106,9 @@ impl JsonQuery { } /// The group of the nodes that sit directly in `parent` (`None` is the top -/// level), with the container elements in it nested as groups. -fn group_of(nodes: &[TermNode], parent: Option) -> JsonProjection { +/// level), with the container elements in it nested as groups. What a +/// needle is, and what a document writes to `q` when it is sealed. +pub(crate) fn group_of(nodes: &[TermNode], parent: Option) -> JsonProjection { let belongs = |index: usize, node: &TermNode| { if node.container_element { Some(index) @@ -131,34 +132,6 @@ fn group_of(nodes: &[TermNode], parent: Option) -> JsonProjection { } } -impl JsonDocument { - /// The stored document's projection, from its group tags: what a - /// [`JsonQuery`] is matched against. The document writes it to the wire - /// as `q`, beside `sv`, and the SQL's `json_tree_query` reads it from - /// there; a document deserialized from the wire derives it again from - /// the entries, so a stored `q` is never trusted over them. - pub fn projection(&self) -> JsonProjection { - fn belongs(e: &JsonEntry) -> Option<&Tag> { - e.n.as_ref().or(e.g.as_ref()) - } - fn group(entries: &[JsonEntry], parent: Option<&Tag>) -> JsonProjection { - JsonProjection { - sv: entries - .iter() - .filter(|e| belongs(e) == parent) - .map(|e| Pair16 { s: e.s, vs: e.vs }) - .collect(), - el: entries - .iter() - .filter(|e| e.n.is_some() && e.g.as_ref() == parent) - .map(|e| group(entries, e.n.as_ref())) - .collect(), - } - } - group(&self.entries, None) - } -} - #[derive(Clone, Serialize, Deserialize)] struct PairWire { s: String, @@ -187,8 +160,8 @@ impl From<&JsonProjection> for GroupWire { .sv .iter() .map(|p| PairWire { - s: hex(&p.s), - vs: hex(&p.vs), + s: base64(&p.s), + vs: base64(&p.vs), }) .collect(), el: group.el.iter().map(GroupWire::from).collect(), @@ -196,11 +169,25 @@ impl From<&JsonProjection> for GroupWire { } } +/// A projection serializes as a stored `q` and a needle do: `{"sv": [{"s", +/// "vs"}, …], "el": [group, …]}`. +impl Serialize for JsonProjection { + fn serialize(&self, s: S) -> Result { + GroupWire::from(self).serialize(s) + } +} + +impl<'de> Deserialize<'de> for JsonProjection { + fn deserialize>(d: D) -> Result { + JsonProjection::try_from(GroupWire::deserialize(d)?).map_err(serde::de::Error::custom) + } +} + impl TryFrom for JsonProjection { type Error = super::WireError; fn try_from(w: GroupWire) -> Result { let tag = |text: &str| { - Tag::try_from(super::unhex(text)?).map_err(|_| super::WireError("not 16 bytes")) + Tag::try_from(super::unbase64(text)?).map_err(|_| super::WireError("not 16 bytes")) }; Ok(Self { sv: w @@ -288,8 +275,8 @@ struct LeafWire { impl Serialize for JsonLeafQuery { fn serialize(&self, s: S) -> Result { LeafWire { - s: hex(&self.s), - op: hex(&self.op), + s: base64(&self.s), + op: base64(&self.op), } .serialize(s) } @@ -298,17 +285,17 @@ impl Serialize for JsonLeafQuery { impl<'de> Deserialize<'de> for JsonLeafQuery { fn deserialize>(d: D) -> Result { let wire = LeafWire::deserialize(d)?; - let s = super::unhex(&wire.s) + let s = super::unbase64(&wire.s) .ok() .and_then(|b| Tag::try_from(b).ok()) .ok_or_else(|| serde::de::Error::custom("not a selector"))?; - let op = super::unhex(&wire.op).map_err(serde::de::Error::custom)?; + let op = super::unbase64(&wire.op).map_err(serde::de::Error::custom)?; Ok(Self { s, op }) } } /// A path selector: what `->` and the path functions take. Prints and -/// serializes as the hex the stored entry carries in `s`. +/// serializes as the base64 the stored entry carries in `s`. #[derive(Clone, Copy, PartialEq, Eq, Hash)] pub struct JsonSelector(pub(crate) Tag); @@ -321,35 +308,64 @@ impl JsonSelector { impl fmt::Display for JsonSelector { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str(&hex(&self.0)) + f.write_str(&base64(&self.0)) } } impl fmt::Debug for JsonSelector { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "JsonSelector({})", hex(&self.0)) + write!(f, "JsonSelector({})", base64(&self.0)) } } impl Serialize for JsonSelector { fn serialize(&self, s: S) -> Result { - s.serialize_str(&hex(&self.0)) + s.serialize_str(&base64(&self.0)) } } -/// The key of a subtree walk: the token of a positional node, and, when -/// the walked path continues with a wildcard below it, the selector of the -/// whole path the walk gates on. Serializes as `{"t": "…", "s": "…"}`. +/// One gate of a subtree walk: what the entry at one depth below the +/// walk's token must be for the walk to descend through it. #[derive(Clone, Copy, PartialEq, Eq)] +pub(crate) enum Gate { + /// The selector of the path down to that depth: a key step, or a + /// wildcard, whose selector every element carries. + Selector(Tag), + /// An array position: the one ordinal probed at that depth, and the + /// selector of an element there (the wildcard's), which the entry at + /// that ordinal must carry. The ordinal alone would admit an object + /// member that ranks there in a row whose node at the path is an + /// object. + Slot(u32, Tag), +} + +impl Gate { + /// The gate as `json_tree_subtree_at` takes it: the selector's base64, or + /// `#n:` and the element selector's base64 for a position. + fn text(&self) -> String { + match self { + Gate::Selector(s) => base64(s), + Gate::Slot(n, s) => format!("#{n}:{}", base64(s)), + } + } +} + +/// The key of a subtree walk: the token of the deepest node on the path +/// the client can derive, one gate per step below it, and the path it was +/// derived from, which [`open`](Self::open) attaches to the rows the walk +/// returns. Serializes as `{"t": "…", "g": ["…", "#0:…", …]}`, `g` omitted +/// when empty; the path is not part of the wire. +#[derive(Clone, PartialEq, Eq)] pub struct JsonSubtreeKey { pub(crate) token: Tag, - pub(crate) selector: Option, + pub(crate) gates: Vec, + pub(crate) path: JsonPath, } impl JsonSubtreeKey { - /// The token, as the hex the walk takes. + /// The token, as the base64 the walk takes. pub fn token(&self) -> String { - hex(&self.token) + base64(&self.token) } /// The token's bytes. A token is a capability over the structure of @@ -359,10 +375,30 @@ impl JsonSubtreeKey { &self.token } - /// The selector the walk gates on, when the path has a wildcard below - /// the token's node. - pub fn selector(&self) -> Option { - self.selector.map(JsonSelector) + /// The gates the walk descends through below the token, one per step, + /// as `json_tree_subtree_at` takes them: a selector's base64, or + /// `#n:` for an array position, the selector being the array + /// elements' own. Empty when the token names the walked node itself, + /// which `json_tree_subtree` takes. + pub fn gates(&self) -> Vec { + self.gates.iter().map(Gate::text).collect() + } + + /// The path the key was derived from. + pub fn path(&self) -> &JsonPath { + &self.path + } + + /// The rows the walk this key keyed returned, ready to open: the only + /// way to build a [`JsonSubtree`], so the rows always carry the path + /// a sealed-root fallback is opened at. With no gates the rows came + /// from `json_tree_subtree(doc, token)` and are the descendants of the + /// path's node; with gates they came from `json_tree_subtree_at(doc, + /// token, gates)` and are the nodes the path reaches, with theirs. The + /// key knows which, so the fallback opens to what the walked rows + /// open to either way. + pub fn open(&self, rows: Vec) -> JsonSubtree { + JsonSubtree::keyed(rows, self.path.clone(), self.gates.is_empty()) } } @@ -370,7 +406,7 @@ impl JsonSubtreeKey { impl fmt::Debug for JsonSubtreeKey { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("JsonSubtreeKey") - .field("gated", &self.selector.is_some()) + .field("gates", &self.gates.len()) .finish_non_exhaustive() } } @@ -380,12 +416,12 @@ impl Serialize for JsonSubtreeKey { #[derive(Serialize)] struct Wire { t: String, - #[serde(skip_serializing_if = "Option::is_none")] - s: Option, + #[serde(skip_serializing_if = "Vec::is_empty")] + g: Vec, } Wire { - t: hex(&self.token), - s: self.selector.as_ref().map(|t| hex(t)), + t: base64(&self.token), + g: self.gates(), } .serialize(s) } @@ -596,9 +632,26 @@ fn ordered(value: &Value) -> Result<(), Error> { } } +/// Whether the index orders the leaves at `path`: some path in its +/// `ordered` option covers it. A path that names a position is refused +/// first: the comparison extracts by selector, which reads the position as +/// the wildcard, so it would compare the first element at the path rather +/// than the one named. +fn orders(index: &Json, path: &JsonPath) -> Result<(), Error> { + if path.names_a_position() { + Err(TermError::JsonOrderingAtPosition.into()) + } else if index.options.orders(path) { + Ok(()) + } else { + Err(TermError::JsonPathNotOrdered.into()) + } +} + /// The orderings at a path: the path's selector and the value's leaf /// ordering term. Refused, when the query is awaited, for a value that is -/// neither a number nor a string. +/// neither a number nor a string, and for a path the index does not order +/// ([`JsonOptions::ordered`]): no stored leaf there carries a term, so the +/// comparison would match no row. macro_rules! orderings { ($($op:ident),+ $(,)?) => {$( impl Supports, (P, V)> for Json @@ -609,7 +662,8 @@ macro_rules! orderings { type Query = JsonLeafQuery; type Verifier = (); fn check(&self, (path, value): &(P, V)) -> Result<(), Error> { - let _ = parse_path(path.as_ref())?; + let path = parse_path(path.as_ref())?; + orders(self, &path)?; ordered(&value.clone().into()) } fn verifier(&self, _: &(P, V)) {} @@ -643,7 +697,8 @@ where type Query = RangeQuery; type Verifier = (); fn check(&self, (path, (lo, hi)): &(P, (V, V))) -> Result<(), Error> { - let _ = parse_path(path.as_ref())?; + let path = parse_path(path.as_ref())?; + orders(self, &path)?; ordered(&lo.clone().into())?; ordered(&hi.clone().into()) } @@ -676,9 +731,9 @@ where /// plan that declares the index carries them, and a query through it takes /// them from there. impl Json { - /// The index's layout options. - pub fn options(&self) -> JsonOptions { - self.options + /// The index's options: its layout, and the paths it orders. + pub fn options(&self) -> &JsonOptions { + &self.options } } @@ -707,8 +762,8 @@ mod tests { assert_eq!( json, json!({ - "sv": [{"s": "01".repeat(16), "vs": "02".repeat(16)}], - "el": [{"sv": [{"s": "03".repeat(16), "vs": "04".repeat(16)}], "el": []}], + "sv": [{"s": "AQEBAQEBAQEBAQEBAQEBAQ==", "vs": "AgICAgICAgICAgICAgICAg=="}], + "el": [{"sv": [{"s": "AwMDAwMDAwMDAwMDAwMDAw==", "vs": "BAQEBAQEBAQEBAQEBAQEBA=="}], "el": []}], }) ); let back: JsonQuery = serde_json::from_value(json).unwrap(); @@ -716,25 +771,42 @@ mod tests { } #[test] - fn a_selector_prints_as_its_hex_and_a_subtree_key_hides_its_token() { + fn a_selector_prints_as_its_base64_and_a_subtree_key_hides_its_token() { let selector = JsonSelector([0xab; 16]); - assert_eq!(selector.to_string(), "ab".repeat(16)); + assert_eq!(selector.to_string(), "q6urq6urq6urq6urq6urqw=="); assert_eq!( format!("{selector:?}"), - format!("JsonSelector({})", "ab".repeat(16)) + format!("JsonSelector({})", "q6urq6urq6urq6urq6urqw==") ); let key = JsonSubtreeKey { token: [0xcd; 16], - selector: Some([0xab; 16]), + gates: vec![Gate::Selector([0xab; 16]), Gate::Slot(3, [0xab; 16])], + path: JsonPath::parse("$.a.b[3]").unwrap(), }; - assert_eq!(key.token(), "cd".repeat(16)); + assert_eq!(key.token(), "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="); assert_eq!(key.token_bytes(), &[0xcd; 16]); - assert_eq!(key.selector(), Some(selector)); + assert_eq!( + key.gates(), + ["q6urq6urq6urq6urq6urqw==", "#3:q6urq6urq6urq6urq6urqw=="] + ); + assert_eq!( + serde_json::to_value(&key).unwrap(), + serde_json::json!({"t": "zc3Nzc3Nzc3Nzc3Nzc3NzQ==", "g": ["q6urq6urq6urq6urq6urqw==", "#3:q6urq6urq6urq6urq6urqw=="]}) + ); assert_eq!( format!("{key:?}"), - "JsonSubtreeKey { gated: true, .. }", + "JsonSubtreeKey { gates: 2, .. }", "a token is a capability: it never prints" ); + let bare = JsonSubtreeKey { + token: [0xcd; 16], + gates: Vec::new(), + path: JsonPath::parse("$.a").unwrap(), + }; + assert_eq!( + serde_json::to_value(&bare).unwrap(), + serde_json::json!({"t": "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="}) + ); } #[test] @@ -749,9 +821,52 @@ mod tests { #[test] fn the_index_reports_its_options() { - let open = JsonOptions { sealed_root: false }; - assert_eq!(Json::new(open).options(), open); - assert_eq!(Json::default().options(), JsonOptions::default()); + let open = JsonOptions { + sealed_root: false, + ..JsonOptions::new() + }; + assert_eq!(Json::new(open.clone()).options(), &open); + assert_eq!(Json::default().options(), &JsonOptions::default()); + } + + #[test] + fn a_projection_round_trips_through_its_wire_form() { + let projection = JsonProjection { + sv: vec![Pair16 { + s: [1; 16], + vs: [2; 16], + }], + el: vec![JsonProjection { + sv: vec![Pair16 { + s: [3; 16], + vs: [4; 16], + }], + el: Vec::new(), + }], + }; + let wire = serde_json::to_value(&projection).unwrap(); + assert_eq!( + wire, + serde_json::json!({ + "sv": [{"s": "AQEBAQEBAQEBAQEBAQEBAQ==", "vs": "AgICAgICAgICAgICAgICAg=="}], + "el": [{"sv": [{"s": "AwMDAwMDAwMDAwMDAwMDAw==", "vs": "BAQEBAQEBAQEBAQEBAQEBA=="}], "el": []}], + }) + ); + let back: JsonProjection = serde_json::from_value(wire).unwrap(); + assert_eq!(back, projection); + assert!( + serde_json::from_value::(serde_json::json!({"sv": [], "el": []})) + .unwrap() + .el + .is_empty() + ); + assert!( + serde_json::from_value::( + serde_json::json!({"sv": [{"s": "zz", "vs": "00"}], "el": []}) + ) + .is_err(), + "a pair that is not two tags is refused" + ); } #[test] diff --git a/packages/stack-encrypt/src/plan/chain.rs b/packages/stack-encrypt/src/plan/chain.rs index 21cf04262..2096fc562 100644 --- a/packages/stack-encrypt/src/plan/chain.rs +++ b/packages/stack-encrypt/src/plan/chain.rs @@ -9,12 +9,12 @@ use uuid::Uuid; use super::build::{FieldPlan, FieldsBuilder, IntoLabel, Opens, PlanContext, Runs}; use super::field_ref::FieldRef; -use super::value::{Indexed, Typed, ValueLayout, ValuePlan, ValueShape}; +use super::value::{Indexed, Laid, Typed, ValueLayout, ValuePlan, ValueShape}; use super::PlanError; use crate::op::{Answers, DefaultIndex, Input, LabelledTarget, Operator, QueryToken, Supports}; use crate::target::{ ciphertext, Borrowed, CallerContext, DeclaredContext, DecryptField, Decryptable, EncryptFrom, - Index, Indexes, Pending, PendingFuture, Select, Whole, + Index, Indexes, Layout, Pending, PendingFuture, Select, Whole, }; use crate::{Error, KeysetCipher, Label, LabelError, NonEmpty, StackCipher, StackCipherText}; @@ -777,12 +777,30 @@ where } } +/// A plan laid out by value answers the operators its layout's index +/// supports, through that index, under the plan's options. +impl Chooses for ValuePlan> +where + Op: Operator, + In: Input, + S: 'static, + X: Layout, + X::Index: Supports + Default, + Laid: ValueShape, +{ + type Via = X::Index; + fn choose(&self, call: Option