Repository navigation
Conversation
… to queries on its plan A typed plan answers an operator through its target type's `op::Answers` impl, but the chain required `Typed<T>: ValueShape<S>`: a label that converts into `T`'s context. An EQL v4 type is sealed under its column (`NonEmpty<Identifier>`), so no plan for one could be queried, and eql-bindings had nowhere to implement `Answers` (#1238). `op::LabelledTarget` is the opt-in: a target type says its stored terms were derived under the context a plan's label names. For such a type, `Chooses` asks only `ValueLayout<S>` (the indexes `T` declares), since a query derives through the index and never runs `T`'s encryption. Every other type keeps needing `ValueShape`, so a type sealed under, say, a tenant id cannot be queried under a label that would match nothing. This crate's term targets implement the trait, which keeps every query that compiled before compiling. `Chooses` now resolves the plan itself (`choose`) rather than through `Resolves`, which keeps `ValueShape`: a named index (`.via(..)`) on a plan whose type only opts in would send a term the type does not send. An EQL query value stores the column it queries, so its index needs the context its terms were derived under, and must refuse one it cannot record. `QueryToken`, which only the chain makes, now carries the label and whether the call extended it (`label`, `is_extended`): with `.extend(..)` the terms are derived under the label and the extension, so a term that records the label alone would match nothing. The defaulted `Supports::check_label` refuses such a context, and the chain runs it in `check`, where the label is resolved, before any keyset is loaded or term derived (and again in `prepare`, as for `check`). `Supports::query` stays infallible: the chain calls it only after `check_label` accepted the token's context. Every index here takes any context. Tests: a labelled target is queried under the plan's or the call's label, and its term reads whether the call extended it; a label its index refuses loads no keyset; and a UI case pins that a type with an `Answers` impl but no `LabelledTarget` answers no query on its plan.
…query chain Rust could store an EQL v4 type but query it only through a data plan: no v4 type implemented `op::Answers`, so `cipher.query(op::Eq, v).on(&plan)` did not compile on a plan for `TextEq`, while the Go SDK already had an `Eq` method per equality type and `Matches` with `Verify` on `TextMatch` (#1216). This gives Rust the same operators, with the same bytes (#1238). eql-codegen writes into `v4/answers.rs`, for each producible type with a query twin: `op::LabelledTarget` (it is sealed under the column a two-segment label names), the twin's assembly, and one `op::Answers` impl per operator. The operators come from the type's index keys, the input the data path's `TargetDescriptor::operators` reads: `"eq"` answers `Eq`, `"match"` answers `Matches`. It stops on any other index, or more than one, so making an ordered type producible has to extend the rule. The same rows give `ANSWERS`, which the `typed_queries` test holds to `TargetDescriptor::operators()` for every type. The file sits beside the target table rather than in it: `v4/targets.rs` is built on Stack Encrypt's `dynamic` layer and needs the `targets` feature, while a typed query needs neither, so `answers.rs` needs only `stack-encrypt`. The index each impl names is `TwinIndex<I, Q>`: Stack Encrypt's own index `I` (the same spec and derivation), whose query term is written into the twin `Q` with the column the query's label names, by the encodings the stored values use. It answers only for the twin's own plaintext (`QueryTwin::Plaintext`), so no plan can send one type's twin for another type's value, and its `check_label` refuses a label that names no column and any extension before any keyset is loaded: an extended query would be derived under more than the column the twin stores, and match nothing. Both refusals are a `QueryLabelError` in `Error::Other`, so a caller matches the variant rather than the message. `TextMatch` returns `MatchTwin<Q>`, which serializes as the twin and keeps the query's tokens for `verify` and `target_limit` under `EqlMatch`. `Hmac256` and `BloomFilter` now declare their indexes, so a type holding one declares the index a query is checked against, and `EqlMatch` implements `MatchCase`. Tests: every typed query is the data path's `query_term` bytes and domain (the float values the semantic baseline folds included), with the data path's inputs built as frames of Rust values; `TextMatch`'s `verify` and `target_limit` agree with the data path's; short text, a non-column label and an extended query are refused before any keyset is loaded; and compile-fail doctests for `op::Lt` on `TextEq`, `op::Matches` on `IntegerEq`, `.via(Equality)` and another type's `TwinIndex` on a `TextEq` plan, and `verify` on an exact term. The new module's example runs as a test in the encryption crate.
…nd pin the twin's plaintext tie `op::Answers` and the stack-encrypt CHANGELOG read as if only a type sealed under a context no label names needed `op::LabelledTarget`. Any type queried on a typed plan needs both, whatever its context; both now say so. The cross-type `TwinIndex` compile-fail doctest is rejected by `Resolves`' `ValueShape` bound before the plaintext tie is reached, so it would still pass if `QueryTwin::Plaintext` were removed. A doctest pair now checks the tie alone, through a `Supports<op::Eq, String>` bound: `TwinIndex<Equality, TextEqQuery>` meets it, and `TwinIndex<Equality, IntegerEqQuery>` fails with `<IntegerEqQuery as QueryTwin<Hmac256>>::Plaintext == String`.
feat(eql-bindings): query EQL v4 types through Stack Encrypt's typed query chain
…perator reads
eql_v4.query_text_search required hm, op and bf on every query value. bf
is the bloom filter built from 3-grams, so text shorter than one 3-gram has
none, and `col = 'Al'` or `col < 'Al'` could not be written against an
eql_v4_text_search column. That is why Stack Encrypt could not produce the
type.
In v4 the search query domains (query_text_search and its _ore twin) now
require at least one of their terms. Each extractor over them reads its
term through eql_v4_internal.require_query_term, which raises
invalid_parameter_value naming the domain, the term and the operators that
compare it when the term is absent, so `=` on a value with no hm, `<` with
no op and `@@` with no bf are refused rather than reading NULL and
matching nothing. The extractors stay LANGUAGE sql, so the wrappers still
inline and the eq_term/ord_term/match_term indexes still engage.
v3 is cipherstash-client's format and does not change. The SQL source is
rendered once per version and had no way to say "only in v4", so the
renderer gains version blocks: lines between `-- {{#v3}}`/`-- {{/v3}}`
or `-- {{#v4}}`/`-- {{/v4}}` render for that version only, and the
directive lines for neither. The generator emits them for the two search
query domains, and the helper sits in a v4 block of
src/v3/scalars/functions.sql. release/cipherstash-encrypt.sql and its
uninstaller are byte-identical to the build before this change.
ADR-0002 gains a dated amendment recording the decision.
…s operator TargetResolver::query took a type name, a label and a value, so a resolver had to build the type's whole query twin, every term of it, for every operator. EQL v4's TextSearch needs to emit only the term its operator reads: a match term cannot be made from text shorter than one 3-gram, and `=` or `<` on such text must not be refused for want of one. query_term already knows the answering index (OpSpec::answered_by over the type's indexes). It now passes it through record::query to TargetResolver::query as an &IndexSpec after the type's name. A resolver whose twins carry every term ignores it. eql-bindings' encryption::targets::query takes it the same way and refuses an index the type does not carry (TargetError::NoIndex) before deriving anything. The Go guest forwards it; an index the type lacks can only come from an engine bug, so the guest reports it as internal. BREAKING CHANGE: TargetResolver::query and dynamic::record::query take an &IndexSpec; eql_bindings::encryption::targets::query takes one after the name.
…perator's term TextSearch was unproducible because its query twin carried hm, op and bf on every query, and text shorter than one 3-gram has no bf: `=` or `<` on a two-letter name would have been refused. EQL v4's search query domain now takes any one term, so the twin can carry just the one its operator reads. In v4, TextSearchQuery (and the unproducible TextSearchOreQuery) hold each term as an Option, skipped in the JSON when None. The generator emits one projection per term, TextSearchEqualityQuery, TextSearchOrderQuery and TextSearchMatchQuery, each deriving EncryptFrom<String> over the envelope and its one term and converting into TextSearchQuery. The by-name query dispatch, which now receives the answering index, runs the projection for that index, so a match term is derived only for `matches`. The twin itself carries no derive. The stored TextSearch carries all three terms, the same bytes TextEq, TextOrd and TextMatch store, so text too short for a 3-gram is refused on write with EmptyTermText, exactly as TextMatch refuses it: a stored row with an empty filter would be one no match query finds. TextSearch joins ENCRYPTION_DOMAINS, the regenerated target table marks it producible, and the Go encrypt/eql table follows. The engine answers eq, lt, lte, gt, gte and matches on it. Tests: each index's query carries its term alone and equal to the stored one; "Al" is queried by eq and ope and refused for match; the typed projections serialize to what the dispatch writes; and, against an EQL v4 database, `=`, `<>`, `<`, `>=`, `@@` and the function forms find the plaintexts' rows, a one-term operand is a query_text_search and a termless one is not, and each operator and extractor refuses an operand missing its term with invalid_parameter_value naming the term and never the operand.
…ng one term A directory Entry model holds one TextSearch field. Generated over the deterministic EQL guest, its stored value carries bf, c, hm, i, op and v and opens back; Eq, Lt and Gte return a query with only hm or op, the stored term, for "Alice" and for the two-letter "Al"; Matches returns one with only bf, which Verify checks; and Matches on "Al", like storing "Qz" or "", is refused with ErrTerm without quoting the value. stashgen gets a golden case for the same field over its static engine, which now lists TextSearch with its three indexes, so the generated methods (Eq, Lt, Lte, Gt, Gte, Matches) are pinned and compile against the stub SDK.
…index `IndexSpec::OreBlock` (key `ore_block`) derives ore-rs 0.9.0's 6-bit block ORE under its own key domain, over the order keys the CLLW indexes use: `ore_rs::vitaminc::Bit6BlockOre` for targets of at most ten bytes (every integer to 64 bits, both floats) and `ChainedBlockOre` for text, bytes and the 128-bit integers. The legacy 8-bit scheme is not used. `sem::OreBlockTerm` is the whole ciphertext. It carries a random nonce, so it compares by the scheme's comparator, never by its bytes, and `from_bytes` checks a stored term's structure. A query is the full ciphertext too: EQL v4's comparator reads the right half of the operand a btree index passes second. Text whose fold is longer than 1023 bytes or ends in NUL is refused, on the query path before any key is touched. The CLLW `Ore` and `Ope` indexes are unchanged, byte for byte. BREAKING CHANGE: `IndexSpec` is exhaustive and gains `OreBlock`, so a `match` over it needs an arm for it.
… chain and exact containment Stack Encrypt had no JSON index: `op::Contains`, `op::ContainedBy` and `op::AtPath` were declared (#1209) and answered by nothing, and the v4 JSON wire format was unconstrained by anything shipped. This is Phase 1 of the spike in docs/plans/2026-10-17-json-index-stack-encrypt-spike.md (#1249). `Json` is an `Index<serde_json::Value>` whose term, `JsonTerms`, is the document's one-way terms: a path selector and a value selector per node, a token per node below the chain's heads, the grouping by container array element, and a leaf ordering term on numbers and strings. Sealing is the separate operation `target::json_document`, which flattens the document, requests one data key under the column's descriptor and seals every node's own label, kind and scalar under it, so a document's ciphertext is linear in its size. The stored `JsonDocument` carries the design's chain fields: position-keyed parent tags, wrapped tokens, and group tags salted by the document's iv, so nothing links across rows until a query supplies a token. By default the root is sealed: the root entry is the whole document and the top-level fields are the chain heads, so no token exists for the root. Queries go through the engine's own operators. `Supports<Contains>` and `ContainedBy` send a `JsonQuery`, the element-nested projection of the needle's selectors, which native `jsonb @>` compares exactly across array elements (the v4 half of #1231). `AtPath<Eq>` sends the projection of the needle document that holds the value at the path; the orderings at a path send a `JsonLeafQuery`, the selector and the leaf's OPE term; `Between` a `RangeQuery` of two. `Input<Pathed<One>>` and `Input<Pathed<Pair>>` are implemented for `(path, value)` and `(path, (lo, hi))`, which #1209 left open until the index existed. `JsonDocument` is an `EncryptFrom<Value>` target and a `DecryptInto<Value>`, and an extracted node (`JsonNode`) or the rows of a subtree walk (`JsonSubtree`) open on their own. `IndexSpec::Json(JsonOptions)` is the data form, key `"json"`, with the object form for a non-default `sealed_root`; a data plan reads it and refuses to run it with `Reason::JsonIndexPending`, since the binding's half is not written. The data form, the dynamic arms and the Go guest's and fuzz crate's lockfiles move with it (stack-encrypt gains `sha2`, already in the graph through vitaminc-hmac). Two deviations from the design are recorded in the module docs and the plan: nonces are the AEAD's own random ones rather than derived, so the crate keeps one leaf cipher, and the ordinal field is `o`, since `i` is the EQL envelope's identifier. Refs #1249, #1231, #1209, #1060
…, EQL v4 only A new family beside `public.eql_v4_json`, not a port of the SteVec surface: `public.eql_v4_json_tree` (the document), `_json_tree_entry` (one node, as the extractors return it), `eql_v4.query_json_tree` (a containment needle) and `eql_v4.query_json_tree_leaf` (an ordering operand at a path). Phase 2 of the spike in docs/plans/2026-10-17-json-index-stack-encrypt-spike.md (#1249). Confined to v4 with `-- ONLY EQL VERSION: 4`, so the v3 bundle is unchanged; the build's ordering, completeness, symbol-order and isolation gates pass as they are. Everything the server does is core PostgreSQL. Containment projects a stored document from its group tags into the element-nested shape the needle has (`eql_v4.json_tree_query`, a recursive plpgsql group builder) and compares the two with native `jsonb @>`, so each needle array element must fit inside one stored element: PostgreSQL's own rule, which closes the cross-element false positives of selector-set containment. A GIN index on that expression serves `@>`; the function is deliberately not STRICT, because a STRICT SQL function whose body is an `OR` is not inlined and the index never engaged until it was changed. The token chain is walked by a recursive CTE from one client-derived token, with `sha256`, `set_byte` and a sixteen-byte XOR: `json_tree_subtree`, `json_tree_subtree_at` (gated on a selector for a wildcard below the anchor), `json_tree_children` and `json_tree_child_count`. `->` by selector grafts the header and the entry's ordinal, so the entry opens on its own; `ord_term` serves a functional btree for the orderings. Every other native jsonb operator on the column is blocked, as on the other domains. `eql-bindings` gains the hand-written `v4::json_tree` types, with Stack Encrypt's `EncryptFrom<Value>` / `DecryptInto<Value>` for the document, the entry and a subtree's rows, under an `Identifier`; the family is not in the catalog yet, so it is not in the target table or `v4::all()`. The oracle is `tests/encryption/tests/json_tree.rs`, run by `mise run test:encryption:postgres`: the design's corpus and random documents stored encrypted beside their plaintexts, with `@>` and `<@` over the encrypted column checked against PostgreSQL's own `jsonb` operators over the plaintext column for every needle, the GIN index shown in `EXPLAIN`, extraction, the walk, the array functions and the orderings. This is Phase 0's oracle folded into Phase 2, against the real SQL and real encryption under the fake key source, so it runs on a fork PR. Refs #1249, #1231
…its design The implementation notes for Phases 1 and 2: where the code is, which of the ten decisions it took, and the deviations a reviewer reading the design first should know before the code (random nonces, the `o` ordinal, the walk excluding its anchor, the non-STRICT containment, the `<@` gap for a stored root scalar, byte-for-byte string selectors, and what the data plan and the catalog still lack). The agent guides point at the family's four homes. Refs #1249
🦋 Changeset detectedLatest commit: c645a98 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
…ional-terms feat(eql)!: EQL v4 text search queries carry only the term their operator reads; produce TextSearch
EQL v4's `ob` is one ore-rs ciphertext: the fixed-width 6-bit scheme
(header `02 02`) or the chained one (`02 03`, then the padding policy).
v3's is cipherstash-client's 8-bit blocks, which need another comparator,
so the v4 SQL holds its own in a version block:
- `{{ore_block}}_hash_bit`: ore-rs's sigma-MMO hash, the GF(2^128)
doubling on two bigint halves with no branch on the data, then AES under
the public key `ORE-rs.v2.H-pi.1` through pgcrypto.
- `{{ore_block}}_scheme`: the shape checks ore-rs's comparators make
(header, block count, exact length, padding policy and the block counts
it can produce, every symbol below 64). A malformed term raises, naming
its length and never its bytes.
- `compare_{{ore_block}}_term`: raises on terms of different schemes,
fixed-width terms of different block counts and chained terms padded
under different policies. For terms of one shape it runs a constant
operation count: one statement per block evaluates both comparisons and
latches the first difference arithmetically, the hash runs once on every
path, and the result is selected without a branch. Chained terms compare
lexicographically, the shorter first when its blocks all match.
- `compare_{{ore_block}}_terms` over an array requires exactly one term.
The v4 bundle does not reuse the 8-bit format's name. The renderer gains a
placeholder, `{{ore_block}}`, rendered `ore_block_256` in v3 and
`ore_block_v2` (ore-rs's wire version 2) in v4, and refuses a literal
`ore_block_256` outside the source directory's path. The hand-written SQL,
the pin script, the generated extractors and the ORE fallback take it.
`release/cipherstash-encrypt.sql` and its uninstaller are byte-identical
before and after.
`tests/timing/ore_block_v2/` is a hand-run timing harness for the v4
comparator, with a detached ciphertext generator. ADR-0002 is amended.
An EQL type whose `ob` term is block ORE lists the index `ore_block` (`record.OreBlock`) in `se_targets`, and the engine answers its orderings through it. The checker accepts it in a type's indexes and as an operator's index, and stashgen maps it to `IndexOreBlock`. It is not a tag word: a field reaches it only by naming such a type with `encrypt_into=`, so it has no entry in `indexNames` or `outputOfIndex`, and a plan's own outputs still refuse it. The static test engine produces `TextOrdOre`, `TextSearchOre` and `IntegerOrdOre`, so the refusal test names `BigintOrdOre` as the type it does not produce.
`IntegerOrdOre`, `SmallintOrdOre`, `BigintOrdOre`, `RealOrdOre`, `DoubleOrdOre`, `TextOrdOre` and `TextSearchOre` are producible. Their `ob` term is Stack Encrypt's block ORE term, one hex element holding the whole ore-rs ciphertext: `OreBlock256` implements `EncryptFrom` for any orderable plaintext and never decrypts. `TextSearchOre` follows `TextSearch`: its stored value takes the empty filter, and its query carries only the term its operator reads. In the target table an `ob` term is the index `ore_block`, not `ore`, which is CLLW ORE and has no EQL domain, and the block-ORE reason in `encryption_gap` is gone. The unproducible types are the date, timestamp and numeric families, which have no Rust plaintext type yet, and the two JSON types. `tests/encryption/tests/v4_ore_block.rs` compares real ore-rs ciphertexts in PostgreSQL, both ways round, against the plaintext order and ore-rs's own comparator: equal values, a first difference in every block, shared prefixes and text in three padding buckets. It checks the hash bit against ore.rs's vectors, that malformed and mismatched terms raise, and that a btree index answers `<`, `>=`, `=` and `ORDER BY`. `test:encryption:postgres` runs it. The Go catalog, the guest's tests and the Go operator test follow. ADR-0009 is amended. The EQL v4 range tests (tests/encryption/tests/v4_range.rs) now derive their order terms from the engine. They arrived from the range domains with cipherstash-client's ORE and OPE indexers writing the `ob` and `op` of every row and bound, which was the only source of block ORE terms while EQL's ordered types were not producible. The comparator this series adds refuses that 8-bit layout, and this commit is where the `OrdOre` types become producible, so the rows are real `IntegerOrdOre`, `TextOrdOpe`, ... values and each bound is the term the single-bound query of that type carries. The smoke test ranges over the families whose ordered types are producible and records that the date, timestamp and numeric ones are not.
…tence test `v3_and_v4_install_side_by_side` asserted that eql_v3 and eql_v4 own the same number of public domains. That held while the two bundles were one SQL tree rendered under two names. The JSON tree family under `src/v3/json_tree/` is confined to v4 with `-- ONLY EQL VERSION: 4`, so v4 now owns `public.eql_v4_json_tree` and `public.eql_v4_json_tree_entry` with no v3 counterpart, and the count assertion failed in CI (52 vs 54). The test now compares the domain sets by name: every v3 public domain must exist in v4 under the v4 prefix, and the domains v4 owns that v3 does not must be exactly the JSON tree family. A future v4-only domain fails the test until it is named here, which is the point: EQL v3's SQL must not change, and a domain leaking from v4 into v3, or a v3 domain missing from v4, is a build regression rather than a design decision. Verified by installing both built bundles into a scratch PostgreSQL 17 and diffing the sets with the same query: 52 v3 domains, 54 v4, nothing missing from v4, and the two JSON tree domains as the only surplus. Refs #1288
…tence test `v3_and_v4_install_side_by_side` asserted that eql_v3 and eql_v4 own the same number of public domains. That held while the two bundles were one SQL tree rendered under two names. The JSON tree family under `src/v3/json_tree/` is confined to v4 with `-- ONLY EQL VERSION: 4`, so v4 now owns `public.eql_v4_json_tree` and `public.eql_v4_json_tree_entry` with no v3 counterpart, and the count assertion failed in CI (52 vs 54). The test now compares the domain sets by name: every v3 public domain must exist in v4 under the v4 prefix, and the domains v4 owns that v3 does not must be exactly the JSON tree family. A future v4-only domain fails the test until it is named here, which is the point: EQL v3's SQL must not change, and a domain leaking from v4 into v3, or a v3 domain missing from v4, is a build regression rather than a design decision. Verified by installing both built bundles into a scratch PostgreSQL 17 and diffing the sets with the same query: 52 v3 domains, 54 v4, nothing missing from v4, and the two JSON tree domains as the only surplus. Refs #1288
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 375 | 0 | 118 | 0 |
Every mutant in the changed lines was caught by a test.
feat(eql)!: produce the block ORE EQL v4 types through ore-rs 6-bit block ORE
This comment has been minimized.
This comment has been minimized.
The Mutants gate on this branch reported 53 surviving and 6 timed-out mutants across stack-encrypt's json module. Each is now caught, removed or excluded: - The path parser and the tree rebuild indexed their input by hand, so a `+= 1` mutated to `*= 1` never advanced and the test run hung. The parser now walks a shrinking &str and the rebuild pulls children from an iterator, so every step consumes input and no such mutant exists. - New tests pin what the survivors showed was unchecked: refusal offsets and reasons for malformed paths, JSON escapes in quoted names, the integer/float boundary in number encoding, every truncation of a header and of a node label, the key tag's u16 length limit, the header and selector Debug forms (a subtree key never prints its token), stored wrapped tokens differing from the token, opening each stored form as a record field, documents without a sealed root, and accessor queries refused before any keyset load. - Four mutants are equivalent and are excluded in .cargo/mutants.toml with the reason beside each, matched on the specific replacement. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
6427424 to
b9ac367
Compare
b8d1c87 to
55bbd66
Compare
…ex-stack-encrypt Brings up the review fixes to the json_tree design record and its SQL prototypes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
The JSON index and the EQL v4 json_tree family do what the description says: I ran cargo nextest run -p stack-encrypt --all-features (746 pass), clippy with all features and targets, mise run test:encryption, and mise run test:encryption:postgres against PostgreSQL 17, and the containment oracle agrees with jsonb @> on every needle. The defects below are in what the code accepts and how it opens what the server returns: one loses data, one exposes key names to anyone who holds a token, and several let a malformed or tampered stored document crash the client.
Findings, most severe first
-
High: a sealed-root document nested 128 to 512 levels deep encrypts and then never decrypts.
packages/stack-encrypt/src/json/derive.rs:40admits 512 levels, but the sealed root's plaintext is the document as JSON, andderive.rs:733reads it back withserde_json::from_slice, whose parser stops at 128 levels. I sealed[[[…1…]]]at depths 100 to 500 under both options: with the default sealed root, every depth from 128 up seals without error and opens toOpenedType; withsealed_root: falseevery depth opens. The write succeeds and the value is unreadable. Fix: refuse at the depth the reader can parse (127), so sealing fails instead of the read, and add a test at the boundary under both options. -
High: anyone who holds a token can test guessed key names, offline, in every row of the column.
derive.rs:239derives a child's token as an unkeyed hash,H("t" ‖ t(parent) ‖ 0x01 ‖ key), andderive.rs:265ranks object members by another,H("o" ‖ t(parent) ‖ key). The walk hands the server every descendant's token, so with one token the server can checkH("t" ‖ t ‖ 0x01 ‖ guess)against each child at every level below it, in every row, for as long as the column lives. Object keys are data whenever an object is a map (keyed by email address, user ID or SKU). Decision 11 in the design proposes the remedy, and it is free: the server unwraps child tokens fromwand never derives one from a label, so only the client, which holds the index key, computes the step. Fix: derive the label step and the rank under the index key with their own PAE domains. Later in the stack, #1329 replaces the label step with an ordinal chain but keeps the unkeyed rank, so the rank fix has to carry up. -
Medium: an extracted node opens under whatever selector the row carries, so the server can answer one path with another path's value.
DecryptInto<Value> for JsonNode(derive.rs:921) builds the AAD fromself.entry.s, which comes from the server's row, and nothing asks the caller which path it requested. The entry's AAD does bind its selector, but the binding only helps if the opener supplies the selector it expects. As it stands, a server answeringcol -> sel($.salary)with the entry for$.bonusfrom the same row gets that entry decrypted and returned as the salary, with no error. This is a leaf not bound to its key from the caller's side, not the ADR-0010 container case.JsonSubtreehas the same shape: the rows of a walk are not checked against the path walked. Fix: make the expected selector part of what opens a node (for example, a context that carries theJsonSelector, compared with the row'ssbefore opening), and for a subtree, check each opened row's label and selector against the path. This changes the decrypt contract ofJsonNode,JsonSubtreeand the bindings'JsonTreeEntry/JsonTreeSubtreeacross the stack, so it is the owner's call. -
Medium: a malformed or tampered stored document can crash the client. These are reads of untrusted input:
packages/stack-encrypt/src/json/mod.rs:366:max(o) + 1overflows for an entry whoseoisu32::MAX. In a debug build that panics (I checked). In a release build it wraps, and anyonear the top makeschildrenbuild a vector of up to four billion tags before it can return.mod.rs:396:walkrecurses with no bound. A document whose tags form a cycle, or a chain thousands of entries long, overflows the stack, and a Rust stack overflow aborts the process rather than unwinding. The server can build either one for any token it has seen.derive.rs:815/derive.rs:993:rebuildrecurses once per container it consumes, andopen_subtreetakes the server's rows as given. Rows that repeat one authentic container entry, or that list only the containers of a wide document, drive the recursion as deep as the server likes.
Fix: in
children, check each entry's ownpagainstparent_tag(iv, token, o), which is the same answer with no fan-out loop; bound the walk by depth and by the number of entries; boundrebuildby the same depth; and refuse duplicate ordinals among a subtree's rows. Add tests that build each malformed document. #1329's bounded probe replaces the first, but the walk and rebuild recursion are still unbounded at the top of the stack. -
Medium:
JsonTermsprints the document's key names and every chain token.derive.rs:152says the terms hold "no plaintext and no key", but everyTermNodekeeps its positionalJsonPath, which holds the plaintext key names and array positions, and its token.#[derive(Debug)]onJsonTerms(and onAtPathTerms, which wraps it) prints both, so a{:?}in a log writes out the shape and key names of the document, plus capabilities thatJsonSubtreeKeyis careful never to print. Fix: a hand-writtenDebugthat prints the node count, and correct the doc comment. -
Low:
AtPath<Eq>is containment, not equality, for a container value or at the root.packages/stack-encrypt/src/json/query.rs:534builds the needle that holds the value at the path and answers it with@>. For a scalar at a path below the root, that is equality. For("$.a", {"x": 1})it matches{"a": {"x": 1, "y": 2}},("$.a", [])matches every array at$.a, and("$", 1)matches a stored root[1]through thealtprobe.op::AtPathis documented as an exact operator over a single value (op/mod.rs:172). Fix: refuse a container value incheck, with a newTermError, and send noaltfor an accessor. -
Low: an extracted entry compared with plain
jsonbfalls through to native jsonb.packages/eql/src/v3/json_tree/blockers.sql:11says every other native operator is blocked, butpublic.eql_v4_json_tree_entryonly blocks=and<>against another entry. Against PostgreSQL 17,(doc -> sel) < '{…}'::jsonband(doc -> sel) = '{…}'::jsonbrun native jsonb comparison over the ciphertext object and return rows with no error. That is exactly what a caller gets if they forget the::eql_v4.query_json_tree_leafcast on the operand.ORDER BY doc -> sellikewise sorts by jsonb's ordering of the entry object (I checked: it sorts by ciphertext), and it can only be prevented with an operator class, which needs a superuser, so it can only be documented. Fix: block<,<=,>,>=,=and<>between the entry andjsonbin both directions, and say onord_termthat ordering goes through it. -
Docs: the implementation notes say "Of the ten listed below" (
docs/plans/2026-10-17-json-tree-index.md:341). There are eleven decisions now, and the notes should record what the code did about 11.
Not reported: the ADR-0010 non-findings (stored container membership, order and length; passthrough fields; interchangeable leaves under one context). Random nonces and the unwiped plaintext copies are deviations the design notes already record. The walk's cost (fan-out times entries per level) and the ordering term on every leaf are replaced by #1325 and #1329.
|
|
||
| /// Deeper than this is refused rather than walked: the walk recurses, and a | ||
| /// document built in memory can nest deeper than any parser admits. | ||
| const MAX_DEPTH: usize = 512; |
There was a problem hiding this comment.
- High:
MAX_DEPTHadmits 512 levels, but the sealed root's plaintext is the document as JSON anddecode_node(line 733) parses it back withserde_json::from_slice, which stops at 128 levels. Sealing[[…1…]]from 128 levels deep up to 512 succeeds under the default sealed root, and the document then opens toOpenedType, so the value is unreadable. Refuse at 127 levels, the depth the reader can parse, and add a boundary test under both options.
There was a problem hiding this comment.
Fixed in 497c01e. MAX_DEPTH is now 127, the depth serde_json parses back, under either root option, so a document that is too deep is refused when it is sealed and not when it is read. the_deepest_document_the_index_takes_opens_again seals 127 levels of arrays and of objects under both options and opens them, and checks that 128 levels is JsonTooDeep. The design's implementation notes record the bound.
| } | ||
|
|
||
| /// `H("t" ‖ t(parent) ‖ 0x01 ‖ key)` or `H("t" ‖ t(parent) ‖ 0x02 ‖ LE64(index))`. | ||
| fn child_token(parent: &Tag, label: &Label) -> Tag { |
There was a problem hiding this comment.
- High: the label step is an unkeyed hash, so once the server holds a token it can test guessed key names against every child token the walk unwraps, at every level below the token, in every row, for as long as the column lives.
rank(line 265) has the same property. Only the client ever derives this step (the server unwrapsw), so keying it under the index key, as decision 11 proposes, costs the server nothing. Derive both under the index PRF with their own PAE domains.
There was a problem hiding this comment.
Fixed in aa90e8b. The token step is now PRF(json/token-step ‖ column, PAE(t(parent), 0x01, key)) (0x02, LE64(index) for an element) and the rank is PRF(json/rank ‖ column, PAE(t(parent), key)), each with its own PAE domain. This is a format change: every token below a chain head and every object ordinal changes. The types are unreleased, the SQL never derives either value, and the design's implementation notes now record decision 11 as taken (and count eleven decisions). the_token_step_and_the_rank_are_keyed checks that both depend on the key and are not the unkeyed hash. In #1329 the label step becomes an ordinal chain, but the keyed rank carries up the stack.
| /// a sealed root. A container node holds no value of its own and is | ||
| /// refused with [`Error::OpenedType`]; its subtree opens through | ||
| /// [`JsonSubtree`]. | ||
| impl DecryptInto<Value> for JsonNode { |
There was a problem hiding this comment.
- Medium: the AAD is built from
self.entry.s, the selector the server's row carries, and the caller cannot say which selector it asked for. A server answeringcol -> sel($.salary)with the$.bonusentry of the same row gets that entry decrypted and returned without an error. The same holds for the rows of aJsonSubtree. Have the opener supply the expected selector, compare it with the row'ss, and open under it. This changes the decrypt contract ofJsonNode,JsonSubtreeand the bindings' entry types, so it is the owner's call.
There was a problem hiding this comment.
Left for the owner. The fix changes what opens a node: the opener supplies the selector it asked for, it is compared with the row's s, and the entry opens under it. For a subtree, each opened row's label is checked against the path walked. That changes the decrypt contract of JsonNode, JsonSubtree and the bindings' JsonTreeEntry / JsonTreeSubtree in every PR of the stack (#1329 adds JsonSubtree::at, which is the natural place for the subtree half). JsonTreeDocument already takes an ExpectedContext<Identifier>, and an expected selector beside it would follow that precedent. Until then, a caller can compare node.entry().selector() with the JsonSelector it sent before decrypting.
| .iter() | ||
| .filter_map(|e| e.o) | ||
| .max() | ||
| .map_or(0, |o| o + 1); |
There was a problem hiding this comment.
- Medium:
o + 1overflows for an entry whoseoisu32::MAX(it panics in a debug build; I checked), and in a release build any largeomakes this allocate up to four billion tags.walk(line 396) recurses with no bound, so a cycle or a long chain of tags overflows the stack and aborts the process.rebuildin derive.rs has the same problem with rows the server repeats. Check each entry's ownpagainstparent_tag(iv, token, o)instead of looping over a fan-out, bound the walk by depth and entry count, boundrebuildby depth, and refuse duplicate ordinals in a subtree.
There was a problem hiding this comment.
Fixed in 151409c. children checks each entry against the parent tag at the entry's own ordinal, which is the set the SQL probe finds but with no fan-out loop. subtree stops below MAX_DEPTH and after as many entries as the document holds. rebuild refuses nesting deeper than MAX_DEPTH as OpenedType, and open_subtree refuses a repeated ordinal. The SQL walks stop below 127 levels too. The new tests build each malformed document: o = u32::MAX, a self-cycle that doubles at every level, a 1000-entry chain, a repeated row, and 200 one-member objects lined up as rows (that test fails without the bound). #1329's bounded probe replaces children, but the walk and rebuild bounds carry up the stack.
There was a problem hiding this comment.
Follow-up in c645a98: the mutants gate found that only the object arm of the rebuild bound was tested (depth + 1 to depth * 1 in the array arm survived). The test now also lines up one-element arrays, and fails with that mutation.
| /// The one-way terms of a document: one term node per node, in the | ||
| /// walk's order, with the root first. The term of the | ||
| /// [`Json`](struct@crate::target::Json) index, and what every query term is built | ||
| /// from; it holds no plaintext and no key. |
There was a problem hiding this comment.
- Medium: "it holds no plaintext" is not true. Each
TermNodekeeps its positionalJsonPath(the plaintext key names and positions) and its chain token, and the derivedDebugprints all of them, so a loggedJsonTermsorAtPathTermswrites out the document's key names and capabilities thatJsonSubtreeKeynever prints. Write aDebugthat prints the node count, and correct this comment.
There was a problem hiding this comment.
Fixed in 71ffa57. JsonTerms now has a hand-written Debug that prints JsonTerms { nodes: N, .. }, and the doc comment says what the terms do hold (the key names and the tokens) and that they should not be logged. the_terms_print_neither_key_names_nor_tokens formats terms over a document keyed by an email address.
| /// Equality at a path: the needle that holds the value at the path, | ||
| /// projected, which containment answers exactly. A position in the path | ||
| /// reads as the wildcard, since selectors do not tell positions apart. | ||
| impl<P, V> Supports<AtPath<Eq>, (P, V)> for Json |
There was a problem hiding this comment.
- Low: answered through containment,
AtPath<Eq>with a container value is not equality.("$.a", {"x": 1})matches{"a": {"x": 1, "y": 2}}, and("$.a", [])matches every array at$.a. At the root,("$", 1)matches[1]throughalt.op::AtPathis documented as an exact operator over a single value. Refuse a container value incheckwith its ownTermError, and send noaltfor an accessor.
There was a problem hiding this comment.
Fixed in 03a9223. check refuses an object or an array with the new TermError::JsonValueNotScalar (code stack_encrypt::json_value_not_scalar, whose help points at op::Contains), and an accessor's needle is built with no alt (JsonQuery::at_path). equality_at_the_root_is_not_containment checks that ("$", 1) matches a stored 1 and not [1], while op::Contains with 1 still matches [1], and an_accessor_asks_at_a_path checks the refusals.
| --! | ||
| --! public.{{prefix}}_json_tree SUPPORTS @> <@ -> ->> (see operators.sql), and | ||
| --! public.{{prefix}}_json_tree_entry the ordered comparisons < <= > >= against an | ||
| --! ordering operand or another entry. Entry equality = <> is blocked: an |
There was a problem hiding this comment.
- Low: for the entry domain only
=and<>against another entry are blocked.(doc -> sel) < '{…}'::jsonband= '{…}'::jsonbresolve to native jsonb over the ciphertext object and return rows with no error, which is what a caller gets for forgetting thequery_json_tree_leafcast. Block the six comparisons between the entry andjsonbin both directions.ORDER BY doc -> selsorts by ciphertext too, and since an operator class needs a superuser, documentord_termas the way to order.
There was a problem hiding this comment.
Fixed in fe7e457. <, <=, >, >=, = and <> between an entry and jsonb are blocked in both directions, and the header here and ord_term's docs say to order by eql_v4.ord_term(col -> 'sel'), since ORDER BY col -> 'sel' can only be redirected by an operator class, which needs a superuser. The PostgreSQL oracle checks that all twelve forms raise.
|
|
||
| **Where it is.** The engine half is `packages/stack-encrypt/src/json/`: `Json` is the index (`Index<serde_json::Value>`, term `JsonTerms`), `target::json_document` the sealing operation, `JsonDocument` the stored term and an `EncryptFrom<Value>` / `DecryptInto<Value>` target, `JsonQuery` the needle for `Contains`, `ContainedBy` and `AtPath<Eq>`, `JsonLeafQuery` the operand for the orderings at a path, and `JsonSelector` / `JsonSubtreeKey` the extraction keys, derived through `KeysetCipher::json_selector` and `json_subtree_key`. `IndexSpec::Json(JsonOptions)` is the data form, with the key `"json"` and `{"json": {"sealed_root": …}}` as its wire form; a data plan reads it and refuses to run it (`Reason::JsonIndexPending`). The SQL is `packages/eql/src/v3/json_tree/`, confined to EQL v4; the EQL types are `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; the oracle is `packages/eql/tests/encryption/tests/json_tree.rs`, which stores the design's corpus and random documents encrypted beside their plaintexts and checks that `@>` and `<@` over the encrypted column select the rows PostgreSQL's own `jsonb` operators select over the plaintext column, with the GIN index shown in `EXPLAIN`. The pure-Rust half of that oracle is `packages/stack-encrypt/tests/json_index.rs`, against a model of `jsonb @>` pinned to the answers in `jsonb-semantics.sql`. | ||
|
|
||
| **Decisions taken by the code.** Of the ten listed below: 1, hex for every sixteen-byte field and base64 for `c` and `h`, pending Phase 3; 2, leaf ordering terms bound to the column only, numbers as the `f64` they round to and strings as text, under the scalar `Ope` index's derivation; 3, wildcard arrays only; 4, group tags on, and only on container array elements; 5, the child count is in every container node's ciphertext, and a truncated subtree is `OpenedType`, not a shorter array; 7, plain SHA-256; 8, the option is the boolean `sealed_root`, on by default, recorded in the header's flags; 9, the projection is built from the tags server-side (`eql_v4.json_tree_query`) and `groups_ok` was not needed; 10, position-keyed parent tags on, the ordinal stored as `o`, object members ranked by a hash under the parent's token. |
There was a problem hiding this comment.
- Docs: there are eleven decisions now. Say what the code did about decision 11.
There was a problem hiding this comment.
Fixed in aa90e8b: the notes say "Of the eleven" and record what the code did about decision 11 (keyed, with its own PAE domains). The notes also gained a bullet for each of the other fixes.
…t reads back The JSON index admitted 512 levels of nesting, but a sealed root's plaintext is the document as JSON and serde_json parses JSON back only to 128 levels. A document nested 128 to 512 levels deep sealed without error under the default option and then opened to OpenedType: the write succeeded and the value was lost. The walk now refuses a document deeper than 127 levels under either root option, so the failure happens when the caller still holds the value, and the option never changes which documents the index accepts. A test seals the deepest admitted document of arrays and of objects under both options and opens it, and checks that one level deeper is refused when sealed. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
A child's token was an unkeyed hash of its parent's token and its label,
H("t" || t(parent) || 0x01 || key), and an object member's rank another,
H("o" || t(parent) || key). The walk hands the server every descendant's
token, so a server that held one token could test guessed key names
against every child below it, in every row of the column, for as long as
the column lived. Object keys are data whenever an object is a map.
The design's decision 11 proposed keying the step, and it is free: only
the client derives a child token from a label (the server unwraps w), and
the client holds the index key. Both steps are now PRFs under the index
key with their own PAE domains, json/token-step/v1 and json/rank/v1. This
changes every stored token below a chain head and every object ordinal, a
format change; the types are unreleased, so no stored document carries
the old form. The SQL never derives either value, so it is unchanged. The
design's implementation notes record decision 11 as taken and count
eleven decisions, not ten.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ake a reader do A stored document comes back from the database, so it is untrusted input, and its chain fields are not authenticated. Three reads let it choose the work: - JsonDocument::children computed max(o) + 1 and derived a tag for every ordinal below it. An entry with o = u32::MAX overflowed (a panic in a debug build) and, in a release build, a large o allocated up to four billion tags. It now checks each entry against the parent tag at the entry's own ordinal, which is the same set the SQL probe finds. - JsonDocument::subtree recursed with no bound, so a cycle of tags, or a chain longer than any sealed document, which whoever holds a token can write, overflowed the stack and aborted the process. It now stops below MAX_DEPTH and after as many entries as the document holds. - The rebuild of a subtree's rows nested whatever containers the server lined up, and the server can repeat a row or return only the containers of a wide document. A repeated ordinal is now refused, and a nesting deeper than MAX_DEPTH is OpenedType. The SQL walks get the same depth bound, so a cyclic document cannot keep a recursive query running. Tests build each malformed document; the rebuild test fails without the bound. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
JsonTerms said it held no plaintext, but each node keeps its positional path, whose key names are the document's plaintext keys, and its chain token, a capability over that subtree in every row. The derived Debug printed both, so a JsonTerms or an AtPathTerms in a log wrote out the document's key names and the tokens JsonSubtreeKey is careful never to print. Debug now prints the node count alone, and the doc comment says what the terms do hold. A test formats terms over a document keyed by an email address and checks that neither the key nor a token appears. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ontainment
op::AtPath is documented as an exact operator over one value at a path,
and the JSON index answered AtPath<Eq> with the containment of the needle
that holds the value there. For a scalar below the root that is
equality. For an object or an array it is containment: ("$.a", {"x": 1})
matched {"a": {"x": 1, "y": 2}} and ("$.a", []) matched every array at
$.a. At the root, the needle's alt made ("$", 1) match a stored [1].
A container value is now refused when the query is awaited, with a new
TermError::JsonValueNotScalar whose help points at op::Contains, and an
accessor's needle carries no alt. Tests cover the refusals and the root
case against op::Contains on the same documents.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The firewall's header said every native jsonb operator other than the
supported ones was blocked, but the entry domain blocked only = and <>
against another entry. (doc -> sel) < '{...}'::jsonb and = '{...}'::jsonb
resolved to the native jsonb operators, compared the entry's ciphertext
object and returned rows with no error, which is exactly what a query
gets when its ordering operand loses the query_json_tree_leaf cast.
The six comparisons between an entry and jsonb are now blocked in both
directions. ORDER BY col -> 'sel' cannot be blocked without an operator
class, which needs a superuser, so the firewall's header and ord_term say
to order by eql_v4.ord_term(col -> 'sel'). The PostgreSQL oracle checks
that all twelve forms raise.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The pull request adds the JSON index, six TermError variants and a sixth IndexSpec variant, but the crate's changelog did not mention any of them and still said the JSON operators had no index to answer them. IndexSpec is exhaustive on purpose, so the new variant is a breaking change and is listed as one. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
One more, found while fixing the review: |
…-and-projection Brings up the #1288 review fixes: the 127-level depth bound, the keyed token step and member rank, the bounds on reading a stored document, the JsonTerms Debug, AtPath<Eq> as equality, the entry-versus-jsonb blockers and the changelog. One conflict, in functions.sql: this branch rewrote json_tree_subtree and json_tree_subtree_at around the parent map and the budget, and the fix added a depth bound to the old walks. This branch's walks are kept, with the bound re-applied to their recursive terms (WHERE c.entry IS NOT NULL AND w.depth < 127), since they still unwrap w and a cyclic document could otherwise keep an uncapped walk recursing. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…-ordinal-chain Brings up the #1288 review fixes. This branch replaced several of the places they touched, and its behaviour is kept wherever both sides edited the same code: - derive.rs: the ordinal chain is kept. Its child token is H(t || o), which the server derives as it walks and which carries no label, so the keyed label step from #1288 (json/token-step/v1) is dropped with it. The keyed member rank (json/rank/v1) stays: the server sees which selector sits at each ordinal under a token, so an unkeyed rank would let it rank guessed key names against them. Its doc and test say so. - mod.rs: this branch's bounded probe in children() is kept, and the walk takes both bounds, the entry count from here and MAX_DEPTH from #1288. The malformed-document test is rewritten for the ordinal chain (a 1000-link chain any token holder can write); the cycle and u32::MAX ordinal tests do not apply to a chain derived by hash with no stored ordinal. - functions.sql: this branch's walks are kept with no depth bound. A chain derived by hash cannot cycle and is no longer than the document, and a recursive CTE iterates rather than recursing on the stack. - codes.rs, sem/mod.rs: both new TermError variants, JsonPathNotOrdered and JsonValueNotScalar. - json_index.rs: both sides' tests, adapted to this branch's options API. - The design notes count eleven decisions on top of this branch's amendments, and record that only the keyed rank survives the ordinal chain. The changelog entry describes this branch's form of the index. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings up the #1288 review fixes as #1329's branch carries them: the 127-level depth bound, the keyed member rank, the walk and rebuild bounds, the JsonTerms Debug, AtPath<Eq> as equality, the entry-versus-jsonb blockers and the changelog. No conflicts. With the rank keyed, the leakage reference's statement that nothing it describes reveals a key name holds for a token's holder too: the member order a token exposes is an order under the index key. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…build bound The mutants gate found that the rebuild's depth bound was exercised only through objects: replacing `depth + 1` with `depth * 1` in the array arm survived, since the test lined up one-member objects alone. The test now also lines up one-element arrays, each taken as the element of the one before, and fails with that mutation. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…-and-projection Brings up the array case of the rebuild-bound test from #1288, which the mutants gate asked for. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…-ordinal-chain Brings up the array case of the rebuild-bound test from #1288, which the mutants gate asked for. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings up the array case of the rebuild-bound test from #1288, which the mutants gate asked for. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This branch has not been deployed
1 participant
Summary
The JSON index for Stack Encrypt (our Rust encryption engine), and the EQL v4 domain family that stores and queries what it writes. EQL is the SQL library we install into a customer's PostgreSQL so it can search encrypted values. Before this, Stack Encrypt could not encrypt a JSON document searchably at all: the JSON operators existed but nothing answered them. After it, a JSON document encrypts into one small ciphertext per node (so the ciphertext grows with the document, not with the document times its depth), the server can walk the subtree under a node from one token the client derives from a path, and containment (
@>) agrees with PostgreSQL's ownjsonb @>across array elements, which the v3 design gets wrong (#1231).This is Phases 1 and 2 of the phased plan in #1249, built against the design in #1250 (this PR is stacked on it: read the design first, then this). Phase 0's oracle is folded into Phase 2. Phases 3 (measure) and 4 (decide the wire) are not here.
Changes
Engine (
packages/stack-encrypt)src/json/: theJsonindex (Index<serde_json::Value>), the sealing operationtarget::json_document, the storedJsonDocument(anEncryptFrom<Value>/DecryptInto<Value>target), the query terms (JsonQueryfor containment andAtPath<Eq>,JsonLeafQueryfor the orderings at a path), the extraction keys (JsonSelector,JsonSubtreeKey, derived on the keyset cipher), and what an extraction opens (JsonNode,JsonSubtree). The module docs are the reference for the wire form and the derivations.op:Supportsimpls forContains,ContainedBy,AtPath<Eq | Lt | Lte | Gt | Gte | Between>, and theInput<Pathed<One>>/Input<Pathed<Pair>>impls for(path, value)and(path, (lo, hi))that feat(stack-encrypt): JSON operators Contains, ContainedBy and AtPath #1209 left open.JsonDocumentanswers them, and they default toJson.IndexSpec::Json(JsonOptions), key"json", with the wire form{"json": {"sealed_root": bool}}. The data plan reads it and refuses to run it with a new reason,json_index_pending.TermErrorvariants for a bad path, a document too deep or wide, the sealed root's missing token, and an ordering on a non-orderable leaf.sha2, already compiled in the graph throughvitaminc-hmac; the four lockfiles that record the edge move with it.EQL (
packages/eql)src/v3/json_tree/{types,functions,operators,blockers}.sql, confined to v4 with-- ONLY EQL VERSION: 4: the four domains, the containment projection and operators,->, the subtree walk, the array functions, the ordering extractors, and the native-jsonb firewall. The v3 bundle is byte-for-byte unchanged.crates/eql-bindings/src/v4/json_tree.rs: the hand-written types with Stack Encrypt's targets. Not in the catalog yet, so not in the target table orv4::all().tests/encryption/tests/json_tree.rs, added tomise run test:encryption:postgres..changeset/eql-json-tree-domain.md(@cipherstash/eqlminor).Docs
Verification
cargo nextest run -p stack-encrypt --all-features: 716 tests pass, including the 14 new ones intests/json_index.rs: round trips under both root options, the chain unwrapping every child from its parent's token and nothing from a wrong one, tamper evidence (reordered, regrafted, truncated, a flipped byte, another column, a lifted node), and containment over the projection agreeing with a Rust model ofjsonb @>on the design's corpus and 300 random pairs. The model is pinned to the 23 answers injsonb-semantics.sql. Clippy clean with all features and with none; rustdoc clean with warnings as errors; the doc examples pass; the trybuild snapshots were regenerated and differ only by the new implementors they list.mise run test:encryption:postgresagainst PostgreSQL 17, with the v4 bundle built from the tree: all six suites pass. The new one stores 72 documents encrypted beside their plaintexts and, for 89 needles, checks that@>and<@over the encrypted column select exactly the rows PostgreSQL's ownjsonboperators select over the plaintext column, thatEXPLAINshows aBitmap Index Scanon the GIN index, that->opens to the leaf, that the walk from a client-derived token returns the six entries under$.nestedand reassembles them, that a wrong token unlocks nothing in any row, that a wildcard path walks each element's subtree, that the array functions count and list elements, that the orderings at a path select and sort as the plaintexts do, and that a native operator on the column raises.mise run test:encryption(both feature sets, clippy with warnings as errors),check:encryption:wasi,test:crates,build,codegen:parity,types:check,docs:validate:coverage,docs:validate:required-tags, anddocs:validate:documented-sql(the whole v4 install order in one transaction): all pass. The Go guest checks and lints with itseqlfeature under--locked. The three EQL CI-derivation tests underscripts/__tests__pass.STRICTcontainment function with anORbody is not inlined by PostgreSQL, so the GIN index never engaged until it was made non-STRICT (recorded in the SQL and the design notes); and an untyped literal on the firewall test resolved to the native operator, as the domain docs warn.stashgen, and the TypeScript export, which the design keeps out of the phased plan.Related
Refs #1249 (Phases 1 and 2)
Refs #1231 (the v4 half: exact containment across array elements)
Refs #1209, #1060
Stacked on #1250
Review notes
Read the design's new "Implementation notes" section first; it lists the deviations in one place. Then
packages/stack-encrypt/src/json/mod.rsfor the wire form,derive.rsfor the walk and the sealing,query.rsfor the projection, andpackages/eql/src/v3/json_tree/functions.sqlfor the server side of the same three things. The oracle intests/encryption/tests/json_tree.rsis what holds the two sides together.Deliberately deferred, with the reason in the design notes: a self tag per node (so a walk could return its anchor), the data-plan form, the catalog entry, and wiping the plaintext copies the walk makes.