Repository navigation
feat(stack-encrypt, golang): JSON operators and data-plan reads, with generated Go JSON document fields (#1374) - #1373
Merged
Merged
Conversation
…n its reads from a data plan A data plan could store an EncryptedJson document but not query or read one: the operators were reachable only from typed Rust, and a read's rows opened only against a JsonSubtreeKey or JsonSelector a binding cannot hold. OpSpec gains the document operators (contains, contained_by, eq_at, the orderings at a path, between_at, selector, subtree, children, child_count), each taking an object of named inputs with JSON values as JSON text. query_term derives the typed chain's operands bit for bit and tags them: the term's wire form gains `encoding`, and Operand gains Text and TextList, so a selector or a token list is named, not anonymous bytes; an operator entry gains `input`. A position is refused where the operand is a selector, which reads it as the wildcard, and a wildcard where the operator names one node. json_open opens a selector or subtree read's rows against the request that prepared it: the engine derives the selector or the key again from the request, the field's resolved options and the keyset, runs the index's own row checks on each database row's group, and confines opening to that keyset. Nothing in the rows says which path they are for. The guest exports it as se_json_open, and maps a wrong entry and a truncated wildcard to STATUS_ENCODING. The Go runtime reads the new keys (QueryTerm.Encoding/Text, record.Operator.Input) and adds Cipher.JSONOpen. tests/json_tree_data_plan.rs holds every operand to the typed chain's and opens reads of `->` and of a wildcard subtree across documents. Refs #1347, #1350, #1352 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
🦋 Changeset detectedLatest commit: 2851026 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Operator and query-term contracts packages/stack-encrypt/src/dynamic/query.rs, packages/stack-encrypt/src/dynamic/target.rs, packages/eql/crates/eql-bindings/src/encryption/targets.rs, languages/golang/internal/record/record.go, languages/golang/encrypt/checker.go, languages/golang/encrypt/query.go, packages/stack-encrypt/tests/*, packages/eql/tests/encryption/tests/json_tree_data_plan.rs, changelogs and changeset |
Operator entries now include input shapes. Query terms add encoding and support text and text-array operands. JSON targets and path-ordering operands carry JSON index and SQL-domain metadata. |
JSON query validation and derivation packages/stack-encrypt/src/dynamic/json_query.rs, packages/stack-encrypt/src/dynamic/query.rs, packages/stack-encrypt/src/dynamic/mod.rs, packages/stack-encrypt/src/json/*, packages/stack-encrypt/tests/json_operators.rs, packages/stack-encrypt/tests/heap_hygiene.rs, packages/eql/tests/encryption/tests/json_tree_data_plan.rs |
The JSON binder validates operator inputs and derives query, leaf, selector, node, and subtree operands. Operator availability follows JSON-index and ordering options. Tests cover operand forms, refusals, and heap handling. |
Request-bound reads and Go integration packages/stack-encrypt/src/dynamic/json_query.rs, packages/stack-encrypt/src/dynamic/query.rs, languages/golang/encrypt/guest/src/{abi.rs,ops.rs,status.rs}, languages/golang/encrypt/{guest.go,json.go}, languages/golang/encrypt/json_tree_test.go, packages/stack-encrypt/tests/{json_operators.rs,heap_hygiene.rs}, packages/eql/tests/encryption/tests/json_tree_data_plan.rs |
The read path validates requests and row groups, derives selector or subtree keys, and opens values. The guest ABI and Cipher.JSONOpen expose the operation. Tests cover matching and absent results, mismatched rows, and read bounds. |
Priority:
Estimated code review effort: 4 (Complex) | ~60 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant GoCipher
participant KMSGuest
participant GuestABI
participant GuestOps
participant DynamicQuery
GoCipher->>KMSGuest: Send rows, plan, field, request, and options
KMSGuest->>GuestABI: Call se_json_open
GuestABI->>GuestOps: Validate and open the JSON read
GuestOps->>DynamicQuery: Call json_open with plan, request, and rows
DynamicQuery-->>GuestOps: Return opened JSON groups
GuestOps-->>GuestABI: Encode the result
GuestABI-->>KMSGuest: Return encoded plaintext
KMSGuest-->>GoCipher: Return opened values by row
Possibly related issues
- cipherstash/stack#1351: Adds runtime query and read support that generated
json_treeGo methods can use. - cipherstash/stack#1347: Implements the Go SDK foundation for JSON data-plan encryption, query terms, and request-bound reads.
- cipherstash/stack#1348: Adds JSON target and data-plan integration for the Go and guest surfaces.
- cipherstash/stack#1349: Adds JSON data-plan querying, typed operands, query terms, and request-bound reads.
- cipherstash/stack#1350: Adds JSON containment, path equality, ordering, range, and selector operator handling.
- cipherstash/stack#1352: Adds request-bound opening for selector and subtree reads; the
json_tree_childrenopener remains outside this change.
Possibly related PRs
- cipherstash/stack#1378: Adds generated typed JSON document methods that use
Cipher.QueryTermandCipher.JSONOpen. - cipherstash/stack#1372: Adds the
JsonSearchtarget and JSON plan setup extended by this change.
Suggested reviewers: freshtonic, auxesis
Merge Risk: 
be251
-
This change adds query and read support for encrypted JSON documents across the Rust engine, the guest, and the Go runtime. The supplied evidence shows no outstanding concrete defect, and the earlier review findings were reported as fixed with tests. No blocking merge risk remains beyond normal checks.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 71.81% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 149 functions across 28 files. (7 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Title check The title uses a conventional commit prefix and names the main change: JSON operators and data-plan reads for stack-encrypt and the Go runtime. The changeset adds JSON document operators, request-boun… Full details: Title check
-
Explanation
The title uses a conventional commit prefix and names the main change: JSON operators and data-plan reads for stack-encrypt and the Go runtime. The changeset adds JSON document operators, request-bound JSON reads, and Go JSONOpen and query-term support, so the title matches the primary change. The title is long at 111 characters, but it remains specific and accurate.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 71.81% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 149 functions across 28 files. (7 skipped: 7 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
This comment has been minimized.
This comment has been minimized.
A json.RawMessage field tagged encrypt_into=EncryptedJson, with optional ordered paths and sealed_root, is sealed by the engine from the field's data plan and gets generated methods for every json_tree query and read: Contains, ContainedBy, EqAt, Selector, Subtree, Children, ChildCount, and an At accessor per ordered path with Lt, Lte, Gt, Gte and Between. Selector and subtree reads return handles (encrypt.JSONSelection, encrypt.JSONRead) that keep the request they were prepared from and open the rows the database returns only as answers to it. TestJSONDocumentsQueryAndReadInPostgres runs every method against the EQL v4 bundle from the tree and joins go:test:postgres. Refs #1347, #1351, #1352 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This was referenced Oct 25, 2026
Merged
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 63 | 0 | 32 | 0 |
Every mutant in the changed lines was caught by a test.
CI's rustdoc, CRAP and coverage gates failed on this branch: two public docs linked the private json_query module, and the JSON operators were exercised only from the EQL workspace, so stack-encrypt's own coverage of read_input, derive, open and read_open was zero. - OpSpec's docs link OpSpec::input, query_term and json_open instead. - tests/json_operators.rs runs every operator, the input refusals, the read refusals and reads through a resolver backed by the engine's own JSON index, so the crate measures its own code. - read_input is split per operator; its complexity alone (31) passed the CRAP threshold at full coverage. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…nst a selector The mutants gate's last survivors in derive: eq_at's needle and an ordering's leaf operand were only checked to be objects, and children's token only against child_count's. The test now holds eq_at to contains' shape and an ordering to another, and a node's token apart from its selector. Parsed's Drop and the JsonText visitor's expecting are excluded with their reasons, as for the PR below. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Automated code review of this pull request at
|
- A document field lists an operator only when its target names the SQL type the operand binds as, so field_operators lists exactly what query_term derives. - json_open drops its scope parameter, which had no effect: a read opens through the keyset that derived it, and that keyset alone. - between_at's arity is one (its input is one object); takes_pair is between alone. Pinned wire bytes re-pinned. - json_query::binds is exhaustive over OpSpec and defines is_json; answered_by lists the document operators in their own arm; json_reason matches JsonTextError's variants. - Go's parseQueryTerm refuses a term whose encoding, shape and domain do not agree as the engine writes them. - Parsed's Drop is no longer excluded from mutation: heap_hygiene's a_json_operators_input_leaves_no_plaintext_on_the_heap fails without it. - Docs: the guest's parse helpers and query_term (with encoding), Rust query_term's inputs and refusals, QueryTerm::domain, Cipher.QueryTerm, parseOperatorName, open()'s refusal reasons. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Review response to the findings listed above (fixed in abdc767 unless noted; merged with the base in accb055 and 978f0cd):
CI failures on the previous push of this branch (rustdoc's private links, the CRAP gate, which needed in-crate coverage of the JSON operators, and the mutants gate's survivors) were fixed in e1b21ce and b2b5e1d. Verified on this branch after the fixes: root |
…est writer CI's Linux run flagged a freed hash table in vitaminc's frame writer that held the marker in bytes left by the test's own input building (serde_json and a growing format! string), not by the engine. The inputs are now joined into strings sized before they are written and framed by the suite's object writer. The test still fails when Parsed's Drop is a no-op. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
- A json.RawMessage is a document only under encrypt_into; sealed or indexed any other way it stays bytes, so existing models generate as before. - An ordered path's accessor has its own Selector, so an ordering and the value it compares come from one path; the README says why. - JSONRead keeps the subtree input it was prepared from and sends it back when it opens; OpenOne refuses a wildcard path before opening anything. - JSONSelection.Open makes no request for a page with no entries and refuses more than one value for a row; a zero JSONSelector or JSONToken is refused rather than bound as ''. - gensupport's JSON helpers check the term's encoding, fidelity (record.Exact) and part count in one place. - stashgen: imports are re-checked after planning adds encoding/json; a document field needs the engine to answer selector on it; an ordered path given twice, or with no name, is refused by name; a key of digits names its accessor; index=json points at encrypt_into=EncryptedJson. - Tests check rows.Err() and no longer assert a driver value's type. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in the v4 JSON document's v3 names (`JsonSearch` over `public.eql_v4_json_search`, `eql_v4.query_json`, `eql_v4.query_json_leaf`, `public.eql_v4_json_entry`) and the SteVec SQL confined to EQL v3. Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
… v3 names The JSON operators and reads this branch adds named the document target `EncryptedJson` and bound their operands as `eql_v4.query_json_tree` and `eql_v4.query_json_tree_leaf`. With the base's rename those are `JsonSearch`, `eql_v4.query_json` and `eql_v4.query_json_leaf`, and a read opens `public.eql_v4_json_entry` rows. The resolvers the tests build, the pinned operand domains, the guest's and Go's operator tests, the operations changeset and the CHANGELOGs now say so. Refs #1350, #1352 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
Brings in, by way of the base, the v4 JSON document's v3 names (`JsonSearch` over `public.eql_v4_json_search`, `eql_v4.query_json`, `eql_v4.query_json_leaf`, `public.eql_v4_json_entry`), the SteVec SQL confined to EQL v3, and the base's JSON operators renamed to match. The one conflict, in cmd/stashgen/main_test.go, keeps this branch's refusal text; the rename of this branch's own code follows in the next commit. Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
Brings in the realigned JSON tree domain list and the corrected json index note in the target descriptor's docs. Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
A zero JSONSelector or JSONToken is refused with ErrEncoding when bound,
because the empty value matches no row without an error. JSONTokens did
not follow the rule: a nil or empty list bound as the text[] literal {},
so a zero JSONRead's tokens read nothing from every row and said nothing.
Every read the engine prepares has at least one token, so an empty list
is always a read that was never prepared. It is now refused the same way,
and the changelog's sentence covers the token list too.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
JSONToken and JSONTokens are documented as capabilities: a token opens the structure of its node's subtree in every row of the column. They still printed in the clear with %v and through slog. JSONSelection and JSONRead printed their selector, tokens and path with %+v, and a path's keys can be a customer's data (Go principle 12). Each now has String, GoString, Format and LogValue, as the query types in query.go do, so every fmt verb and slog record shows the type, the field and a token count, with [redacted] for the tokens, the selector and the path. The values still bind through driver.Valuer, which pgx and database/sql consult before a Stringer. JSONValue's doc now says it holds plaintext JSON that prints in the clear, as the generated plaintext types' docs do. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
NewJSONSelection and NewJSONRead are exported, so their parameter names are part of what godoc shows. The parameter was named plan, a word for the engine's internal form that a user never sees (Go principle 7). It is now declaration, for what it is: the declaration of the record type the field belongs to, and the docs say so. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
stashgen wrote a document field's At accessors only when the engine answered lt_at on it. A tag that names ordered paths on a field the engine answers no ordering on means the generator and the engine it embeds disagree, and the generator then wrote the field with no At and said nothing: the paths the user asked to order had no methods, and the build still passed. planJSON now refuses that field by name, so the drift fails generation instead of producing a quieter API. jsonEntry writes At whenever the field has ordered paths, since planJSON has already checked the engine answers on them. The test runs the embedded engine with lt_at removed from its answer, which is the one way to reach the refusal: the real engine always answers the orderings on an ordered field. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The "a sealed null" case asserted only that stderr named Customer.Prefs, so any refusal of that field passed it, including one for the wrong reason. It now asserts the field and the reason together: a document field carries an index other than equality, a nil has only an equality term, so a sealed nil would show which documents are absent, and the field must say null=sql. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
stashgen accepts a scalar field declared with any type defined over the scalar (type Email string), so a reader would expect type Doc json.RawMessage to work for JsonSearch too. It does not: stashgen knows a document by the json.RawMessage type itself, so a defined type is bytes, and JsonSearch refuses it with "declare the field json.RawMessage". An alias (type Doc = json.RawMessage) is the type itself and is accepted. Both READMEs now say so, and the stashgen test pins both: the alias generates an eql.JSONSearch field, and the defined type is refused by name with that reason. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The encrypt README's JSON section said how the selector, token and token list bind, but not that a zero one is refused, nor that the read types print and log redacted while JSONValue prints the plaintext it holds. A reader deciding what is safe to log needs both, and the README is where they look before the godoc. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in next's JSON tree cascade and random storage order (#1379) through the base, with the base's review fixes. The guest's status map keeps both sides' arms: JsonWrongEntry and JsonWildcardTruncated from this branch, JsonNotALeaf from #1379, all STATUS_ENCODING. The operators' derivation follows #1379's renames (a path builds the document `holding` a value; JsonQuery::of_terms and at_path take the terms by value), and the guest's json_open bounds IndexKeySource, which awaiting a pending now needs. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…in the JSON operators #1379 stores a document's entries in a random order, shuffles every containment query's selector set, and makes an entry's ciphertext commit to its children's. Three things here assumed otherwise: - the data plan's operand parity test compared a containment or equality operand with the typed chain's byte for byte; it now reads both back as JsonQuery, which compares as sets; - `->` on a container with members now fails as JsonNotALeaf (Go: ErrEncoding, code stack_encrypt::json_not_a_leaf), so a selector read of `$.visits` is pinned to that refusal in the data plan test and through the Go guest, and json_open's and Cipher.JSONOpen's docs say it; - the docs said an ordering or `->` through a wildcard path compares the first element; it sees an unspecified one, and an ordering over every element is written over all the matches (json_tree_path_query). The same wording is corrected in JsonOptions::ordered, JsonOrderingAtPosition and the typed orderings' position check. The data plan's identifiers follow #1379's naming: a containment operand is a document's selector set, not a needle (DocumentInput, Input::Document, Binds::Query). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
eq_at and the orderings build the document holding a value at the path they are given, one level per step, before anything checks its depth, and JsonPath::parse caps no length. A path of fifty thousand steps from a binding built a value that deep, whose recursive drop can overflow the stack and abort the host. A path with more steps than a document has levels (MAX_DEPTH) names no node, so every operator's path is now refused as JsonTooDeep when it is read, before anything is built or any keyset resolved. The rest of the operator module's docs: check_query and check_json_open say they refuse what the input alone shows (the limits a read's key hits are derive's and open's, after the keyset), json_open documents its query_context_field refusal (also in the changelog), OpSpec::fidelity says what exact means for the reads and tokens, and comments say why an ordering's position refusal is the index's own error while eq_at's and selector's is the data plan's, and what is_positional means at node_path's call site. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… read's, and their heap The malformed-input test mapped every refusal without a data plan reason to "engine", so a refusal of another kind passed; each row now names the index's term error (JsonOrderingAtPosition, JsonPathNotOrdered, ...), and gains the over-deep path rows. The orderings test's doc promised a check it does not make; it now says where options on a target that takes none are refused. A new test refuses rows that are not the read's: a document sealed under another keyset (JsonWrongEntry for a selector, ForeignKeyset for a subtree, both before any key is retrieved), an entry of another field of the plan, and a subtree read's rows handed over as a selector read's; a leaf's selector row read as a subtree of the same path opens to the same value, which is the node that read names. heap_hygiene gains a watched run of query_term (Contains, EqAt, GtAt, BetweenAt) and of json_open (selector and subtree) through the data plan, with the marker in the operands and the stored values. It fails when the JSON terms operation drops its owned document unwiped (checked by making that release a plain drop). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
A subtree read's request carries its wildcard bound as Go writes an integer; the guest test now opens one (a row with no node opens to no values) and refuses a bound that is not an integer as ErrEncoding. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
A merge left the JsonSearch entry's first line twice. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The re-export the operator path cap reads is unused without the dynamic feature, which warnings-as-errors refuses; it is now gated on it. The operator test passes its rows by value, as clippy asks. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in next's JSON tree cascade and random storage order (#1379) through the base, with the base's review fixes. The per-type PostgreSQL test's comment takes the base's wording, which names this branch's TestJSONDocumentsQueryAndReadInPostgres. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
A JSON document's entries are now stored in a random order (#1379), so `col -> sel` at a wildcard path, and an ordering through it, sees an unspecified element of each row rather than the first. The At accessors stay: their operands are what a comparison over every match takes, and the engine stores ordering terms at a wildcard path when asked to, so refusing ordered= there would leave those terms unreachable from Go. What changes is the SQL they are documented and tested with. - The generated XPath comment gives both forms: col -> Selector() at a path with no array step, and at a wildcard path EXISTS over eql_v4.json_tree_path_query for a comparison and the least or greatest match's ordering term for ORDER BY. Lt, Lte, Gt, Gte and Between get doc comments of their own, which they lacked, and Selector says which form it binds into. The READMEs and the CHANGELOG say the same. - TestJSONDocumentsQueryAndReadInPostgres compares at $.visits[*].at with EXISTS (some visit > 2, and some visit < 2, which only Alice's second visit meets), orders rows by their least visit ascending and greatest descending against the plaintexts' order, and reads a wildcard past its bound from PostgreSQL's own rows, which Open refuses as ErrEncoding. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Review-body findings, at 83bee57 (every inline thread is answered on its line):
This branch now carries #1379 (the cascade and the random stored order). 0d38691 follows it in the operators: containment and equality operands are compared with the typed chain's as sets, a selector read of a container is |
The mutants gate found the depth cap's comparison unpinned: refusing a path of MAX_DEPTH steps instead of more than MAX_DEPTH passed every test. A selector path of 127 steps, the deepest that can name a node, is now among the inputs the check accepts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in the base's test that pins the JSON path depth cap at a document's depth. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Carries the vitaminc Go bindings pin to 06568ba93 on vitaminc main. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Carries the vitaminc Go bindings pin to 06568ba93 on vitaminc main. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
feat(go): generate encrypted JSON document fields
coderdan
changed the title
coderdan
added a commit
that referenced
this pull request
Oct 26, 2026
Brings in the rest of the Go json_tree stack (#1373, #1374, #1378). The only conflicts were adjacent CHANGELOG entries in the Go module and stack-encrypt, resolved by keeping both. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request lets a language SDK query and read an encrypted JSON document (
JsonSearch, EQL'sjson_treetype) through a data plan, the declaration a non-Rust SDK sends Stack Encrypt (the Rust engine) as data. It is slice C of the plan in #1353: the JSON operators, selector and token preparation, and reads opened against the request that prepared them, in the engine and in the WebAssembly guest (the engine compiled for Go). The Go runtime learns to carry the new answers; generated Go methods follow in the next pull request.Every operand is the typed Rust chain's: a containment or equality operand is the same selector set, which each derivation sends in a random order of its own, and every other operand is the same bytes. A read's rows open only against the trusted request (operator, path, bound, field and keyset), which the engine re-derives from, so the database never says which path a row is for.
Changes
Operators (
packages/stack-encrypt/src/dynamic)OpSpecgains the document operators. Each takes an object of named inputs, with JSON values as JSON text:contains,contained_by{"json"}eql_v4.query_jsoneq_at{"path", "json"}lt_at…gte_at{"path", "json"}at an ordered patheql_v4.query_json_leafbetween_at{"path", "low", "high"}selector{"path"}textsubtree{"path", "wildcard_bound"?}text[]children,child_count{"path"}, no wildcardtext(the node's token)field_operatorslists them from the field's resolved options (the orderings only when the field orders a path), and only those whose SQL type the target names, so it lists exactly whatquery_termderives.between_at's arity isone: its input is one object.The query term's wire form gains
encoding(bytes,eql,text,text_array), andOperandgainsTextandTextList, so a selector or a token list is a named value, not anonymous bytes. An operator entry gainsinput(its input's shape), so a generator can tell inputs apart without a copy of the rules.Refusals before anything is derived: a position in a path where the operand is a selector (
json_path_position; a positional read is asubtreeread), a wildcard where the operator names one node (json_path_wildcard), malformed inputs (json_input), and the engine's existing ones (container at a path, unordered path, non-ordered leaf).json_open(cipher, plan, field, request, rows, resolver)/check_json_open: open the rows of aselectororsubtreeread, one group per database row, in one key request. The engine derives the selector or the subtree key again from the request and the field's options under the caller's keyset, runs the index's own row checks, and opens through that keyset alone.EQL (
packages/eql)Target::leaf_query_sql_domain, so the engine knows what an ordering at a path binds as.tests/json_tree_data_plan.rs: every operand equals the typed chain's; the operator list; the refusals; reads of->and of a wildcard subtree across two documents, absence as no value, rows refused against another path's request, and a truncated wildcard refused.Guest (
languages/golang/encrypt/guest)se_json_open; JSON read refusals (JsonWrongEntry,JsonWildcardTruncated) areSTATUS_ENCODINGwith their own diagnostic codes, not internal errors.Go runtime (
languages/golang)QueryTerm.EncodingandQueryTerm.Text;record.Operator.Input;record.JSONIndex;Cipher.JSONOpen.json_tree_test.go: every operator through the guest with its encoding and domain, a refused positional selector, and a selector read opened against its request (and refused against another's).Docs: CHANGELOGs (
stack-encrypt,eql-bindings, Go) and a changeset for@cipherstash/eql.Assumptions
The five owner decisions in #1347 are open; this stack takes the plan's recommended answer to each (the Go-facing ones land in the next pull request):
json.RawMessage/*json.RawMessage.{"json": …}), not the field's struct type.Also assumed:
contains,contained_by,eq_at,lt_at,lte_at,gt_at,gte_at,between_at,selector,subtree,children,child_count; input objects follow the plan's examples ({"json"},{"path", "json"},{"path", "wildcard_bound"}).json_tree_children/json_tree_child_count. The plaintext child-opening helper the plan lists (parent-bound child validation) is not in this pull request; children are exposed as encrypted entries for SQL. See "Not done" below.text/text[], named in the term'sdomain.Not done here
json_tree_childrenrows (child scalars without a subtree read). A subtree read of the node returns the same values today.Verification
After merging #1372 (which carries #1379, the cascade and the random stored order) and the review fixes, at 83bee57, on macOS arm64, Rust 1.94.1 (the root
mise.tomlpin), Go 1.26.10, PostgreSQL 17.10 (a fresh database per run):cargo fmt --all --checkandRUSTFLAGS=-D warnings cargo clippy --workspace --all-features --tests(and without--all-features): clean.mise x --env test -- cargo nextest run --workspace --all-features: 1,534 passed, 1 skipped, at 9b01531's tree; 83bee57 then gated a re-export on thedynamicfeature and dropped a needless borrow, after which clippy was clean andjson_operatorspassed again (6 tests). be25106 pins the depth cap's boundary (a path exactly as deep as a document is accepted), which the CI mutants gate found unpinned; the test fails under that mutant (checked). The new heap test (a_document_fields_operators_and_reads_leave_no_plaintext_or_key_on_the_heap) fails when the JSON terms operation drops its owned document unwiped (checked).packages/eql: fmt, clippy andcargo nextest run --workspace --exclude eql_tests(629 passed);types:generatethentypes:check(no drift);mise run test:encryption(538 passed);mise run test:encryption:postgres: every suite passed, thejson_treeoracle 14 of 14.wasm:guest:build,:eql, both deterministic builds,wasm:auth-guest:build),mise run wasm:guest:test,mise run go:test(every packageok),mise run go:lint(0 issues),gofmt -l(clean) andgo generate ./...(no drift).cargo metadata --lockedin the root, both guest workspaces andpackages/eql: clean.Related
Stack
Merge order is bottom to top; each pull request is based on the one below it.
06568ba93(docs: refresh root README around the three Stack pillars #526 onmain) by git reference (temporary) (merged)JsonSearchstored through a data plan (engine, EQL catalog, guest, Go runtime) (merged)https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Summary by CodeRabbit