Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/plans/2026-10-17-json-tree-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ Each phase is a PR against `next`, where #1209 landed.

The index, the SQL and the live oracle landed in the pull request stacked on this document; what follows records where the code is and where it departs from the design above, so the two can be read against each other. Phase 0 was folded into Phase 2, and Phases 3 and 4 remain open.

**Where it is.** The engine half is `packages/stack-encrypt/src/json/`: `Json` is the index (`Index<serde_json::Value>`, term `JsonTerms`), `target::json_document` the sealing operation, `JsonDocument` the stored term and an `EncryptFrom<Value>` / `DecryptInto<Value>` target, `JsonQuery` the needle for `Contains`, `ContainedBy` and `AtPath<Eq>`, `JsonLeafQuery` the operand for the orderings at a path, and `JsonSelector` / `JsonSubtreeKey` the extraction keys, derived through `KeysetCipher::json_selector` and `json_subtree_key`. `IndexSpec::Json(JsonOptions)` is the data form, with the key `"json"` and `{"json": {"sealed_root": …}}` as its wire form; a data plan reads it and refuses to run it (`Reason::JsonIndexPending`). The SQL is `packages/eql/src/v3/json_tree/`, confined to EQL v4; the EQL types are `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; the oracle is `packages/eql/tests/encryption/tests/json_tree.rs`, which stores the design's corpus and random documents encrypted beside their plaintexts and checks that `@>` and `<@` over the encrypted column select the rows PostgreSQL's own `jsonb` operators select over the plaintext column, with the GIN index shown in `EXPLAIN`. The pure-Rust half of that oracle is `packages/stack-encrypt/tests/json_index.rs`, against a model of `jsonb @>` pinned to the answers in `jsonb-semantics.sql`.
**Where it is.** The engine half is `packages/stack-encrypt/src/json/`: `Json` is the index (`Index<serde_json::Value>`, term `JsonTerms`), `target::json_document` the sealing operation, `JsonDocument` the stored term and an `EncryptFrom<Value>` / `DecryptInto<Value>` target, `JsonQuery` the needle for `Contains`, `ContainedBy` and `AtPath<Eq>`, `JsonLeafQuery` the operand for the orderings at a path, and `JsonSelector` / `JsonSubtreeKey` the extraction keys, derived through `KeysetCipher::json_selector` and `json_subtree_key`. `IndexSpec::Json(JsonOptions)` is the data form, with the key `"json"` and `{"json": {"sealed_root": …}}` as its wire form; a data plan reads it and refuses to run it (`Reason::JsonIndexPending`). The SQL is `packages/eql/src/v3/json_tree/`, confined to EQL v4; the EQL types are `packages/eql/crates/eql-bindings/src/v4/json_tree.rs`; the oracle is `packages/eql/tests/encryption/tests/json_tree.rs`, which stores the design's corpus, and batches of documents proptest generates (`json_tree/generated.rs`), encrypted beside their plaintexts and checks that `@>` and `<@` over the encrypted column select the rows PostgreSQL's own `jsonb` operators select over the plaintext column, on the GIN index and on a sequential scan, each plan shown in `EXPLAIN`. The pure-Rust half of that oracle is `packages/stack-encrypt/tests/json_index.rs` (`given_the_design_corpus` and the proptest properties in `given_generated_documents`), against a model of `jsonb @>` pinned to the answers in `jsonb-semantics.sql`.

**Decisions taken by the code.** Of the eleven listed below: 1, hex for every sixteen-byte field and base64 for `c` and `h`, pending Phase 3 (taken on 2026-10-23, Phase 3d: base64 for every field the SQL reads, base85 for `c`); 2, leaf ordering terms bound to the column only, numbers as the `f64` they round to and strings as text, under the scalar `Ope` index's derivation; 3, wildcard arrays only; 4, group tags on, and only on container array elements; 5, the child count is in every container node's ciphertext, and a truncated subtree is `OpenedType`, not a shorter array; 7, plain SHA-256; 8, the option is the boolean `sealed_root`, on by default, recorded in the header's flags; 9, the projection is built from the tags server-side (`eql_v4.json_tree_query`) and `groups_ok` was not needed; 10, position-keyed parent tags on, the ordinal stored as `o`, object members ranked under the parent's token; 11, keyed: in this first form a child's token was `PRF(json/token-step ‖ column, PAE(t(parent), 0x01, key))` (`0x02, LE64(index)` for an element) and an object member's rank `PRF(json/rank ‖ column, PAE(t(parent), key))`, each under its own PAE domain, so a token's holder could neither test key names against the child tokens the walk unwraps nor rank guessed names against the ordinals. Decision 6 is out of scope, as proposed. (Both steps were first hashed unkeyed, as the token chain section above writes them, and were corrected before any release.)

Expand Down
1 change: 1 addition & 0 deletions packages/eql/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion packages/eql/mise.toml
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,11 @@ psql "$EQL_TEST_DATABASE_URL" -v ON_ERROR_STOP=1 -f release/cipherstash-encrypt-
for suite in text_eq v4_types v4_order v4_date_time_numeric v4_range v4_text_search v4_ore_block json_tree; do
listing=$(cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored --list | sed -n 's/: test$//p')
[ -n "$listing" ] || { echo "error: '--ignored' selects no test in $suite; the PostgreSQL test has gone missing." >&2; exit 1; }
cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored
# `--show-output`: a passing test's output is otherwise captured, and the
# generated containment properties in json_tree log the proptest seed and
# case count they ran under (PROPTEST_RNG_SEED and PROPTEST_CASES replay
# them) and what they checked.
cargo test --locked -p eql-encryption-tests --test "$suite" -- --ignored --show-output
done
"""

Expand Down
4 changes: 4 additions & 0 deletions packages/eql/tests/encryption/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ postgres = "0.19"
# compared.
#
cipherstash-client = "=0.42.3"
# The generated containment oracle (tests/json_tree/generated.rs). The same
# requirement eql-domains and tests/sqlx carry, so the lockfile gains no
# second copy.
proptest = "1"
# The tampering test decodes a stored body, changes one byte and re-encodes
# it. The same line eql-bindings uses, so the lockfile gains no second copy.
base64 = "0.22"
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
15 changes: 14 additions & 1 deletion packages/eql/tests/encryption/tests/json_tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
//! `@>` and `<@` over the encrypted column against PostgreSQL's own
//! `jsonb @>` over the plaintexts, for the design's corpus and random
//! documents, with a GIN index on the projection that the planner is shown
//! to use.
//! to use, and again as proptest properties over generated batches of
//! documents and needles (`json_tree/generated.rs`; `PROPTEST_RNG_SEED`
//! replays a run).

mod common;

Expand Down Expand Up @@ -435,6 +437,17 @@ fn postgres_containment_over_the_encrypted_column_is_jsonb_containment() {
assert!(nulls.get::<_, bool>(0));
}

// =============================================================================
// PostgreSQL: generated documents
// =============================================================================

/// The containment oracle again, over documents and needles proptest
/// generates. A crate root's modules resolve beside it, and a file there would
/// be a test target of its own, hence the path.
#[allow(clippy::unwrap_used)]
#[path = "json_tree/generated.rs"]
mod generated;

/// Needs an EQL v4 installation. Extraction by selector opens to the leaf;
/// the subtree walk by a client-derived token returns exactly the nodes
/// under it and reassembles; the array functions count and list elements;
Expand Down
Loading
Loading