From effe92b48990a603ac72b3a05d92b98f685619a4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 00:34:42 +1100 Subject: [PATCH 01/19] test(stack-encrypt): check the json_tree leakage page against the scheme The leakage page for EQL v4's json_tree index states in prose what a database holding the column learns at rest and from each query. Nothing held the code to it: a leak the page leaves out would pass every test. This adds an executable form of the page. `profile.rs` writes each of its sentences as a clause of a leakage function over the plaintext alone, with every keyed value replaced by an opaque name that is equal exactly where the real bytes are. A simulator builds a wire-shaped column and query transcript from that function's output and seeded randomness only, chaining parent tags with the real hash below every token the profile says a query reaches. Twenty-seven distinguishers, each an exact feature an observer of the column could compute (including the database's own walk of the parent tags from a token it was sent), compare the simulated column with one the crate actually sealed under the fake key source. Three kinds of check make that mean something: the full profile is indistinguishable from the real scheme on handcrafted and 48 random columns; each of 34 mutant profiles, one clause dropped, is told apart by at least one distinguisher; and the page as written (without the one clause it lacks) is told apart by exactly the distinguisher that sees the missing clause. What it found: an ordering term's length (65 bytes for a number, 8 * (len + 1) + 1 for a string) is not on the page; the page's summary "not revealed at rest: whether two different paths hold the same value" contradicts its own op row, since equal values at two ordered paths have equal terms; and "sorted order of their key names" is UTF-8 byte order, not jsonb's. The two issues it was built to check (#1357, the stored order reveals shape; #1360, a subtree read's token reaches its whole top-level field) come out as the page states them. The page is left unchanged. The profile and the real-side adapter are expected to change with the 2026-10-24 design change (ordinal-free member tokens, a self tag, no gates); the module docs say how. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- Cargo.lock | 1 + packages/stack-encrypt/Cargo.toml | 4 + packages/stack-encrypt/tests/json_leakage.rs | 6 + .../tests/json_leakage/calibration.rs | 382 +++++++ .../stack-encrypt/tests/json_leakage/chain.rs | 58 + .../tests/json_leakage/corpus.rs | 291 +++++ .../tests/json_leakage/distinguishers.rs | 993 ++++++++++++++++++ .../stack-encrypt/tests/json_leakage/mod.rs | 208 ++++ .../tests/json_leakage/mutants.rs | 670 ++++++++++++ .../tests/json_leakage/observe.rs | 256 +++++ .../tests/json_leakage/profile.rs | 862 +++++++++++++++ .../stack-encrypt/tests/json_leakage/real.rs | 104 ++ .../tests/json_leakage/simulator.rs | 302 ++++++ 13 files changed, 4137 insertions(+) create mode 100644 packages/stack-encrypt/tests/json_leakage.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/calibration.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/chain.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/corpus.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/distinguishers.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/mod.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/mutants.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/observe.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/profile.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/real.rs create mode 100644 packages/stack-encrypt/tests/json_leakage/simulator.rs diff --git a/Cargo.lock b/Cargo.lock index 611e066e1..271420eac 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3406,6 +3406,7 @@ dependencies = [ "clww-ore", "miette", "ore-rs", + "proptest", "rust_decimal", "serde", "serde_json", diff --git a/packages/stack-encrypt/Cargo.toml b/packages/stack-encrypt/Cargo.toml index ec6812093..5578feea8 100644 --- a/packages/stack-encrypt/Cargo.toml +++ b/packages/stack-encrypt/Cargo.toml @@ -117,6 +117,10 @@ zerokms-protocol = { workspace = true } tokio = { workspace = true, features = ["rt", "macros"] } # Compile-fail tests for the derive diagnostics (`tests/ui`). trybuild = "1" +# The JSON index's executable leakage profile (`tests/json_leakage`): random +# columns for its positive control, and the seeded generator its simulator +# draws from. Already in the workspace graph (stack-auth, stack-guest-abi). +proptest = { version = "1.7", default-features = false, features = ["std"] } # The derive diagnostics are recorded with every feature on, as `test:unit` # and CI run them. They cannot be relied on to hold under both feature sets diff --git a/packages/stack-encrypt/tests/json_leakage.rs b/packages/stack-encrypt/tests/json_leakage.rs new file mode 100644 index 000000000..c67a511c0 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage.rs @@ -0,0 +1,6 @@ +//! An executable leakage profile for the JSON index (EQL v4 `json_tree`). +//! See `json_leakage/mod.rs` for what it is and how to read it. + +mod common; +#[path = "json_leakage/mod.rs"] +mod leakage; diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs new file mode 100644 index 000000000..de1794295 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -0,0 +1,382 @@ +//! The checks: the full profile is indistinguishable from the real scheme +//! (the positive control), the page as written is not (the finding), and +//! every clause's removal is noticed (the calibration). + +use proptest::test_runner::{Config, RngAlgorithm, TestCaseError, TestRng, TestRunner}; +use serde_json::json; +use stack_encrypt::json::{JsonOptions, JsonPath}; +use stack_encrypt::StackCipher; +use stack_kms::FakeDataKeySource; + +use super::corpus::{handcrafted, random_world, World}; +use super::distinguishers::{self, walk_positions}; +use super::mutants; +use super::observe::ObservedWorld; +use super::profile::{flatten, leak, Clauses, Column, KeyOrder, Leakage}; +use super::{real, simulator}; +use crate::common::stack_cipher; + +/// The seed of every simulator run and every mutant's choices. Any seed +/// works; a fixed one makes a failure reproduce. +const SEED: [u8; 32] = *b"json_tree leakage profile, v4 #1"; + +fn rng() -> TestRng { + TestRng::from_seed(RngAlgorithm::ChaCha, &SEED) +} + +fn runtime() -> tokio::runtime::Runtime { + tokio::runtime::Builder::new_current_thread() + .build() + .expect("a runtime") +} + +fn real_world( + runtime: &tokio::runtime::Runtime, + cipher: &StackCipher, + world: &World, +) -> ObservedWorld { + let (rows, queries) = runtime.block_on(real::world( + cipher, + &world.column, + &world.docs, + &world.queries, + )); + ObservedWorld::from_wire(&rows, &queries) +} + +fn simulated_world(leakage: &Leakage, rng: &mut TestRng) -> ObservedWorld { + let (rows, queries) = simulator::simulate(leakage, rng); + ObservedWorld::from_wire(&rows, &queries) +} + +/// Every distinguisher's feature of `world`, by name. +fn features(world: &ObservedWorld) -> Vec<(&'static str, String)> { + distinguishers::all() + .into_iter() + .map(|d| (d.name, (d.run)(world))) + .collect() +} + +/// The distinguishers whose features differ between two worlds, each with +/// the two features. +fn separating( + real: &[(&'static str, String)], + simulated: &ObservedWorld, +) -> Vec<(&'static str, String, String)> { + real.iter() + .zip(features(simulated)) + .filter_map(|((name, a), (_, b))| (*a != b).then(|| (*name, a.clone(), b))) + .collect() +} + +fn report(separated: &[(&'static str, String, String)]) -> String { + separated + .iter() + .map(|(name, real, sim)| format!("{name}:\n real: {real}\n sim: {sim}")) + .collect::>() + .join("\n") +} + +mod given_the_full_profile { + use super::*; + + /// The positive control on the handcrafted worlds: the simulator, given + /// `L` and randomness, produces a world no distinguisher tells from the + /// real one. + #[test] + fn no_distinguisher_separates_the_handcrafted_worlds() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + for world in handcrafted() { + let real = real_world(&runtime, &cipher, &world); + let leakage = leak( + &world.column, + Clauses::default(), + &world.docs, + &world.queries, + ); + let simulated = simulated_world(&leakage, &mut rng()); + let separated = separating(&features(&real), &simulated); + assert!( + separated.is_empty(), + "{}: the real scheme reveals more than L states:\n{}", + world.name, + report(&separated) + ); + } + } + + /// The positive control on random columns. Deterministic: the runner's + /// own generator is seeded, and so is every simulation. + #[test] + fn no_distinguisher_separates_random_worlds() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let config = Config { + cases: 48, + failure_persistence: None, + rng_algorithm: RngAlgorithm::ChaCha, + ..Config::default() + }; + let mut runner = TestRunner::new_with_rng(config, rng()); + let result = runner.run(&random_world(), |world| { + let real = real_world(&runtime, &cipher, &world); + let leakage = leak( + &world.column, + Clauses::default(), + &world.docs, + &world.queries, + ); + let simulated = simulated_world(&leakage, &mut rng()); + let separated = separating(&features(&real), &simulated); + if separated.is_empty() { + Ok(()) + } else { + Err(TestCaseError::fail(format!( + "the real scheme reveals more than L states on {world:?}:\n{}", + report(&separated) + ))) + } + }); + assert!(result.is_ok(), "{result:?}"); + } +} + +mod given_the_page_as_written { + use super::*; + + /// The finding, held as a test: with only what the page states, a + /// distinguisher separates the simulated world from the real one, and + /// it is the ordering term's length. When the page states it, move the + /// clause into the page's profile and this test changes with it. + #[test] + fn the_ordering_term_length_separates_it_from_the_real_scheme() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let mut fired = std::collections::BTreeSet::new(); + for world in handcrafted() { + let real = real_world(&runtime, &cipher, &world); + let leakage = leak( + &world.column, + Clauses::page_only(), + &world.docs, + &world.queries, + ); + let simulated = simulated_world(&leakage, &mut rng()); + fired.extend( + separating(&features(&real), &simulated) + .into_iter() + .map(|(n, _, _)| n), + ); + } + assert_eq!( + fired.into_iter().collect::>(), + ["ordering_term_lengths"], + "the page as written is separated from the real scheme by exactly the ordering term's length" + ); + } +} + +mod given_the_jsonb_reading_of_the_key_order { + use super::*; + + /// The page's "sorted order of their key names" does not say which + /// sort. Read as `jsonb`'s (shorter keys first), the profile is told + /// from the real scheme on a row whose keys sort differently the two + /// ways (`{"a": …, "a b": …, "é": …, "0": …}`): the sort is the keys' + /// UTF-8 bytes, and a reader inferring key names from the order needs to + /// know that. + #[test] + fn the_stored_order_tells_it_apart() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let clauses = Clauses { + key_order: KeyOrder::JsonbLengthFirst, + ..Clauses::default() + }; + let world = &handcrafted()[0]; + let real = real_world(&runtime, &cipher, world); + let leakage = leak(&world.column, clauses, &world.docs, &world.queries); + let fired: Vec<&str> = separating(&features(&real), &simulated_world(&leakage, &mut rng())) + .into_iter() + .map(|(n, _, _)| n) + .collect(); + assert!( + fired.contains(&"selector_pattern_across_rows"), + "the order of a row's selectors tells byte order from jsonb order: {fired:?}" + ); + } +} + +mod given_a_mutant_profile { + use super::*; + + /// Mutants no distinguisher is expected to notice, with the reason. A + /// mutant listed here that is noticed fails the test too, so the list + /// stays true. + const UNNOTICED: &[(&str, &str)] = &[]; + + /// The calibration: for each clause, the profile without it is told + /// from the real scheme by at least one distinguisher on some + /// handcrafted world. Prints the table of which. + #[test] + fn every_clause_is_load_bearing() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let worlds: Vec<(Vec<(&'static str, String)>, Leakage)> = handcrafted() + .iter() + .map(|world| { + ( + features(&real_world(&runtime, &cipher, world)), + leak( + &world.column, + Clauses::default(), + &world.docs, + &world.queries, + ), + ) + }) + .collect(); + let mut table = Vec::new(); + let mut failures = Vec::new(); + for mutant in mutants::all() { + let mut fired = std::collections::BTreeSet::new(); + let mut rng = rng(); + for (real, leakage) in &worlds { + let mut mutated = leakage.clone(); + (mutant.apply)(&mut mutated, &mut rng); + let simulated = simulated_world(&mutated, &mut rng); + fired.extend(separating(real, &simulated).into_iter().map(|(n, _, _)| n)); + } + let expected_unnoticed = UNNOTICED.iter().any(|(name, _)| *name == mutant.name); + if fired.is_empty() != expected_unnoticed { + failures.push(mutant.name); + } + table.push(format!( + "| `{}` | {} | {} |", + mutant.name, + mutant.clause, + if fired.is_empty() { + "**none**".to_owned() + } else { + fired + .iter() + .map(|n| format!("`{n}`")) + .collect::>() + .join(", ") + } + )); + } + eprintln!( + "| mutant | clause dropped | caught by |\n|---|---|---|\n{}", + table.join("\n") + ); + assert!( + failures.is_empty(), + "mutants whose detection disagrees with UNNOTICED: {failures:?}" + ); + } +} + +mod the_harness { + use super::*; + + /// The harness's hash chain is the crate's: walking a sealed document's + /// observed bytes from a subtree key's token finds exactly the entries + /// `JsonDocument::subtree` does, at the same depths. + #[test] + fn chain_is_the_crates_chain() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let doc = json!({"a": {"b": [1, {"c": 2}], "d": "e"}, "f": [[3], {"g": null}]}); + for sealed_root in [true, false] { + let options = JsonOptions { + sealed_root, + ..JsonOptions::new() + }; + let column = Column { + sealed_root, + ordered: Vec::new(), + }; + let (rows, _) = runtime.block_on(real::world( + &cipher, + &column, + std::slice::from_ref(&doc), + &[], + )); + let sealed: stack_encrypt::json::JsonDocument = + serde_json::from_value(rows[0].clone()).expect("a document"); + let observed = ObservedWorld::from_wire(&rows, &[]); + for path in ["$.a", "$.f", "$.f[1]", "$.f[0]"] { + let key = runtime + .block_on(async { + cipher + .default_keyset() + .json_subtree_key( + &JsonPath::parse(path).unwrap(), + &options, + stack_encrypt::nonempty!("users").with("profile"), + ) + .await + }) + .expect("derives"); + let crates: Vec<(usize, usize)> = sealed + .subtree(key.token_bytes()) + .iter() + .map(|(j, _, d)| (*j, *d as usize)) + .collect(); + assert_eq!( + walk_positions(&observed.rows[0], key.token_bytes()), + crates, + "{path} with the root sealed: {sealed_root}" + ); + assert!( + !crates.is_empty(), + "{path} reaches something, so the comparison is not vacuous" + ); + } + } + } + + /// The page's claim that the stored order reveals each row's shape holds + /// on a column whose rows vary: the order-only attack of + /// `distinguishers::depth_recovery` recovers every row's depth sequence + /// exactly, from the stored rows alone. + #[test] + fn the_order_reveals_the_shape() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let docs = vec![ + json!({"a": {"b": 1, "c": {"d": 2}}, "e": [1, {"f": 3}]}), + json!({"a": {"c": {"d": 2}}, "e": []}), + json!({"a": {"b": 1}, "e": [{"f": 3}]}), + json!({"e": [{"f": 1}, {"g": 2}]}), + json!({"e": [{"g": 1}]}), + json!({"a": {"c": {}}}), + ]; + let world = World { + name: "varied shapes", + column: Column { + sealed_root: true, + ordered: Vec::new(), + }, + docs: docs.clone(), + queries: Vec::new(), + }; + let observed = real_world(&runtime, &cipher, &world); + let truth: Vec>> = docs + .iter() + .map(|doc| { + flatten(doc, KeyOrder::Utf8Bytes) + .iter() + .map(|n| Some(n.positional.len())) + .collect() + }) + .collect(); + assert_eq!( + distinguishers::depth_recovery(&observed), + truth, + "the stored order alone gives every row's depth sequence" + ); + } +} diff --git a/packages/stack-encrypt/tests/json_leakage/chain.rs b/packages/stack-encrypt/tests/json_leakage/chain.rs new file mode 100644 index 000000000..af0067f75 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/chain.rs @@ -0,0 +1,58 @@ +//! The token chain's two public hash functions, written out from the design +//! rather than imported, because both sides of the harness need them on +//! bytes they observe: the database's half of the walk recomputes them over +//! a stored row's `iv`, a token it was sent and the selectors it holds, and +//! the simulator programs its random tags with them. +//! +//! These are the only functions of the real scheme the harness evaluates. +//! Everything keyed (selectors, value selectors, head tokens, member ranks) +//! stays on the client and is replaced by randomness in the simulator. +//! `the_harness_chain_is_the_crates_chain` (in `calibration.rs`) holds these +//! two to the crate's own walk, `JsonDocument::subtree`. + +use sha2::{Digest, Sha256}; + +/// A sixteen-byte tag: a selector, a value selector or a token. +pub type Tag16 = [u8; 16]; + +/// An eight-byte parent tag. +pub type Tag8 = [u8; 8]; + +fn sha256(parts: &[&[u8]]) -> [u8; 32] { + let mut hasher = Sha256::new(); + for part in parts { + hasher.update(part); + } + let mut out = [0; 32]; + out.copy_from_slice(&hasher.finalize()); + out +} + +/// `H8("p" ‖ iv ‖ t(parent) ‖ LE32(o))`: the parent tag of the child at +/// ordinal `o` under the node whose token is `parent`. +pub fn parent_tag(iv: &Tag16, parent: &Tag16, ordinal: u32) -> Tag8 { + let full = sha256(&[b"p", iv, parent, &ordinal.to_le_bytes()]); + let mut tag = [0; 8]; + tag.copy_from_slice(&full[..8]); + tag +} + +/// `H16("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`: the token of the child at +/// ordinal `o`, whose selector is `selector`. +pub fn child_token(parent: &Tag16, selector: &Tag16, ordinal: u32) -> Tag16 { + let full = sha256(&[b"t", parent, selector, &ordinal.to_le_bytes()]); + let mut tag = [0; 16]; + tag.copy_from_slice(&full[..16]); + tag +} + +/// The simulator's stand-in for the keyed member rank: any keyed function of +/// the parent's token and the member's selector will do, since the real one +/// is a PRF under a key the database never sees. The simulator samples its +/// own `key`. +pub fn simulated_rank(key: &[u8; 32], parent: &Tag16, selector: &Tag16) -> Tag16 { + let full = sha256(&[b"rank", key, parent, selector]); + let mut tag = [0; 16]; + tag.copy_from_slice(&full[..16]); + tag +} diff --git a/packages/stack-encrypt/tests/json_leakage/corpus.rs b/packages/stack-encrypt/tests/json_leakage/corpus.rs new file mode 100644 index 000000000..70f7ee918 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/corpus.rs @@ -0,0 +1,291 @@ +//! The worlds the harness compares: handcrafted columns that exercise every +//! clause, and a proptest generator of random columns over a small, shared +//! key vocabulary, so that rows have paths and values in common and the +//! equality patterns are rich. + +use proptest::prelude::*; +use serde_json::{json, Map, Value}; +use stack_encrypt::json::JsonPath; + +use super::profile::{Column, Query}; + +/// A column, its rows and the queries sent to it. +#[derive(Clone, Debug)] +pub struct World { + pub name: &'static str, + pub column: Column, + pub docs: Vec, + pub queries: Vec, +} + +fn ordered(paths: &[&str]) -> Vec { + paths.iter().map(|p| JsonPath::parse(p).unwrap()).collect() +} + +/// The design's fixture document, with the shapes of `json_index.rs` around +/// it and rows that vary it, so that each clause has something to show. +fn shapes() -> Vec { + vec![ + json!({"hello": "world-1", "nested": {"deep": "constant", "list": [{"x": 1}, {"x": 2}]}, "number": 1}), + json!({"hello": "world-2", "nested": {"deep": "constant", "list": [{"x": 2}]}, "number": 5}), + json!({"hello": "world-1", "nested": {"list": [{"x": 1, "y": [true, null]}, 3, [4, {"x": 1}]]}, "number": -2}), + json!({"number": 3, "nested": {"deep": "other", "list": []}, "zeta": {"k": "v", "a": 1}}), + json!({"a": "b", "a b": 1, "é": [], "0": {}}), + json!({"a": {"b": {"c": {"d": [1, "x", false]}}}}), + json!({"n": 18446744073709551615u64, "m": -9223372036854775808i64, "f": 1e300, "g": 2.5}), + json!([[1, 2], [3, [4, {"a": null}]]]), + json!([{"x": 1}, {"y": 2}]), + json!({}), + json!([]), + json!("a"), + json!(2.5), + json!(null), + json!(true), + ] +} + +fn shape_queries(sealed_root: bool) -> Vec { + // `$.nested` is only ever read through gates, so its token is sent but + // nothing under it is named whole: what that token reaches beyond the + // paths asked is the page's subtree-read clause. `$.zeta` is read twice, + // for determinism. + let mut queries = vec![ + Query::Subtree("$.nested.list".into()), + Query::Subtree("$.nested.list[1]".into()), + Query::Subtree("$.nested.list[*].x".into()), + Query::Subtree("$.zeta".into()), + Query::Subtree("$[1]".into()), + Query::Subtree("$[1][1]".into()), + Query::Subtree("$.zeta".into()), + Query::Extract("$.nested.list[0].x".into()), + Query::Extract("$.nested.list[1].x".into()), + Query::Extract("$.hello".into()), + Query::Extract("$".into()), + Query::Contains(json!({"nested": {"list": [{"x": 1}]}})), + Query::Contains(json!("a")), + Query::Contains(json!({"hello": "world-1"})), + Query::Contains(json!([{"x": 1}])), + Query::Equals("$.hello".into(), json!("world-1")), + Query::Equals("$.nested.list[*].x".into(), json!(2)), + Query::Order("$.number".into(), json!(0)), + Query::Order("$.nested.list[*].x".into(), json!(2)), + Query::Order("$.number".into(), json!(5)), + Query::Order("$.number".into(), json!(0)), + ]; + if !sealed_root { + queries.extend([ + Query::Subtree("$".into()), + Query::Subtree("$.a.b".into()), + Query::Subtree("$[*].x".into()), + ]); + } + queries +} + +/// A column that orders strings, so the string half of the ordering clause +/// has something to compare (a world orders one kind of leaf; see +/// [`super::profile::Ids::ordering_ranks`]). +fn names() -> World { + World { + name: "ordered strings", + column: Column { + sealed_root: true, + ordered: ordered(&["$.name", "$.tags[*]"]), + }, + docs: vec![ + json!({"name": "carol", "tags": ["b", "aa"]}), + json!({"name": "al", "tags": ["aa"]}), + json!({"name": "carol", "age": 3}), + json!({"name": "bob", "tags": []}), + json!({"tags": ["zzzzzzzz", "b"]}), + ], + queries: vec![ + Query::Order("$.name".into(), json!("b")), + Query::Order("$.tags[*]".into(), json!("aa")), + Query::Subtree("$.tags".into()), + Query::Subtree("$.tags[1]".into()), + ], + } +} + +/// A column with three wide top-level objects, each read whole, so that +/// whether members' ordinals follow their keys' order is decidable rather +/// than a coin toss (see `distinguishers::member_ordinals_follow_key_order`). +fn wide() -> World { + let object = |prefix: &str, n: usize| { + Value::Object( + (0..n) + .map(|i| (format!("{prefix}{i}"), Value::from(i as u64))) + .collect(), + ) + }; + World { + name: "wide objects", + column: Column { + sealed_root: true, + ordered: Vec::new(), + }, + docs: vec![ + json!({"w1": object("a", 8), "w2": object("b", 8), "w3": object("c", 8)}), + json!({"w1": object("a", 5), "w3": object("c", 8)}), + ], + queries: vec![ + Query::Subtree("$.w1".into()), + Query::Subtree("$.w2".into()), + Query::Subtree("$.w3".into()), + ], + } +} + +/// The handcrafted worlds: the shapes under each root option, and a column +/// of ordered strings. +pub fn handcrafted() -> Vec { + vec![ + World { + name: "shapes, sealed root", + column: Column { + sealed_root: true, + ordered: ordered(&["$.number", "$.nested.list[*].x"]), + }, + docs: shapes(), + queries: shape_queries(true), + }, + World { + name: "shapes, open root", + column: Column { + sealed_root: false, + ordered: ordered(&["$.number", "$.nested.list[*].x"]), + }, + docs: shapes(), + queries: shape_queries(false), + }, + names(), + wide(), + ] +} + +// ============================================================================= +// Random columns +// ============================================================================= + +const KEYS: &[&str] = &["a", "b", "c", "list", "n", "name", "tags", "zz"]; + +fn leaf() -> impl Strategy { + prop_oneof![ + Just(Value::Null), + any::().prop_map(Value::Bool), + (-3i64..6).prop_map(Value::from), + prop::sample::select(vec!["x", "yy", "x", "zzz", "ab", ""]).prop_map(Value::from), + ] +} + +fn object( + inner: impl Strategy, + size: std::ops::Range, +) -> impl Strategy { + prop::collection::btree_map(prop::sample::select(KEYS), inner, size).prop_map(|members| { + Value::Object( + members + .into_iter() + .map(|(k, v)| (k.to_owned(), v)) + .collect::>(), + ) + }) +} + +fn value() -> impl Strategy { + leaf().prop_recursive(3, 24, 4, |inner| { + prop_oneof![ + prop::collection::vec(inner.clone(), 0..4).prop_map(Value::Array), + object(inner, 0..4), + ] + }) +} + +/// Every value under the key `n` made a number, since the random columns +/// order `$.n` and `$.list[*].n` and a world orders one kind of leaf. +fn numberize(value: Value) -> Value { + match value { + Value::Object(map) => Value::Object( + map.into_iter() + .map(|(k, v)| { + let v = if k == "n" && !v.is_number() { + Value::from(i64::try_from(v.to_string().len() % 7).unwrap()) + } else { + numberize(v) + }; + (k, v) + }) + .collect(), + ), + Value::Array(items) => Value::Array(items.into_iter().map(numberize).collect()), + other => other, + } +} + +fn document() -> impl Strategy { + prop_oneof![ + 8 => object(value(), 0..5), + 1 => prop::collection::vec(value(), 0..3).prop_map(Value::Array), + 1 => leaf(), + ] + .prop_map(numberize) +} + +fn query(sealed_root: bool) -> impl Strategy { + let mut subtree = vec![ + "$.a", + "$.list", + "$.list[0]", + "$.list[1]", + "$.list[1].n", + "$.a.b", + "$.list[*].a", + "$.tags[0].a", + "$.a.list[2]", + "$[0]", + "$[1].a", + "$[0][1]", + "$.a.tags[*]", + ]; + if !sealed_root { + subtree.extend(["$", "$[*].a", "$.list[*]"]); + } + prop_oneof![ + 3 => prop::sample::select(subtree).prop_map(|p| Query::Subtree(p.into())), + 2 => prop::sample::select(vec!["$", "$.a", "$.a.b", "$.list[0].n", "$.list[3].n", "$.tags[*]"]) + .prop_map(|p| Query::Extract(p.into())), + 2 => prop_oneof![ + leaf(), + object(leaf(), 1..3), + prop::collection::vec(leaf(), 1..3).prop_map(Value::Array), + object(prop::collection::vec(object(leaf(), 1..2), 1..2).prop_map(Value::Array), 1..2), + ] + .prop_map(|v| Query::Contains(numberize(v))), + 1 => (prop::sample::select(vec!["$.a", "$.tags[0]", "$.list[*].b"]), + prop_oneof![(-3i64..6).prop_map(Value::from), prop::sample::select(vec!["x", "ab"]).prop_map(Value::from)]) + .prop_map(|(p, v)| Query::Equals(p.into(), v)), + 1 => (prop::sample::select(vec!["$.n", "$.list[*].n"]), -3i64..6) + .prop_map(|(p, v)| Query::Order(p.into(), Value::from(v))), + ] +} + +/// A random column: a root option, two to six documents, and four to ten +/// queries, some of which repeat. +pub fn random_world() -> impl Strategy { + any::().prop_flat_map(|sealed_root| { + ( + prop::collection::vec(document(), 2..7), + prop::collection::vec(query(sealed_root), 4..11), + ) + .prop_map(move |(docs, queries)| World { + name: "random", + column: Column { + sealed_root, + ordered: ordered(&["$.n", "$.list[*].n"]), + }, + docs, + queries, + }) + }) +} diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs new file mode 100644 index 000000000..22fdf3bfc --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -0,0 +1,993 @@ +//! The distinguishers: pure functions of an [`ObservedWorld`], each the +//! computation an observer holding the column (and, for the query features, +//! the terms it was sent) could run. +//! +//! Each returns an exact, combinatorial feature: byte strings are replaced by +//! integers in order of first appearance, so two worlds with the same +//! equality pattern give the same feature whatever their bytes. A real world +//! and a simulated one are compared feature by feature for equality; no +//! statistics are involved. The one source of chance is a collision of +//! random bytes (an eight-byte parent tag colliding with another, say), which +//! changes a feature with probability below `n² / 2^64` for `n` tags in a +//! world, about `10^-13` at the harness's sizes. +//! +//! What is deliberately not here, and why, is in the module docs +//! (`mod.rs`, "Out of reach"). + +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::hash::Hash; + +use super::chain::{child_token, parent_tag, Tag16, Tag8}; +use super::observe::{ObservedGate, ObservedGroup, ObservedQuery, ObservedRow, ObservedWorld}; + +/// One distinguisher: a name and the feature it computes, printed so that +/// any two features compare as strings and a mismatch reads as a diff. +pub struct Distinguisher { + pub name: &'static str, + pub run: fn(&ObservedWorld) -> String, +} + +macro_rules! registry { + ($($name:ident),+ $(,)?) => { + /// Every distinguisher, in a fixed order. + pub fn all() -> Vec { + vec![$(Distinguisher { + name: stringify!($name), + run: |world| format!("{:?}", $name(world)), + }),+] + } + }; +} + +registry!( + wire_fields, + entry_counts, + parent_tag_presence, + ordering_term_presence, + ciphertext_lengths, + ciphertext_length_multisets, + ciphertext_framing, + selector_pattern_within_rows, + selector_pattern_across_rows, + parent_tag_equalities, + headers, + projections, + value_selector_partitions, + chain_head_runs, + selector_precedence, + depth_recovery, + token_reach, + ordinal_patterns, + member_ordinals_follow_key_order, + token_derivability, + gated_walk_results, + transcripts, + extraction_matches, + containment_matches, + ordering_term_order, + ordering_term_lengths, + ordering_matches, +); + +/// Integers for byte strings, in order of first appearance. +struct Canon { + ids: HashMap, +} + +impl Canon { + fn new() -> Self { + Self { + ids: HashMap::new(), + } + } + + fn id(&mut self, value: T) -> usize { + let next = self.ids.len(); + *self.ids.entry(value).or_insert(next) + } +} + +/// The selectors of a world, named in one fixed traversal: every row's +/// entries, then every row's projection, then the queries. +fn selector_names(world: &ObservedWorld) -> Canon { + let mut canon = Canon::new(); + for row in &world.rows { + for entry in &row.entries { + let _ = canon.id(entry.s); + } + } + canon +} + +// ============================================================================= +// At rest: fields, counts and lengths +// ============================================================================= + +/// The wire keys of every row and entry. Catches a stored field the profile +/// does not account for, or one it invents. +pub fn wire_fields(world: &ObservedWorld) -> Vec<(Vec, Vec>)> { + world + .rows + .iter() + .map(|row| { + ( + row.fields.clone(), + row.entries.iter().map(|e| e.fields.clone()).collect(), + ) + }) + .collect() +} + +/// How many entries each row has: the node count. +pub fn entry_counts(world: &ObservedWorld) -> Vec { + world.rows.iter().map(|r| r.entries.len()).collect() +} + +/// Which entries carry a parent tag, by position. Catches a leak in which +/// entries are chain heads. +pub fn parent_tag_presence(world: &ObservedWorld) -> Vec> { + world + .rows + .iter() + .map(|r| r.entries.iter().map(|e| e.p.is_some()).collect()) + .collect() +} + +/// Which entries carry an ordering term, by position. +pub fn ordering_term_presence(world: &ObservedWorld) -> Vec> { + world + .rows + .iter() + .map(|r| r.entries.iter().map(|e| e.op.is_some()).collect()) + .collect() +} + +/// Every ciphertext's length, by position. Catches anything a length says +/// that the profile's length clause does not. +pub fn ciphertext_lengths(world: &ObservedWorld) -> Vec> { + world + .rows + .iter() + .map(|r| r.entries.iter().map(|e| e.c.len()).collect()) + .collect() +} + +/// Each row's ciphertext lengths as a multiset: what lengths reveal with the +/// order taken away. +pub fn ciphertext_length_multisets(world: &ObservedWorld) -> Vec> { + ciphertext_lengths(world) + .into_iter() + .map(|mut lengths| { + lengths.sort_unstable(); + lengths + }) + .collect() +} + +/// The ciphertexts' public framing (the set of leading version bytes) and +/// how many pairs of entries anywhere in the world share a ciphertext. +/// Catches a deterministic ciphertext (equal values sealing to equal bytes). +pub fn ciphertext_framing(world: &ObservedWorld) -> (BTreeSet, usize) { + let mut frames = BTreeSet::new(); + let mut seen: HashMap<&[u8], usize> = HashMap::new(); + for row in &world.rows { + for entry in &row.entries { + let _ = frames.insert(entry.c[0]); + *seen.entry(&entry.c).or_default() += 1; + } + } + let pairs = seen.values().map(|n| n * (n - 1) / 2).sum(); + (frames, pairs) +} + +// ============================================================================= +// At rest: equality patterns +// ============================================================================= + +/// Which positions of a row share a selector, named within the row alone. +pub fn selector_pattern_within_rows(world: &ObservedWorld) -> Vec> { + world + .rows + .iter() + .map(|row| { + let mut canon = Canon::new(); + row.entries.iter().map(|e| canon.id(e.s)).collect() + }) + .collect() +} + +/// Which positions anywhere in the column share a selector: the paths each +/// row has, linked across rows. +pub fn selector_pattern_across_rows(world: &ObservedWorld) -> Vec> { + let mut canon = selector_names(world); + world + .rows + .iter() + .map(|row| row.entries.iter().map(|e| canon.id(e.s)).collect()) + .collect() +} + +/// Every pair of entries, within a row or across rows, that share a parent +/// tag. The page says a tag "does not say which entry is the parent" and that +/// the same position in two rows is unrelated bytes, so this is expected +/// empty: siblings do not share a tag, because it is keyed by the child's +/// ordinal. +pub fn parent_tag_equalities(world: &ObservedWorld) -> Vec<((usize, usize), (usize, usize))> { + let mut seen: HashMap> = HashMap::new(); + for (r, row) in world.rows.iter().enumerate() { + for (j, entry) in row.entries.iter().enumerate() { + if let Some(p) = entry.p { + seen.entry(p).or_default().push((r, j)); + } + } + } + let mut out: Vec<_> = seen + .into_values() + .filter(|at| at.len() > 1) + .flat_map(|at| { + let first = at[0]; + at.into_iter().skip(1).map(move |other| (first, other)) + }) + .collect(); + out.sort_unstable(); + out +} + +/// Each header's public fields, and the equality patterns of its keyset, +/// `iv` and key tag across rows. Catches a shared `iv` (which would link +/// every row's parent tags) and a keyset that varies within a column. +#[allow(clippy::type_complexity)] +pub fn headers(world: &ObservedWorld) -> Vec<(u8, u8, [u8; 4], usize, usize, usize, usize, usize)> { + let mut keysets = Canon::new(); + let mut ivs = Canon::new(); + let mut tags = Canon::new(); + world + .rows + .iter() + .map(|row| { + let h = &row.header; + ( + h.version, + h.flags, + h.epoch, + keysets.id(h.keyset), + ivs.id(h.iv), + tags.id(h.tag.clone()), + h.tag.len(), + h.trailing, + ) + }) + .collect() +} + +/// A projection with its tags named. Read only through `Debug`, which is +/// how every feature is compared. +#[derive(Debug)] +#[allow(dead_code)] +pub struct NamedGroup { + pairs: Vec<(usize, usize)>, + elements: Vec, +} + +fn name_group( + group: &ObservedGroup, + selectors: &mut Canon, + values: &mut Canon, +) -> NamedGroup { + NamedGroup { + pairs: group + .pairs + .iter() + .map(|(s, vs)| (selectors.id(*s), values.id(*vs))) + .collect(), + elements: group + .elements + .iter() + .map(|g| name_group(g, selectors, values)) + .collect(), + } +} + +/// Every row's projection `q`: its grouping by array element, the order of +/// its pairs, and which pairs share a selector or a value selector across +/// the column. Catches a value equality, a grouping or an order the profile +/// does not state. +pub fn projections(world: &ObservedWorld) -> Vec { + let mut selectors = selector_names(world); + let mut values = Canon::new(); + world + .rows + .iter() + .map(|row| name_group(&row.q, &mut selectors, &mut values)) + .collect() +} + +fn pairs_of(group: &ObservedGroup, out: &mut Vec<(Tag16, Tag16)>) { + out.extend(group.pairs.iter().copied()); + for element in &group.elements { + pairs_of(element, out); + } +} + +/// For each selector, the rows grouped by the value selector they hold +/// there: the frequency histogram of every path the page describes, with +/// grouping and order taken away. +pub fn value_selector_partitions( + world: &ObservedWorld, +) -> BTreeMap>> { + let mut selectors = selector_names(world); + let mut values = Canon::new(); + let mut out: BTreeMap>> = BTreeMap::new(); + for (r, row) in world.rows.iter().enumerate() { + let mut pairs = Vec::new(); + pairs_of(&row.q, &mut pairs); + for (s, vs) in pairs { + out.entry(selectors.id(s)) + .or_default() + .entry(values.id(vs)) + .or_default() + .push(r); + } + } + out +} + +// ============================================================================= +// At rest: what the order of the entries says +// ============================================================================= + +/// The runs that start at each entry without a parent tag (other than the +/// first): under the sealed root these are the top-level fields, and since +/// everything beneath a field follows it, each run's length is that field's +/// subtree size. "The node count of each top-level field's subtree can be +/// read off a single row exactly." Parent tags are keyed by ordinal, so no +/// two entries share one and there are no equal-tag runs to read; this is +/// the contiguity structure there is. +pub fn chain_head_runs(world: &ObservedWorld) -> Vec> { + let mut canon = selector_names(world); + world + .rows + .iter() + .map(|row| { + let starts: Vec = (1..row.entries.len()) + .filter(|&j| row.entries[j].p.is_none()) + .collect(); + starts + .iter() + .enumerate() + .map(|(k, &start)| { + let end = starts.get(k + 1).copied().unwrap_or(row.entries.len()); + (canon.id(row.entries[start].s), end - start) + }) + .collect() + }) + .collect() +} + +/// For every two selectors that occur in one row, which of them came first +/// in the rows that hold both: `(a first, b first)`. Under a depth-first +/// order with members sorted by key, every two sibling members appear in one +/// order in every row, which is "the alphabetical order of sibling key +/// names"; a stored order independent of the keys would show both. +pub fn selector_precedence(world: &ObservedWorld) -> BTreeMap<(usize, usize), (bool, bool)> { + let mut canon = selector_names(world); + let mut out: BTreeMap<(usize, usize), (bool, bool)> = BTreeMap::new(); + for row in &world.rows { + let mut first: Vec = Vec::new(); + for entry in &row.entries { + let id = canon.id(entry.s); + if !first.contains(&id) { + first.push(id); + } + } + for (i, &a) in first.iter().enumerate() { + for &b in &first[i + 1..] { + let (key, a_first) = if a < b { + ((a, b), true) + } else { + ((b, a), false) + }; + let seen = out.entry(key).or_default(); + if a_first { + seen.0 = true; + } else { + seen.1 = true; + } + } + } + } + out +} + +/// An observer's reconstruction of every row's depth sequence from the +/// stored order alone, as the page says one can: "an observer comparing rows +/// learns" which selector's path sits under which. +/// +/// The attack: a selector's parent is among the selectors that precede it +/// in every row that holds it; of those it takes the one that itself has the +/// most such predecessors (the deepest candidate), and gives up where two +/// tie. An entry with no parent tag after the first is a top-level field +/// (depth 1). Each entry's depth is then its selector's chain of parents. +/// The attack is not always right (from order alone, a field that always +/// follows a sibling in every row reads as its child), and it does not need +/// to be: the feature is compared between worlds, and +/// `the_order_reveals_the_shape` checks that it is right on a corpus whose +/// rows vary. +pub fn depth_recovery(world: &ObservedWorld) -> Vec>> { + let mut canon = selector_names(world); + let rows: Vec> = world + .rows + .iter() + .map(|row| row.entries.iter().map(|e| canon.id(e.s)).collect()) + .collect(); + let mut candidates: HashMap> = HashMap::new(); + let mut heads: BTreeSet = BTreeSet::new(); + let mut roots: BTreeSet = BTreeSet::new(); + for (row, ids) in world.rows.iter().zip(&rows) { + for (k, &b) in ids.iter().enumerate() { + if k == 0 { + let _ = roots.insert(b); + } else if row.entries[k].p.is_none() { + let _ = heads.insert(b); + } + let before: BTreeSet = ids[..k].iter().copied().filter(|&a| a != b).collect(); + let _ = candidates + .entry(b) + .and_modify(|c| *c = c.intersection(&before).copied().collect()) + .or_insert(before); + } + } + let parent = |b: usize| -> Option { + let cands = candidates.get(&b)?; + let size = |a: &usize| candidates.get(a).map_or(0, BTreeSet::len); + let best = cands.iter().map(size).max()?; + let mut top = cands.iter().filter(|a| size(a) == best); + let first = *top.next()?; + top.next().is_none().then_some(first) + }; + let depth = |b: usize| -> Option { + let mut at = b; + // A chain of parents longer than there are selectors is a cycle. + for d in 0..=candidates.len() { + if roots.contains(&at) { + return Some(d); + } + if heads.contains(&at) { + return Some(d + 1); + } + at = parent(at)?; + } + None + }; + rows.iter() + .map(|ids| ids.iter().map(|&b| depth(b)).collect()) + .collect() +} + +// ============================================================================= +// Under a token: the walk, with the real hash +// ============================================================================= + +/// One entry a walk from a token reaches. +#[derive(Clone, Copy, Debug)] +struct Hit { + position: usize, + /// The position of the hit's parent, or `None` for a child of the + /// token's own node, which the walk never identifies. + parent: Option, + ordinal: u32, + depth: usize, + token: Tag16, + parent_token: Tag16, +} + +/// The children of the node whose token is `token`, as the database finds +/// them: the parent tag of ordinal `0, 1, 2, …` until the first miss. +fn children( + row: &ObservedRow, + index: &HashMap, + token: &Tag16, +) -> Vec<(usize, u32, Tag16)> { + let mut out = Vec::new(); + for o in 0..u32::try_from(row.entries.len()).unwrap_or(u32::MAX) { + let Some(&j) = index.get(&parent_tag(&row.header.iv, token, o)) else { + break; + }; + out.push((j, o, child_token(token, &row.entries[j].s, o))); + } + out +} + +fn parent_index(row: &ObservedRow) -> HashMap { + let mut index = HashMap::new(); + for (j, entry) in row.entries.iter().enumerate() { + if let Some(p) = entry.p { + let _ = index.entry(p).or_insert(j); + } + } + index +} + +/// Everything below `token` in `row`, depth first, bounded as the SQL walk +/// is: no deeper than the index seals, and no more hits than entries. +fn walk(row: &ObservedRow, token: &Tag16) -> Vec { + fn go( + row: &ObservedRow, + index: &HashMap, + token: &Tag16, + parent: Option, + depth: usize, + out: &mut Vec, + ) { + if depth > 128 || out.len() >= row.entries.len() { + return; + } + for (j, o, child) in children(row, index, token) { + if out.len() >= row.entries.len() { + return; + } + out.push(Hit { + position: j, + parent, + ordinal: o, + depth, + token: child, + parent_token: *token, + }); + go(row, index, &child, Some(j), depth + 1, out); + } + } + let index = parent_index(row); + let mut out = Vec::new(); + go(row, &index, token, None, 1, &mut out); + out +} + +fn subtree_queries(world: &ObservedWorld) -> Vec<(usize, Tag16, &[ObservedGate])> { + world + .queries + .iter() + .enumerate() + .filter_map(|(i, q)| match q { + ObservedQuery::Subtree { t, gates } => Some((i, *t, gates.as_slice())), + _ => None, + }) + .collect() +} + +/// One entry a token links: its position, its parent's position (`None` +/// for a child of the token's own node) and its depth below that node. +pub type Link = (usize, Option, usize); + +/// For every subtree read and every row, what the token links: each entry +/// the walk from the token reaches, its parent among them, and its depth +/// below the token's node, found by recomputing the real parent tags over +/// the row's `iv`. Catches a token that reaches more (or less) than the +/// profile's reach clause says: the page says a token covers its node's +/// whole subtree, in every row, and not only the path asked for (#1360). +pub fn token_reach(world: &ObservedWorld) -> Vec>> { + subtree_queries(world) + .into_iter() + .map(|(_, t, _)| { + world + .rows + .iter() + .map(|row| { + let mut hits: Vec<_> = walk(row, &t) + .into_iter() + .map(|h| (h.position, h.parent, h.depth)) + .collect(); + hits.sort_unstable(); + hits + }) + .collect() + }) + .collect() +} + +/// What the ordinals a walk probes say, in the two forms that do not depend +/// on the keyed rank's random order: how many reached nodes have children +/// that all share one selector (an array's elements, or an object's one +/// member) whose ordinals do not run in stored order, and every case where +/// one parent token ranks two child selectors one way in one row and the +/// other way in another. Both are zero for the real scheme: an element's +/// ordinal is its position, and a member's rank is a keyed function of its +/// parent's token and its key. +pub fn ordinal_patterns(world: &ObservedWorld) -> (usize, Vec<(usize, usize, usize)>) { + let mut selectors = selector_names(world); + let mut tokens = Canon::new(); + let mut out_of_order = 0; + let mut ranked: BTreeMap<(usize, usize, usize), BTreeSet> = BTreeMap::new(); + for (_, t, _) in subtree_queries(world) { + for row in &world.rows { + let hits = walk(row, &t); + let mut by_parent: BTreeMap<(Option, Tag16), Vec> = BTreeMap::new(); + for hit in hits { + by_parent + .entry((hit.parent, hit.parent_token)) + .or_default() + .push(hit); + } + for ((_, parent_token), mut kids) in by_parent { + kids.sort_by_key(|h| h.position); + let same = kids + .windows(2) + .all(|w| row.entries[w[0].position].s == row.entries[w[1].position].s); + if same + && kids + .iter() + .enumerate() + .any(|(i, h)| h.ordinal as usize != i) + { + out_of_order += 1; + } + if !same { + let parent = tokens.id(parent_token); + for a in &kids { + for b in &kids { + let (sa, sb) = ( + selectors.id(row.entries[a.position].s), + selectors.id(row.entries[b.position].s), + ); + if sa < sb { + let _ = ranked + .entry((parent, sa, sb)) + .or_default() + .insert(a.ordinal < b.ordinal); + } + } + } + } + } + } + } + let inconsistent = ranked + .into_iter() + .filter(|(_, seen)| seen.len() > 1) + .map(|(key, _)| key) + .collect(); + (out_of_order, inconsistent) +} + +/// Whether the members a token's holder finds directly under a token it was +/// sent carry ordinals in their stored order, which is their keys' sorted +/// order: the question decision 11 (the keyed member rank) is there to make +/// the answer to "no". Reported only where chance cannot answer it. +/// +/// The bound: a member's ordinal is its rank under a keyed function of its +/// parent's token and its key, so for one token the members of its largest +/// observed object fall in key order with probability `1 / k!` for `k` +/// members, and distinct tokens are independent keyed inputs. All of them +/// fall in key order by chance with probability at most the product of +/// `1 / k!` over the tokens sent. The feature is `None` unless that product +/// is below `2^-40`; above it, real and simulated could disagree by chance. +/// Only objects directly under a token sent are counted, since which deeper +/// nodes share a derived token depends on the ranks themselves. +pub fn member_ordinals_follow_key_order(world: &ObservedWorld) -> Option { + let mut chance = 1.0f64; + let mut all_in_order = true; + let mut sent = BTreeSet::new(); + for (_, t, _) in subtree_queries(world) { + if !sent.insert(t) { + continue; + } + let mut widest = 0usize; + for row in &world.rows { + let mut kids: Vec = walk(row, &t) + .into_iter() + .filter(|h| h.parent.is_none()) + .collect(); + let distinct: BTreeSet = + kids.iter().map(|h| row.entries[h.position].s).collect(); + if kids.len() < 2 || distinct.len() != kids.len() { + continue; + } + widest = widest.max(kids.len()); + kids.sort_by_key(|h| h.position); + if kids + .iter() + .enumerate() + .any(|(i, h)| h.ordinal as usize != i) + { + all_in_order = false; + } + } + chance /= (1..=widest).map(|k| k as f64).product::(); + } + (chance < 2f64.powi(-40)).then_some(all_in_order) +} + +/// Which subtree reads' tokens the database can derive from which others': +/// `(i, j)` when the token sent by query `j` is one the walk from query +/// `i`'s token derives in some row, or is the child token of `i`'s under +/// any selector the column holds and an ordinal below 8 (a bound on what +/// the harness tries, not on what the database can). The page: a token's +/// holder can "derive the tokens of the nodes beneath". Catches tokens that +/// are not chained (a positional token unrelated to its head's) as well as +/// tokens chained where the profile says they are not. +pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { + let queries = subtree_queries(world); + let selectors: BTreeSet = world + .rows + .iter() + .flat_map(|r| r.entries.iter().map(|e| e.s)) + .collect(); + let mut out = BTreeSet::new(); + for &(i, ti, _) in &queries { + let mut derived: BTreeSet = world + .rows + .iter() + .flat_map(|row| walk(row, &ti).into_iter().map(|h| h.token)) + .collect(); + for s in &selectors { + for o in 0..8 { + let _ = derived.insert(child_token(&ti, s, o)); + } + } + for &(j, tj, _) in &queries { + if i != j && derived.contains(&tj) { + let _ = out.insert((i, j)); + } + } + } + out +} + +/// What each subtree read returns from each row, as the SQL walk computes +/// it: from the token, through each gate (a child carrying the gate's +/// selector, or the child at ordinal `n` carrying it), then each node +/// reached with everything beneath it; with no gates, everything beneath the +/// token's node. +pub fn gated_walk_results(world: &ObservedWorld) -> Vec>> { + subtree_queries(world) + .into_iter() + .map(|(_, t, gates)| { + world + .rows + .iter() + .map(|row| { + let index = parent_index(row); + let mut frontier: Vec<(Option, Tag16)> = vec![(None, t)]; + for gate in gates { + let mut next = Vec::new(); + for (_, token) in &frontier { + let kids = children(row, &index, token); + match gate { + ObservedGate::Selector(s) => next.extend( + kids.iter() + .filter(|(j, _, _)| row.entries[*j].s == *s) + .map(|(j, _, child)| (Some(*j), *child)), + ), + ObservedGate::Slot(n, s) => next.extend( + kids.get(*n as usize) + .filter(|(j, _, _)| row.entries[*j].s == *s) + .map(|(j, _, child)| (Some(*j), *child)), + ), + } + } + frontier = next; + } + let mut rows = Vec::new(); + for (reached, token) in frontier { + rows.extend(reached); + rows.extend(walk(row, &token).into_iter().map(|h| h.position)); + } + rows.sort_unstable(); + rows + }) + .collect() + }) + .collect() +} + +// ============================================================================= +// The queries +// ============================================================================= + +/// A query term with its tags named. Read only through `Debug`. +#[derive(Debug)] +#[allow(dead_code)] +pub enum NamedQuery { + Extract(usize), + Subtree(usize, Vec<(Option, usize)>), + Contains(NamedGroup, Option), + Equals(NamedGroup), + Order(usize, usize), +} + +/// Every query term, its tags named across the column and the other terms: +/// which terms repeat (a query is deterministic), which selectors a term +/// shares with the stored rows and with other terms (two paths that differ +/// only in an array position send one selector), a gate's position in the +/// clear, and a needle's shape. +pub fn transcripts(world: &ObservedWorld) -> Vec { + let mut selectors = selector_names(world); + let mut values = Canon::new(); + for row in &world.rows { + let _ = name_group(&row.q, &mut selectors, &mut values); + } + let mut tokens = Canon::new(); + let mut terms = Canon::new(); + world + .queries + .iter() + .map(|q| match q { + ObservedQuery::Extract { s } => NamedQuery::Extract(selectors.id(*s)), + ObservedQuery::Subtree { t, gates } => NamedQuery::Subtree( + tokens.id(*t), + gates + .iter() + .map(|g| match g { + ObservedGate::Selector(s) => (None, selectors.id(*s)), + ObservedGate::Slot(n, s) => (Some(*n), selectors.id(*s)), + }) + .collect(), + ), + ObservedQuery::Contains { needle, alt } => NamedQuery::Contains( + name_group(needle, &mut selectors, &mut values), + alt.as_ref() + .map(|a| name_group(a, &mut selectors, &mut values)), + ), + ObservedQuery::Equals { needle } => { + NamedQuery::Equals(name_group(needle, &mut selectors, &mut values)) + } + ObservedQuery::Order { s, op } => { + NamedQuery::Order(selectors.id(*s), terms.id(op.clone())) + } + }) + .collect() +} + +/// For every extraction, the positions in each row whose selector it sent: +/// which rows `->` answers, and with how many entries. +pub fn extraction_matches(world: &ObservedWorld) -> Vec>> { + world + .queries + .iter() + .filter_map(|q| match q { + ObservedQuery::Extract { s } => Some( + world + .rows + .iter() + .map(|row| { + (0..row.entries.len()) + .filter(|&j| row.entries[j].s == *s) + .collect() + }) + .collect(), + ), + _ => None, + }) + .collect() +} + +/// `jsonb @>` over two projections, as the server decides it. +fn contains(stored: &ObservedGroup, needle: &ObservedGroup) -> bool { + needle.pairs.iter().all(|p| stored.pairs.contains(p)) + && needle + .elements + .iter() + .all(|n| stored.elements.iter().any(|s| contains(s, n))) +} + +/// For every containment and equality needle, the rows it matches. +pub fn containment_matches(world: &ObservedWorld) -> Vec> { + world + .queries + .iter() + .filter_map(|q| { + let (needle, alt) = match q { + ObservedQuery::Contains { needle, alt } => (needle, alt.as_ref()), + ObservedQuery::Equals { needle } => (needle, None), + _ => return None, + }; + Some( + (0..world.rows.len()) + .filter(|&r| { + let q = &world.rows[r].q; + contains(q, needle) || alt.is_some_and(|a| contains(q, a)) + }) + .collect(), + ) + }) + .collect() +} + +/// Every ordering term in the world, stored and sent, by its rank in byte +/// order (the order the SQL compares them in), ties sharing a rank. The +/// stored terms' ranks by position, then each bound's. +#[allow(clippy::type_complexity)] +pub fn ordering_term_order(world: &ObservedWorld) -> (Vec>>, Vec) { + let mut all: Vec<&[u8]> = world + .rows + .iter() + .flat_map(|r| r.entries.iter().filter_map(|e| e.op.as_deref())) + .collect(); + all.extend(world.queries.iter().filter_map(|q| match q { + ObservedQuery::Order { op, .. } => Some(op.as_slice()), + _ => None, + })); + all.sort_unstable(); + all.dedup(); + let rank = |op: &[u8]| all.binary_search(&op).expect("collected above"); + ( + world + .rows + .iter() + .map(|r| { + r.entries + .iter() + .map(|e| e.op.as_deref().map(rank)) + .collect() + }) + .collect(), + world + .queries + .iter() + .filter_map(|q| match q { + ObservedQuery::Order { op, .. } => Some(rank(op)), + _ => None, + }) + .collect(), + ) +} + +/// Every ordering term's length, stored and sent. +#[allow(clippy::type_complexity)] +pub fn ordering_term_lengths(world: &ObservedWorld) -> (Vec>>, Vec) { + ( + world + .rows + .iter() + .map(|r| { + r.entries + .iter() + .map(|e| e.op.as_ref().map(Vec::len)) + .collect() + }) + .collect(), + world + .queries + .iter() + .filter_map(|q| match q { + ObservedQuery::Order { op, .. } => Some(op.len()), + _ => None, + }) + .collect(), + ) +} + +/// For every ordering bound, the entries `col -> s >= bound` holds for: the +/// ones at its selector whose term is at least the bound's. +pub fn ordering_matches(world: &ObservedWorld) -> Vec>> { + world + .queries + .iter() + .filter_map(|q| match q { + ObservedQuery::Order { s, op } => Some( + world + .rows + .iter() + .map(|row| { + (0..row.entries.len()) + .filter(|&j| { + let e = &row.entries[j]; + e.s == *s && e.op.as_ref().is_some_and(|stored| stored >= op) + }) + .collect() + }) + .collect(), + ), + _ => None, + }) + .collect() +} + +/// The positions and depths a walk from `token` reaches in `row`, in the +/// walk's order: what `JsonDocument::subtree` returns. +pub fn walk_positions(row: &ObservedRow, token: &Tag16) -> Vec<(usize, usize)> { + walk(row, token) + .into_iter() + .map(|h| (h.position, h.depth)) + .collect() +} diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs new file mode 100644 index 000000000..8020b9743 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -0,0 +1,208 @@ +//! An executable leakage profile for the JSON index, EQL v4's `json_tree`. +//! +//! `packages/eql/docs/reference/json-tree-leakage.md` says, in prose, what a +//! database holding a `json_tree` column learns from the stored rows and +//! from each query. This module turns that page into a checked claim. If the +//! real scheme reveals anything the claim leaves out, a test here fails and +//! names the observation that gave it away. +//! +//! # The pieces +//! +//! ```text +//! documents, queries ──► profile::leak ──► Leakage ──► simulator::simulate ──► wire JSON ─┐ +//! (plaintext) (L: no key) (+ seeded randomness) │ +//! ├─► observe ──► ObservedWorld ──► distinguishers +//! documents, queries ──► real::world ─────────────────────────────────────────► wire JSON ─┘ +//! (the crate, under the fake key source) +//! ``` +//! +//! - [`profile`] is `L`, the claim: [`profile::l_store`] (what the stored +//! rows reveal) and [`profile::l_query`] (what one query reveals), both +//! functions of the plaintext alone. Every keyed value is an opaque integer +//! that is equal exactly where the real values are equal. Each field of the +//! leakage types is one clause, and its doc comment quotes the sentence of +//! the page it encodes. The two take a shared [`profile::Ids`] beside the +//! documents, so that a query's selector and a stored selector get the same +//! name when the real bytes are equal. +//! - [`simulator`] builds a world in the wire form from a +//! [`profile::Leakage`] and randomness, with no plaintext and no key: +//! random bytes for every name, random ciphertexts of the stated length, +//! ordering terms laid out in the stated order, and parent tags chained with +//! the real hash below every token a query sends, as far as the profile +//! says the token reaches. +//! - [`real`] seals the same documents and derives the same query terms +//! through the crate's public API, and serializes them. +//! - [`observe`] parses either world's wire JSON into one +//! [`observe::ObservedWorld`]. +//! - [`distinguishers`] are pure functions of an observed world: what an +//! observer holding the column and the query terms could compute. Each +//! feature is exact (byte strings renamed to integers in order of first +//! appearance), so a real and a simulated world are compared for equality, +//! with no statistics. One feature needs a probability bound and carries it +//! in its doc comment ([`distinguishers::member_ordinals_follow_key_order`]). +//! - [`mutants`] are profiles with one clause dropped or coarsened. +//! - [`corpus`] holds the handcrafted columns and a proptest generator of +//! random ones over a shared key vocabulary. +//! - [`chain`] is the token chain's two public hash functions, the only part +//! of the real scheme the harness evaluates itself. +//! +//! # The checks ([`calibration`]) +//! +//! - **The positive control.** The simulator, run from the full profile, is +//! told from the real scheme by no distinguisher, on the handcrafted +//! columns and on 48 random ones. If this fails, the real scheme reveals +//! something `L` does not state: that is a finding against the page (or +//! against the code), and the failure names the distinguisher and prints +//! both features. +//! - **The page as written.** `L` carries one clause the page does not state, +//! marked **beyond the page** in [`profile::Clauses`]. +//! [`profile::Clauses::page_only`] turns it off, and a test holds that the +//! result is then told from the real scheme, by exactly the distinguisher +//! that sees it. +//! - **The calibration.** Each mutant profile drops one clause. The test +//! expects some distinguisher to tell each mutant's world from the real +//! one, and prints a table of which (run with `--no-capture`). A mutant +//! nothing notices goes in its `UNNOTICED` list with the reason; there is +//! none today. This is what shows the distinguishers can see what the +//! clauses describe, so that the positive control passing means something. +//! - **The harness itself.** The chain functions agree with the crate's own +//! walk (`JsonDocument::subtree`), and the order-only attack in +//! [`distinguishers::depth_recovery`] recovers every row's depth sequence +//! on a column whose rows vary, which is the page's claim that the stored +//! order reveals each row's shape. +//! +//! # What the profile found against the page +//! +//! 1. **The ordering term's length is not on the page.** A CLLW OPE term is +//! `8 × n + 1` bytes for an `n`-byte encoding: every number's term is 65 +//! bytes, and a string's is `8 × (len + 1) + 1`. So at an ordered path +//! the term says which of number or string the leaf is (the page says +//! only that its presence says "a number or a string"), and repeats the +//! string's length. `the_ordering_term_length_separates_it_from_the_real_scheme`. +//! 2. **The page contradicts itself on equal values at two paths.** "Not +//! revealed at rest: […] whether two different paths hold the same value" +//! is false once two paths are ordered: the `op` row's "leaves at two +//! ordered paths compare with each other too" includes equality, and the +//! real terms of equal values at two ordered paths are equal. `L` follows +//! the `op` row; the mutant that follows the summary sentence +//! (`ordering_terms_per_path`) is told apart by `ordering_term_order`. +//! 3. **"The sorted order of their key names" is the keys' UTF-8 byte +//! order.** `jsonb` sorts shorter keys first; read that way, the profile +//! is told apart (`given_the_jsonb_reading_of_the_key_order`). A reader +//! inferring key names from the order needs the byte order. +//! +//! Where the page is vague and `L` is as precise as the code forces, the +//! clause says so: what a token's holder sees in the ordinals it probes +//! ([`profile::EntryLeakage::step`]), the needle's `alt` +//! ([`profile::QueryLeakage::Contains`]), and the fixed parts of a +//! ciphertext's length ([`profile::ciphertext_len`]). +//! +//! The two issues the profile was built to check come out as the page +//! states them. The stored order reveals each row's shape (#1357): the order +//! clause is load-bearing (`order_random_permutation`, `members_unsorted`), +//! and the depth attack works. And a subtree read's token reaches its whole +//! top-level field in every row, not only the path asked (#1360): the +//! mutant that narrows the reach to the path (`token_reaches_named_nodes_only`) +//! is told apart by `token_reach`, which walks the real parent tags from the +//! real token. +//! +//! # When the format changes +//! +//! The format is about to change (`docs/plans/2026-10-17-json-tree-index.md`, +//! "Design change (2026-10-24)"): member tokens lose their ordinal, every +//! entry gains a self tag `x`, the client expands wildcards and gates go +//! away. `L` and the real-side adapter are expected to change with it, and +//! with the leakage page; the distinguishers should mostly not. To add or +//! change a clause: +//! +//! 1. Change the page first, or at least decide what it will say. +//! 2. Change [`profile`]: a field on the leakage type, with the page's +//! sentence quoted in its doc comment, computed from the plaintext. +//! 3. Make the [`simulator`] produce it from that field and randomness only. +//! For `x`, that is `H8("x" ‖ iv ‖ t(node))` on every entry whose token +//! some query's reach covers, and random bytes elsewhere, exactly as `p` +//! is handled now. +//! 4. Teach [`observe`] the new wire field, and [`real`] any new API. +//! 5. Add a distinguisher if no existing one would see the clause, and a +//! mutant that drops the clause, and check the calibration catches it. +//! +//! # Reading a failure +//! +//! - **Positive control fails.** The real scheme shows something the profile +//! does not state, or states differently. The message names each +//! distinguisher that separated the worlds, with the real and the +//! simulated feature. Decide whether the page or the code is wrong. Never +//! widen `L` silently to make it pass: a clause the page does not state +//! goes in as **beyond the page** ([`profile::Clauses`]), with a test like +//! the one for the ordering term's length, so the gap stays visible until +//! the page closes it. +//! - **Calibration fails for a mutant nothing noticed.** Either the clause +//! is not load-bearing on the calibration columns (add a column that +//! exercises it, as the wide-objects column does for member ranks), or no +//! distinguisher can see it (write one). If neither is possible, list the +//! mutant in `UNNOTICED` with the reason. Do not delete the clause. +//! - **Calibration fails for a listed mutant that is now noticed.** Remove it +//! from the list: something got sharper. +//! +//! # A tension this module states and does not resolve +//! +//! PR #1358's read check (`JsonSubtreeKey::open`) refuses a walk's rows +//! unless each subtree's rows sit at consecutive positions `j` in `sv`, which +//! is what a subtree is in a depth-first stored order, and `j` is bound into +//! every ciphertext's AAD. That consecutiveness is exactly the property +//! #1357 wants gone: a stored order independent of the shape (a keyed +//! permutation of `sv`, say) would scatter a subtree's rows. The check and +//! "order independent of shape" cannot both hold as written: one of them +//! has to give, and which is the owner's decision. The design change's +//! first open question discusses the same check from the other side. +//! +//! # Out of reach +//! +//! - **The inside of an ordering term.** CLLW OPE terms share leading bytes +//! when the encoded values do (the page says so), with a carry that can +//! move a byte by one. The simulator reproduces a term's presence, length, +//! equality and order, not its prefix structure, and no distinguisher +//! looks at it: the prefix a pair of terms shares is not an exact function +//! of the plaintexts, so it is a statistical question. +//! - **The keyed member rank as a permutation.** Under a token, a member's +//! ordinal is a keyed rank. Its relation to the stored (key) order is a +//! random permutation, decidable exactly only where enough wide objects sit +//! under the tokens sent; elsewhere the feature reports nothing. For the +//! same reason, which descendants of a token share a derived token across +//! rows depends on the ranks, so no feature compares it. +//! - **The primitives.** Ciphertexts beyond their length, selectors, value +//! selectors and head tokens beyond their equality patterns: the profile +//! assumes AES-GCM and the PRFs do their job, and the simulator draws +//! random bytes for them. +//! - **The database's side channels.** Timing, which rows a query touches +//! beyond its result, the walk's budget fallback, and the `i` and `v` +//! envelope fields the SQL adds. One column at a time; no cross-column +//! linkage is modelled. +//! - **The key service.** The fake key source mints 32-byte tags; ZeroKMS's +//! tag length is its own, and a column constant either way. +//! +//! # Determinism +//! +//! Every simulation and every mutant draws from one seeded generator, and +//! the random columns come from a seeded proptest runner. The real side +//! draws data keys and nonces from the operating system; those reach a +//! feature only through equality patterns, which change only on a collision +//! of random bytes, below `n² / 2^64` for `n` eight-byte tags in a column +//! (about `10^-13` here). The fake key source's index key is fixed, so the +//! real member ranks are too. +//! +//! The profile sorts members itself rather than trusting `serde_json`'s +//! iteration order, so a build that unified serde_json's `preserve_order` +//! feature on (which would store the application's field order, #1357) fails +//! the positive control on the handcrafted rows whose keys are written out +//! of order. + +pub mod calibration; +pub mod chain; +pub mod corpus; +pub mod distinguishers; +pub mod mutants; +pub mod observe; +pub mod profile; +pub mod real; +pub mod simulator; diff --git a/packages/stack-encrypt/tests/json_leakage/mutants.rs b/packages/stack-encrypt/tests/json_leakage/mutants.rs new file mode 100644 index 000000000..c722e163d --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/mutants.rs @@ -0,0 +1,670 @@ +//! Mutant profiles: each drops or coarsens one clause of `L`, so that the +//! simulator built from it reveals less (or other) than the real scheme in +//! exactly that respect. The calibration runs each and expects some +//! distinguisher to notice; a clause whose mutant nothing notices is either +//! not load-bearing or has no distinguisher, and is listed in +//! `calibration.rs` with the reason. +//! +//! A mutant transforms a [`Leakage`] value. It never sees the plaintext, so +//! it can only coarsen what `L` already states, which is the point: the +//! mutant is "the profile without this clause". + +use std::collections::{BTreeMap, HashMap}; + +use proptest::prelude::RngCore; +use proptest::test_runner::TestRng; + +use super::profile::{ + EntryLeakage, GateLeak, GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step, AEAD_OVERHEAD, +}; + +/// One mutant: its name, the clause it drops (a field or sentence of the +/// profile), and the transformation. +pub struct Mutant { + pub name: &'static str, + pub clause: &'static str, + pub apply: fn(&mut Leakage, &mut TestRng), +} + +/// Fresh opaque names, far above any the profile hands out. +struct Fresh(usize); + +impl Fresh { + fn new() -> Self { + Self(1 << 40) + } + + fn next(&mut self) -> usize { + self.0 += 1; + self.0 + } +} + +fn shuffle(items: &mut [T], rng: &mut TestRng) { + for i in (1..items.len()).rev() { + let j = (rng.next_u64() % (i as u64 + 1)) as usize; + items.swap(i, j); + } +} + +fn groups_mut(group: &mut GroupLeakage, f: &mut impl FnMut(&mut GroupLeakage)) { + f(group); + for element in &mut group.elements { + groups_mut(element, f); + } +} + +/// Every projection in the leakage, stored and sent. +fn every_group(leakage: &mut Leakage, mut f: impl FnMut(&mut GroupLeakage)) { + for row in &mut leakage.rows { + groups_mut(&mut row.projection, &mut f); + } + for query in &mut leakage.queries { + match query { + QueryLeakage::Contains { needle, alt } => { + groups_mut(needle, &mut f); + if let Some(alt) = alt { + groups_mut(alt, &mut f); + } + } + QueryLeakage::Equals { needle } => groups_mut(needle, &mut f), + _ => {} + } + } +} + +/// Reorder a row's entries: `order[k]` is the old position of the entry now +/// at `k`. Parents and every query's positions follow. +fn reorder(leakage: &mut Leakage, r: usize, order: &[usize]) { + let mut new_at = vec![0; order.len()]; + for (k, &old) in order.iter().enumerate() { + new_at[old] = k; + } + let row = &mut leakage.rows[r]; + let old = std::mem::take(&mut row.entries); + row.entries = order + .iter() + .map(|&o| EntryLeakage { + parent: old[o].parent.map(|p| new_at[p]), + ..old[o].clone() + }) + .collect(); + for query in &mut leakage.queries { + if let QueryLeakage::Subtree { reach, named, .. } = query { + for at in reach[r].iter_mut().chain(named[r].iter_mut()) { + *at = new_at[*at]; + } + } + } +} + +fn children(row: &RowLeakage) -> Vec> { + let mut kids = vec![Vec::new(); row.entries.len()]; + for (i, entry) in row.entries.iter().enumerate() { + if let Some(p) = entry.parent { + kids[p].push(i); + } + } + kids +} + +// ============================================================================= +// At rest +// ============================================================================= + +fn selectors_all_distinct(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for row in &mut l.rows { + for entry in &mut row.entries { + entry.selector = fresh.next(); + } + } + every_group(l, |g| { + for pair in &mut g.pairs { + pair.0 = fresh.next(); + } + }); +} + +fn selectors_per_row_only(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for row in &mut l.rows { + let mut mine: HashMap = HashMap::new(); + for entry in &mut row.entries { + entry.selector = *mine.entry(entry.selector).or_insert_with(|| fresh.next()); + } + groups_mut(&mut row.projection, &mut |g| { + for pair in &mut g.pairs { + pair.0 = *mine.entry(pair.0).or_insert_with(|| fresh.next()); + } + }); + } +} + +/// Array positions kept in selectors, as if `$.a[0]` and `$.a[1]` had +/// different selectors: each element's selector is told apart by the +/// positions on its way down. +fn positions_in_selectors(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + let mut names: HashMap<(usize, Vec), usize> = HashMap::new(); + for row in &mut l.rows { + let positions: Vec> = (0..row.entries.len()) + .map(|i| { + let mut out = Vec::new(); + let mut at = Some(i); + while let Some(j) = at { + if let Step::Element(n) = row.entries[j].step { + out.push(n); + } + at = row.entries[j].parent; + } + out + }) + .collect(); + for (entry, positions) in row.entries.iter_mut().zip(positions) { + if !positions.is_empty() { + entry.selector = *names + .entry((entry.selector, positions)) + .or_insert_with(|| fresh.next()); + } + } + } +} + +fn entry_count_padded(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for row in &mut l.rows { + row.entries.push(EntryLeakage { + selector: fresh.next(), + parent: Some(0), + step: Step::Member, + head: None, + has_parent_tag: true, + ciphertext_len: AEAD_OVERHEAD + 10, + ordering: None, + }); + } +} + +fn order_random_permutation(l: &mut Leakage, rng: &mut TestRng) { + for r in 0..l.rows.len() { + let mut order: Vec = (1..l.rows[r].entries.len()).collect(); + shuffle(&mut order, rng); + order.insert(0, 0); + reorder(l, r, &order); + } +} + +/// Still depth-first, but an object's members in random order rather than +/// sorted by key. +fn members_unsorted(l: &mut Leakage, rng: &mut TestRng) { + for r in 0..l.rows.len() { + let kids = children(&l.rows[r]); + let mut order = Vec::new(); + let mut stack = vec![0]; + while let Some(i) = stack.pop() { + order.push(i); + let mut mine = kids[i].clone(); + if mine + .iter() + .all(|&k| l.rows[r].entries[k].step == Step::Member) + { + shuffle(&mut mine, rng); + } + stack.extend(mine.into_iter().rev()); + } + reorder(l, r, &order); + } +} + +fn parent_tag_on_every_entry(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + entry.has_parent_tag = true; + } + } +} + +fn parent_tag_on_no_entry(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + entry.has_parent_tag = false; + } + } +} + +fn ciphertext_length_bucketed(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + entry.ciphertext_len = entry.ciphertext_len.div_ceil(32) * 32; + } + } +} + +fn sealed_root_length_hidden(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + if row.sealed_root { + row.entries[0].ciphertext_len = AEAD_OVERHEAD + 6; + } + } +} + +fn value_selectors_all_distinct(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + every_group(l, |g| { + for pair in &mut g.pairs { + pair.1 = fresh.next(); + } + }); +} + +/// A value selector that names the path and nothing of the value. +fn value_selectors_path_only(l: &mut Leakage, _: &mut TestRng) { + every_group(l, |g| { + for pair in &mut g.pairs { + pair.1 = pair.0; + } + }); +} + +fn flatten_group(group: &mut GroupLeakage) { + let mut elements = std::mem::take(&mut group.elements); + for element in &mut elements { + flatten_group(element); + group.pairs.append(&mut element.pairs); + } +} + +fn projection_flat(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + flatten_group(&mut row.projection); + } +} + +fn projection_shuffled(l: &mut Leakage, rng: &mut TestRng) { + for row in &mut l.rows { + groups_mut(&mut row.projection, &mut |g| shuffle(&mut g.pairs, rng)); + } +} + +fn ordering_terms_absent(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + entry.ordering = None; + } + } +} + +fn ordering_terms_unordered(l: &mut Leakage, rng: &mut TestRng) { + shuffle(&mut l.ordering_ranks, rng); +} + +/// Every stored term a class of its own, so equal values no longer show, +/// with the order otherwise kept: ties broken at random. +fn ordering_terms_all_distinct(l: &mut Leakage, rng: &mut TestRng) { + let mut fresh = Fresh::new(); + let mut keyed: Vec<((usize, u64), usize)> = Vec::new(); + for row in &mut l.rows { + for entry in &mut row.entries { + if let Some(op) = &mut entry.ordering { + let class = fresh.next(); + keyed.push(((l.ordering_ranks[op.class], rng.next_u64()), class)); + op.class = class; + } + } + } + for query in &mut l.queries { + if let QueryLeakage::Order { bound, .. } = query { + let class = fresh.next(); + keyed.push(((l.ordering_ranks[bound.class], rng.next_u64()), class)); + bound.class = class; + } + } + keyed.sort_unstable(); + let ranks: BTreeMap = keyed + .into_iter() + .enumerate() + .map(|(rank, (_, class))| (class, rank)) + .collect(); + // Rebuild the rank table densely over the new classes. + let mut remap: HashMap = HashMap::new(); + let mut table = Vec::new(); + for (class, rank) in ranks { + let _ = remap.insert(class, table.len()); + table.push(rank); + } + for row in &mut l.rows { + for entry in &mut row.entries { + if let Some(op) = &mut entry.ordering { + op.class = remap[&op.class]; + } + } + } + for query in &mut l.queries { + if let QueryLeakage::Order { bound, .. } = query { + bound.class = remap[&bound.class]; + } + } + l.ordering_ranks = table; +} + +/// Equal values at two ordered paths get different terms: what the page's +/// summary sentence "Not revealed at rest: […] whether two different paths +/// hold the same value" would need, against its own `op` row ("leaves at +/// two ordered paths compare with each other too"). The order is kept, and +/// ties across paths are broken by path. +fn ordering_terms_per_path(l: &mut Leakage, _: &mut TestRng) { + let mut keyed: BTreeMap<(usize, usize), usize> = BTreeMap::new(); + for row in &l.rows { + for entry in &row.entries { + if let Some(op) = &entry.ordering { + let _ = keyed.insert((l.ordering_ranks[op.class], entry.selector), 0); + } + } + } + for query in &l.queries { + if let QueryLeakage::Order { selector, bound } = query { + let _ = keyed.insert((l.ordering_ranks[bound.class], *selector), 0); + } + } + // Classes in (rank, path) order, so a class's index is its new rank. + for (class, slot) in keyed.values_mut().enumerate() { + *slot = class; + } + let old_ranks = l.ordering_ranks.clone(); + for row in &mut l.rows { + for entry in &mut row.entries { + if let Some(op) = &mut entry.ordering { + op.class = keyed[&(old_ranks[op.class], entry.selector)]; + } + } + } + for query in &mut l.queries { + if let QueryLeakage::Order { selector, bound } = query { + bound.class = keyed[&(old_ranks[bound.class], *selector)]; + } + } + l.ordering_ranks = (0..keyed.len()).collect(); +} + +fn ordering_term_length_constant(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + if let Some(op) = &mut entry.ordering { + op.len = 65; + } + } + } + for query in &mut l.queries { + if let QueryLeakage::Order { bound, .. } = query { + bound.len = 65; + } + } +} + +/// A member's ordinal is its place in the stored (key) order rather than a +/// keyed rank: what the chain would be without decision 11. +fn member_ranks_follow_key_order(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + let kids = children(row); + for siblings in kids { + let members: Vec = siblings + .into_iter() + .filter(|&k| row.entries[k].step == Step::Member) + .collect(); + for (o, k) in members.into_iter().enumerate() { + row.entries[k].step = Step::Element(o as u64); + } + } + } +} + +fn sealed_flag_flipped(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + row.sealed_root = !row.sealed_root; + } +} + +fn data_key_shared(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + row.data_key = 0; + } +} + +fn keyset_per_row(l: &mut Leakage, _: &mut TestRng) { + for (r, row) in l.rows.iter_mut().enumerate() { + row.keyset = r; + } +} + +/// Head tokens per row rather than column-wide. +fn tokens_per_row(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for row in &mut l.rows { + for entry in &mut row.entries { + if entry.head.is_some() { + entry.head = Some(fresh.next()); + } + } + } +} + +// ============================================================================= +// Queries +// ============================================================================= + +fn extraction_selector_fresh(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for query in &mut l.queries { + if let QueryLeakage::Extract { selector } = query { + *selector = fresh.next(); + } + } +} + +/// A query that repeats an earlier one sends unrelated terms: queries are +/// not deterministic. +fn queries_nondeterministic(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + let mut seen: Vec = Vec::new(); + for query in &mut l.queries { + if seen.contains(query) { + match query { + QueryLeakage::Extract { selector } => *selector = fresh.next(), + QueryLeakage::Subtree { token, gates, .. } => { + token.head = fresh.next(); + for gate in gates { + match gate { + GateLeak::Selector(s) | GateLeak::Slot(_, s) => *s = fresh.next(), + } + } + } + QueryLeakage::Contains { needle, .. } | QueryLeakage::Equals { needle } => { + groups_mut(needle, &mut |g| { + for pair in &mut g.pairs { + *pair = (fresh.next(), fresh.next()); + } + }); + } + QueryLeakage::Order { selector, .. } => *selector = fresh.next(), + } + } else { + seen.push(query.clone()); + } + } +} + +/// No query's token links any entry: parent tags are random everywhere, as +/// if they were not chained from the tokens at all. +fn parent_tags_unchained(l: &mut Leakage, _: &mut TestRng) { + for query in &mut l.queries { + if let QueryLeakage::Subtree { reach, .. } = query { + for row in reach { + row.clear(); + } + } + } +} + +/// A token links only what the read names: the nodes on the path below the +/// token's node and the subtrees of the nodes the path reaches. What the +/// page would say if the subtree read revealed no more than it asked for. +fn token_reaches_named_nodes_only(l: &mut Leakage, _: &mut TestRng) { + for q in 0..l.queries.len() { + let QueryLeakage::Subtree { reach, named, .. } = &l.queries[q] else { + continue; + }; + let narrowed: Vec> = reach + .iter() + .zip(named) + .enumerate() + .map(|(r, (reach, named))| { + let row = &l.rows[r]; + reach + .iter() + .copied() + .filter(|&i| { + named.iter().any(|&n| { + // `i` is on the path to `n`, is `n`, or is below it. + let under = |a: usize, b: usize| { + let mut at = Some(a); + while let Some(j) = at { + if j == b { + return true; + } + at = row.entries[j].parent; + } + false + }; + under(n, i) || under(i, n) + }) + }) + .collect() + }) + .collect(); + if let QueryLeakage::Subtree { reach, .. } = &mut l.queries[q] { + *reach = narrowed; + } + } +} + +/// A token for a path through array positions is not chained from its +/// head's: the client would send an unrelated token. +fn position_tokens_unchained(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for query in &mut l.queries { + if let QueryLeakage::Subtree { token, .. } = query { + if !token.positions.is_empty() { + token.head = fresh.next(); + token.positions.clear(); + } + } + } +} + +fn slot_positions_hidden(l: &mut Leakage, _: &mut TestRng) { + for query in &mut l.queries { + if let QueryLeakage::Subtree { gates, .. } = query { + for gate in gates { + if let GateLeak::Slot(n, _) = gate { + *n = 0; + } + } + } + } +} + +fn gates_dropped(l: &mut Leakage, _: &mut TestRng) { + for query in &mut l.queries { + if let QueryLeakage::Subtree { gates, .. } = query { + gates.clear(); + } + } +} + +fn needle_flat(l: &mut Leakage, _: &mut TestRng) { + for query in &mut l.queries { + if let QueryLeakage::Contains { needle, .. } | QueryLeakage::Equals { needle } = query { + flatten_group(needle); + } + } +} + +fn needle_alt_dropped(l: &mut Leakage, _: &mut TestRng) { + for query in &mut l.queries { + if let QueryLeakage::Contains { alt, .. } = query { + *alt = None; + } + } +} + +/// A bound whose place among the stored terms is random. +fn bound_rank_random(l: &mut Leakage, rng: &mut TestRng) { + let top = l.ordering_ranks.len() as u64 + 1; + for q in 0..l.queries.len() { + if let QueryLeakage::Order { bound, .. } = &mut l.queries[q] { + bound.class = l.ordering_ranks.len(); + l.ordering_ranks + .push((rng.next_u64() % top) as usize * 2 + 1); + } + } + // Stored ranks on the even numbers, so a random bound never ties one. + let stored = l.ordering_ranks.len() + - l.queries + .iter() + .filter(|q| matches!(q, QueryLeakage::Order { .. })) + .count(); + for rank in &mut l.ordering_ranks[..stored] { + *rank *= 2; + } +} + +macro_rules! mutants { + ($($name:ident => $clause:literal),+ $(,)?) => { + /// Every mutant, in a fixed order. + pub fn all() -> Vec { + vec![$(Mutant { + name: stringify!($name), + clause: $clause, + apply: $name, + }),+] + } + }; +} + +mutants!( + selectors_all_distinct => "EntryLeakage::selector (equality)", + selectors_per_row_only => "EntryLeakage::selector (\"the same in every row\")", + positions_in_selectors => "EntryLeakage::selector (\"Array elements share their array's selector\")", + entry_count_padded => "RowLeakage::entries (\"the number of entries\")", + order_random_permutation => "RowLeakage::entries (the stored order)", + members_unsorted => "RowLeakage::entries (\"an object's members in the sorted order of their key names\")", + parent_tag_on_every_entry => "EntryLeakage::has_parent_tag", + parent_tag_on_no_entry => "EntryLeakage::has_parent_tag", + ciphertext_length_bucketed => "EntryLeakage::ciphertext_len", + sealed_root_length_hidden => "EntryLeakage::ciphertext_len (\"the root entry's length is the document's length as JSON\")", + value_selectors_all_distinct => "GroupLeakage::pairs (value selector equality)", + value_selectors_path_only => "GroupLeakage::pairs (\"a keyed hash of the path and the value\")", + projection_flat => "GroupLeakage::elements (\"nested by array element\")", + projection_shuffled => "GroupLeakage::pairs (\"in the same order\")", + ordering_terms_absent => "EntryLeakage::ordering (presence)", + ordering_terms_unordered => "Leakage::ordering_ranks (\"the order of that leaf's value among every row\")", + ordering_terms_all_distinct => "OrderingLeak::class (\"which rows hold equal values\")", + ordering_term_length_constant => "OrderingLeak::len (beyond the page)", + ordering_terms_per_path => "OrderingLeak::class across paths (\"leaves at two ordered paths compare with each other too\")", + member_ranks_follow_key_order => "EntryLeakage::step (a member's ordinal is its keyed rank)", + sealed_flag_flipped => "RowLeakage::sealed_root", + data_key_shared => "RowLeakage::data_key (\"the data key's identifier (unique to the row)\")", + keyset_per_row => "RowLeakage::keyset (\"the keyset the data key was minted under\")", + tokens_per_row => "EntryLeakage::head (\"Tokens are column-wide\")", + extraction_selector_fresh => "QueryLeakage::Extract (\"The selector of the path asked\")", + queries_nondeterministic => "every query term is a function of the query (\"Tokens are column-wide and do not expire\")", + parent_tags_unchained => "QueryLeakage::Subtree::reach (a token links anything at all)", + token_reaches_named_nodes_only => "QueryLeakage::Subtree::reach (\"its node's subtree in every row\", not the path asked)", + position_tokens_unchained => "TokenName::positions (\"while the path goes on through array positions, the element they reach\")", + slot_positions_hidden => "GateLeak::Slot (\"with the position n in the clear\")", + gates_dropped => "QueryLeakage::Subtree::gates (\"one gate per step below it\")", + needle_flat => "QueryLeakage::Contains::needle (\"its element grouping\")", + needle_alt_dropped => "QueryLeakage::Contains::alt", + bound_rank_random => "QueryLeakage::Order::bound (\"the bound's place in the order of the stored values\")", +); diff --git a/packages/stack-encrypt/tests/json_leakage/observe.rs b/packages/stack-encrypt/tests/json_leakage/observe.rs new file mode 100644 index 000000000..4a405badd --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/observe.rs @@ -0,0 +1,256 @@ +//! The observed world: everything the database holds or is sent, parsed from +//! the wire form into one shape that the real scheme and the simulator both +//! produce. Every distinguisher reads an [`ObservedWorld`] and nothing else, +//! so it runs on the real and the simulated world unchanged. +//! +//! The parser takes the wire JSON, not the crate's types: the real side +//! serializes a sealed `JsonDocument` and each query term with `serde_json`, +//! the simulator writes the same JSON directly, and both go through +//! [`ObservedWorld::from_wire`]. A simulator that wrote a field the real form +//! lacks, or the other way round, shows in [`ObservedRow::fields`] and +//! [`ObservedEntry::fields`]. + +use base64ct::{Base64, Encoding}; +use serde_json::Value; + +use super::chain::{Tag16, Tag8}; + +/// The header `h`, at the byte offsets its frozen layout gives. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObservedHeader { + pub version: u8, + pub flags: u8, + pub epoch: [u8; 4], + pub keyset: Tag16, + pub iv: Tag16, + pub tag: Vec, + /// Bytes after the tag its length field promises: none in a well-formed + /// header. + pub trailing: usize, +} + +/// One entry of `sv`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObservedEntry { + pub s: Tag16, + pub c: Vec, + pub p: Option, + pub op: Option>, + /// The entry's wire keys. + pub fields: Vec, +} + +/// One group of a projection: a stored `q`, or a needle. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ObservedGroup { + pub pairs: Vec<(Tag16, Tag16)>, + pub elements: Vec, +} + +/// One stored row. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObservedRow { + /// The document's wire keys. + pub fields: Vec, + pub header: ObservedHeader, + pub entries: Vec, + pub q: ObservedGroup, +} + +/// Which SQL surface a query term was sent to. The database knows which +/// function it is evaluating, so the kind is part of what it observes. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum QueryKind { + /// `->`, `json_tree_path_query`: a selector. + Extract, + /// `json_tree_subtree`, `json_tree_subtree_at`: a token and gates. + Subtree, + /// `@>`: a containment needle, with its `alt`. + Contains, + /// `AtPath`: a containment needle with no `alt`. + Equals, + /// `AtPath`: a selector and an ordering term. + Order, +} + +/// One gate of a subtree read. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ObservedGate { + Selector(Tag16), + Slot(u32, Tag16), +} + +/// One query term, as the database receives it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ObservedQuery { + Extract { + s: Tag16, + }, + Subtree { + t: Tag16, + gates: Vec, + }, + Contains { + needle: ObservedGroup, + alt: Option, + }, + Equals { + needle: ObservedGroup, + }, + Order { + s: Tag16, + op: Vec, + }, +} + +/// The column as the database sees it: its rows, then the query terms it was +/// sent, in order. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObservedWorld { + pub rows: Vec, + pub queries: Vec, +} + +impl ObservedWorld { + /// Parse a world from its wire form: each row as its stored JSON, each + /// query as the JSON its term serializes to. + pub fn from_wire(rows: &[Value], queries: &[(QueryKind, Value)]) -> Self { + Self { + rows: rows.iter().map(row).collect(), + queries: queries.iter().map(|(kind, q)| query(*kind, q)).collect(), + } + } +} + +fn b64(text: &Value) -> Vec { + let text = text.as_str().expect("a base64 field is a string"); + Base64::decode_vec(text).expect("padded standard base64") +} + +fn b64_16(text: &Value) -> Tag16 { + b64(text).try_into().expect("a sixteen-byte tag") +} + +fn b64_8(text: &Value) -> Tag8 { + b64(text).try_into().expect("an eight-byte tag") +} + +fn keys(value: &Value) -> Vec { + value + .as_object() + .expect("an object") + .keys() + .cloned() + .collect() +} + +fn header(bytes: &[u8]) -> ObservedHeader { + assert!(bytes.len() >= 40, "a header holds its fixed fields"); + let tag_len = usize::from(u16::from_le_bytes([bytes[38], bytes[39]])); + assert!( + bytes.len() >= 40 + tag_len, + "a header holds the tag it promises" + ); + ObservedHeader { + version: bytes[0], + flags: bytes[1], + epoch: bytes[2..6].try_into().unwrap(), + keyset: bytes[6..22].try_into().unwrap(), + iv: bytes[22..38].try_into().unwrap(), + tag: bytes[40..40 + tag_len].to_vec(), + trailing: bytes.len() - 40 - tag_len, + } +} + +fn entry(value: &Value) -> ObservedEntry { + ObservedEntry { + s: b64_16(&value["s"]), + c: base85_simd::decode(value["c"].as_str().expect("`c` is a string")) + .expect("`c` is base85"), + p: value.get("p").map(b64_8), + op: value.get("op").map(b64), + fields: keys(value), + } +} + +/// A projection group, `{"sv": [{"s", "vs"}, …], "el": [group, …]}`. +pub fn group(value: &Value) -> ObservedGroup { + ObservedGroup { + pairs: value["sv"] + .as_array() + .expect("`sv` is a list") + .iter() + .map(|pair| (b64_16(&pair["s"]), b64_16(&pair["vs"]))) + .collect(), + elements: value["el"] + .as_array() + .expect("`el` is a list") + .iter() + .map(group) + .collect(), + } +} + +fn row(value: &Value) -> ObservedRow { + ObservedRow { + fields: keys(value), + header: header(&b64(&value["h"])), + entries: value["sv"] + .as_array() + .expect("`sv` is a list") + .iter() + .map(entry) + .collect(), + q: group(&value["q"]), + } +} + +fn gate(text: &Value) -> ObservedGate { + let text = text.as_str().expect("a gate is a string"); + match text.strip_prefix('#') { + Some(slot) => { + let (n, selector) = slot.split_once(':').expect("`#n:`"); + ObservedGate::Slot( + n.parse().expect("a position"), + b64_16(&Value::String(selector.to_owned())), + ) + } + None => ObservedGate::Selector(b64_16(&Value::String(text.to_owned()))), + } +} + +fn query(kind: QueryKind, value: &Value) -> ObservedQuery { + match kind { + QueryKind::Extract => ObservedQuery::Extract { s: b64_16(value) }, + QueryKind::Subtree => ObservedQuery::Subtree { + t: b64_16(&value["t"]), + gates: value + .get("g") + .map(|g| { + g.as_array() + .expect("`g` is a list") + .iter() + .map(gate) + .collect() + }) + .unwrap_or_default(), + }, + QueryKind::Contains => ObservedQuery::Contains { + needle: group(value), + alt: value.get("alt").map(group), + }, + QueryKind::Equals => { + assert!( + value.get("alt").is_none(), + "an equality needle has no `alt`" + ); + ObservedQuery::Equals { + needle: group(value), + } + } + QueryKind::Order => ObservedQuery::Order { + s: b64_16(&value["s"]), + op: b64(&value["op"]), + }, + } +} diff --git a/packages/stack-encrypt/tests/json_leakage/profile.rs b/packages/stack-encrypt/tests/json_leakage/profile.rs new file mode 100644 index 000000000..84e4e7940 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/profile.rs @@ -0,0 +1,862 @@ +//! The leakage profile `L`: what `packages/eql/docs/reference/json-tree-leakage.md` +//! says the database learns, written as a function of the plaintext alone. +//! +//! [`l_store`] is what a set of stored rows reveals, [`l_query`] what one +//! query reveals on top of them. Neither takes a key or calls the crate: they +//! read the documents, the column's options and the query, and they name +//! every keyed value (a selector, a value selector, a token, an ordering +//! term) by an opaque integer that is equal wherever the real values are +//! equal, so the profile states equality patterns and never names. The +//! simulator ([`super::simulator`]) builds a wire-shaped world from this +//! output and its own randomness, and the distinguishers +//! ([`super::distinguishers`]) compare it with the real one. +//! +//! Each field of the leakage types is one clause, and its doc comment quotes +//! the sentence of the page it encodes. Where the page is vague the clause is +//! as precise as the code forces, and says so. One clause is marked +//! **beyond the page**: the code reveals it and the page does not say so. +//! [`Clauses::page_only`] turns it off, and the calibration shows a +//! distinguisher separating that profile from the real scheme. +//! +//! **This module is expected to change with the format.** The design change +//! of 2026-10-24 (member tokens without ordinals, a self tag `x` on every +//! entry, wildcards expanded on the client, no gates; +//! `docs/plans/2026-10-17-json-tree-index.md`, "Design change (2026-10-24)") +//! rewrites the subtree clauses of [`QueryLeakage`] and adds a clause for +//! `x`. When it lands, this file and the real-side adapter +//! ([`super::real`]) change together with the leakage page; the simulator +//! changes where the wire does; the distinguishers should not need to. + +use std::collections::HashMap; +use std::hash::Hash; + +use serde_json::{Number, Value}; +use stack_encrypt::json::{JsonPath, Segment}; + +/// What every entry's ciphertext adds to its plaintext: the frame's version +/// byte, the 96-bit random nonce and the 128-bit GCM tag +/// (`vitaminc-encrypt`'s `Aes256Cipher`). The page's "less the AEAD's fixed +/// overhead". +pub const AEAD_OVERHEAD: usize = 1 + 12 + 16; + +/// The length of the ZeroKMS key tag the header carries. The fake key source +/// the tests seal under mints 32-byte tags; it is a property of the key +/// service, not of any document, so it is a column constant here. +pub const KEY_TAG_LEN: usize = 32; + +/// A column's configuration, which the database knows: it is in every +/// header (the root option) or in the plan the queries run under. +#[derive(Clone, Debug)] +pub struct Column { + pub sealed_root: bool, + pub ordered: Vec, +} + +/// A query, in plaintext: what the client asks. +#[derive(Clone, Debug)] +pub enum Query { + /// `->` on a path. + Extract(String), + /// A subtree read of a path (`json_tree_subtree`, `json_tree_subtree_at`). + Subtree(String), + /// Containment, `@>`, of a needle document. + Contains(Value), + /// Equality at a path, `AtPath`, of a scalar. + Equals(String, Value), + /// An ordering bound at a path, `AtPath`, of a number or string. + Order(String, Value), +} + +/// Which sort "the sorted order of their key names" means. The page does +/// not say; the code sorts by the keys' UTF-8 bytes. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum KeyOrder { + /// Byte order of the UTF-8 key: serde_json's `Map` without + /// `preserve_order`, which is what the walk iterates. + Utf8Bytes, + /// PostgreSQL `jsonb`'s own order: shorter keys first, then bytes. The + /// other reading a reader of the page could reasonably take. + JsonbLengthFirst, +} + +/// The clauses of the profile that are on or off as a whole, and the one +/// reading the page leaves open. The default is what the code needs; +/// [`Clauses::page_only`] is the page as written. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Clauses { + /// How "an object's members in the sorted order of their key names" + /// sorts. + pub key_order: KeyOrder, + /// **Beyond the page.** An ordering term's length: `8 × n + 1` bytes for + /// a value encoded in `n` bytes (`clww-ore`'s `OpeClww8VariableV1`). A + /// number encodes in eight bytes, so its term is 65 bytes; a string in + /// its UTF-8 bytes and one more, so its term is `8 × (len + 1) + 1` + /// bytes and gives its length. The page says only that "a term's + /// presence at an ordered path also says the leaf is a number or a + /// string"; the length also says which, unless the string is seven bytes + /// long, and repeats the string's length that `c` already shows. Off, + /// every term is stated as one length. + pub ordering_term_length: bool, +} + +impl Default for Clauses { + fn default() -> Self { + Self { + key_order: KeyOrder::Utf8Bytes, + ordering_term_length: true, + } + } +} + +impl Clauses { + /// The page as written: every clause beyond it off. + pub fn page_only() -> Self { + Self { + key_order: KeyOrder::Utf8Bytes, + ordering_term_length: false, + } + } +} + +/// An opaque name for a path selector `s`: equal for every node anywhere in +/// the column whose wildcard-folded path is the same. +pub type SelId = usize; +/// An opaque name for a value selector `vs`. +pub type ValueId = usize; +/// An opaque name for a chain head's token. +pub type HeadId = usize; +/// An opaque name for an ordering term's value: equal for equal values. +pub type OrderClass = usize; + +/// How a node hangs off its parent, which decides its ordinal under the +/// parent's token. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Step { + Root, + /// An object member: its ordinal is its keyed rank among its siblings. + Member, + /// An array element at this position, which is its ordinal. + Element(u64), +} + +/// The ordering term of one leaf, or of one query bound. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct OrderingLeak { + /// "The order of that leaf's value among every row that holds one, and + /// which rows hold equal values." Equal values share a class; the + /// classes' order is [`Leakage::ordering_ranks`]. + pub class: OrderClass, + /// **Beyond the page** ([`Clauses::ordering_term_length`]): the term's + /// length in bytes. + pub len: usize, +} + +/// One entry of `sv`, as the page says a stored row shows it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EntryLeakage { + /// "`s`, the path selector on every node: which paths the row has, and + /// how many nodes sit at each. A selector is a keyed hash of the path, + /// the same in every row of the column […] Array elements share their + /// array's selector, so `$.tags[0]` and `$.tags[1]` look alike, but each + /// element is its own entry." + pub selector: SelId, + /// The shape: the position in [`RowLeakage::entries`] of this node's + /// parent. "Which selector's path sits under which is not written down, + /// but it is the same in every row, so an observer comparing rows learns + /// it, and with it each row's parent and child relations and the node + /// count of every subtree." The simulator uses it to place the chain + /// below a token a query sends; at rest the order clause + /// ([`RowLeakage::entries`]) is what shows it. + pub parent: Option, + /// How the node hangs off its parent, which a token's holder sees in the + /// ordinals it probes: an element's ordinal is its position. The page + /// says a token's holder can "see how the nodes nest, link each one to + /// its parent"; it does not say that it also sees each element's + /// position and each member's keyed rank. Vague; stated here as the code + /// forces it. + pub step: Step, + /// The chain head this entry is, if it is one: under the sealed root a + /// top-level field ("the top-level field the path starts with"), with the + /// root open the root. A head's token is a keyed function of its path, + /// "column-wide", so every row's entry at that path is one head. + pub head: Option, + /// "Which entries carry no `p` does show: with the sealed root on, those + /// are the root's direct members […] With the sealed root off, only the + /// root lacks one." (The root lacks one under the sealed root too.) + pub has_parent_tag: bool, + /// "Each ciphertext's length […] for each node the length of its key name + /// (for an object member) plus, for a scalar, the length of its value, + /// exactly, less the AEAD's fixed overhead. […] With the sealed root on, + /// the root entry's length is the document's length as JSON." The fixed + /// parts the page leaves out are written in [`ciphertext_len`]. + pub ciphertext_len: usize, + /// "`op`, the ordering term, only on number and string leaves at the + /// paths the index orders." + pub ordering: Option, +} + +/// One group of a projection, stored (`q`) or sent (a needle). +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct GroupLeakage { + /// "One `{s, vs}` pair per node, […] in the same order": the pairs of the + /// nodes that sit directly in this group, in the walk's order. + /// "Whether two rows hold the same value at the same path, through the + /// value selector `vs`, a keyed hash of the path and the value (for an + /// object or an array, of its kind alone)." + pub pairs: Vec<(SelId, ValueId)>, + /// "Nested by array element […] Which nodes sit inside the same array + /// element, through the nesting, for every element that is an object or + /// an array." + pub elements: Vec, +} + +/// One stored row. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RowLeakage { + /// The header's sealed-root flag: "whether the root is sealed". + pub sealed_root: bool, + /// The header's keyset: "the keyset the data key was minted under", + /// which "is the same for every column it seals". One name for every row. + pub keyset: usize, + /// The header's key identifiers (the `iv` and the ZeroKMS tag): "the + /// data key's identifier (unique to the row)". A different name in every + /// row. + pub data_key: usize, + /// "The number of entries", and the entries in their stored order: + /// "Entries are stored in depth-first order, the root first, each node + /// followed by everything beneath it, and an object's members in the + /// sorted order of their key names." The page does not say which sort; + /// the code's is the keys' UTF-8 bytes (serde_json's `Map` without + /// `preserve_order`), so that is the order here. + pub entries: Vec, + /// The containment projection `q`. + pub projection: GroupLeakage, +} + +/// A gate of a subtree read. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GateLeak { + /// "A gate is the step's selector". + Selector(SelId), + /// "or, for an array position, `#n:`, with the position `n` in + /// the clear." + Slot(u64, SelId), +} + +/// The token a subtree read sends, by name: a head, then the array positions +/// the client chains below it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TokenName { + pub head: HeadId, + /// Each position step after the head: the element selector and the + /// position. + pub positions: Vec<(SelId, u64)>, +} + +/// What one query reveals, beyond the rows it returns. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum QueryLeakage { + /// "Path extraction […] The selector of the path asked. A position in the + /// path reads as the wildcard, so a position is not sent." + Extract { selector: SelId }, + /// "A subtree read […] A token, and one gate per step below it. The token + /// is not the token of the node you asked for: it is the token of the + /// deepest node on the path the client can derive, which is the + /// top-level field the path starts with or, while the path goes on + /// through array positions, the element they reach." + Subtree { + token: TokenName, + gates: Vec, + /// Per row, the entries the token lets the database link: "A token + /// is a capability over the structure of its node's subtree in every + /// row of the column: whoever holds it can find each node under that + /// field in any row, see how the nodes nest, link each one to its + /// parent, and derive the tokens of the nodes beneath." So: every + /// strict descendant of the token's node, in every row. + reach: Vec>, + /// Per row, the nodes the path names (what the walk returns at depth + /// `0`). Not a clause of its own: the database computes it from the + /// token, the gates and the rows. Carried so a mutant profile can + /// narrow `reach` to what a read would reveal if the token covered + /// only the path asked (`token_reaches_named_nodes_only`). + named: Vec>, + }, + /// "Containment […] The needle's own projection: the selectors of its + /// paths, the value selectors of its nodes, and its element grouping." + /// The page does not mention `alt`, the extra one-pair projection a + /// needle whose root is a scalar carries so that it matches a stored + /// root array holding it; it is part of "the needle's own projection" + /// here. + Contains { + needle: GroupLeakage, + alt: Option, + }, + /// "Equality at a path, which is sent as a containment needle": the + /// projection of the document that holds the value at the path. + Equals { needle: GroupLeakage }, + /// "Ordering at a path […] The path's selector and the bound's own + /// ordering term, which compares with the stored terms exactly as they + /// compare with each other." + Order { + selector: SelId, + bound: OrderingLeak, + }, +} + +/// The leakage of a whole column: its rows and its queries, over one shared +/// set of opaque names. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Leakage { + /// "The keyset the data key was minted under […] the same for every + /// column it seals": one keyset per column, and the ZeroKMS tag's length. + pub key_tag_len: usize, + pub rows: Vec, + pub queries: Vec, + /// The rank of each [`OrderClass`] among all of them: "the order of that + /// leaf's value among every row that holds one", and the bound's "place + /// in the order of the stored values at every ordered path" ("leaves at + /// two ordered paths compare with each other too"). + pub ordering_ranks: Vec, +} + +// ============================================================================= +// The opaque names +// ============================================================================= + +struct Interner { + ids: HashMap, +} + +impl Default for Interner { + fn default() -> Self { + Self { + ids: HashMap::new(), + } + } +} + +impl Interner { + fn id(&mut self, value: T) -> usize { + let next = self.ids.len(); + *self.ids.entry(value).or_insert(next) + } +} + +/// A value an ordering term orders by. Numbers order as `f64` (the term is +/// derived from `as_f64`); strings by their bytes. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +enum OrderKey { + Number(u64), + String(Vec), +} + +/// The names shared by [`l_store`] and [`l_query`]: one id space per kind of +/// keyed value, so a query's selector is the stored selector's id when the +/// real bytes are equal. +#[derive(Default)] +pub struct Ids { + selectors: Interner>, + values: Interner<(SelId, u8, Vec)>, + heads: Interner>, + ordering: Interner, + ordering_keys: Vec, + pub clauses: Clauses, +} + +impl Ids { + pub fn new(clauses: Clauses) -> Self { + Self { + clauses, + ..Self::default() + } + } + + /// The selector of a path, every array step read as the wildcard. + fn selector(&mut self, path: &[Segment]) -> SelId { + self.selectors.id(path + .iter() + .map(|s| match s { + Segment::Key(k) => Segment::Key(k.clone()), + Segment::Index(_) | Segment::Wildcard => Segment::Wildcard, + }) + .collect()) + } + + fn value(&mut self, selector: SelId, value: &Value) -> ValueId { + self.values + .id((selector, kind_byte(value), canonical(value))) + } + + fn head(&mut self, positional: &[Segment]) -> HeadId { + self.heads.id(positional.to_vec()) + } + + fn ordering(&mut self, value: &Value) -> OrderingLeak { + let (key, len) = match value { + Value::Number(n) => { + let f = n.as_f64().expect("a JSON number has an f64"); + assert!( + f != 0.0 || f.is_sign_positive(), + "the corpus has no -0.0, whose term the page does not describe" + ); + (OrderKey::Number(f.to_bits()), 8 * 8 + 1) + } + Value::String(s) => ( + OrderKey::String(s.as_bytes().to_vec()), + 8 * (s.len() + 1) + 1, + ), + _ => unreachable!("only numbers and strings carry a term"), + }; + let class = self.ordering.id(key.clone()); + if class == self.ordering_keys.len() { + self.ordering_keys.push(key); + } + OrderingLeak { + class, + len: if self.clauses.ordering_term_length { + len + } else { + 65 + }, + } + } + + /// The rank of every ordering class. The page does not say how a + /// number's term compares with a string's, so a column that orders both + /// is refused rather than guessed at. + pub fn ordering_ranks(&self) -> Vec { + let numbers = self + .ordering_keys + .iter() + .any(|k| matches!(k, OrderKey::Number(_))); + let strings = self + .ordering_keys + .iter() + .any(|k| matches!(k, OrderKey::String(_))); + assert!( + !(numbers && strings), + "the page does not say how a number's ordering term compares with a string's; \ + a world orders one kind of leaf" + ); + let mut order: Vec = (0..self.ordering_keys.len()).collect(); + order.sort_by( + |&a, &b| match (&self.ordering_keys[a], &self.ordering_keys[b]) { + (OrderKey::Number(x), OrderKey::Number(y)) => { + f64::from_bits(*x).total_cmp(&f64::from_bits(*y)) + } + (OrderKey::String(x), OrderKey::String(y)) => x.cmp(y), + _ => unreachable!("refused above"), + }, + ); + let mut ranks = vec![0; order.len()]; + for (rank, class) in order.into_iter().enumerate() { + ranks[class] = rank; + } + ranks + } +} + +// ============================================================================= +// Plaintext helpers: what the scheme computes over, with no key +// ============================================================================= + +fn kind_byte(value: &Value) -> u8 { + match value { + Value::Null => 0, + Value::Bool(_) => 1, + Value::Number(_) => 2, + Value::String(_) => 3, + Value::Object(_) => 4, + Value::Array(_) => 5, + } +} + +/// A number's identity for its value selector: an integral value is one +/// value whichever JSON spelling it had (`2` and `2.0`), as `jsonb` compares +/// them. +fn canonical_number(number: &Number) -> Vec { + if let Some(i) = number.as_i64() { + return i128::from(i).to_le_bytes().to_vec(); + } + if let Some(u) = number.as_u64() { + return i128::from(u).to_le_bytes().to_vec(); + } + let f = number.as_f64().unwrap_or(0.0); + if f.fract() == 0.0 && f.abs() < 1e38 { + return (f as i128).to_le_bytes().to_vec(); + } + let mut out = vec![b'f']; + out.extend_from_slice(&f.to_le_bytes()); + out +} + +fn canonical(value: &Value) -> Vec { + match value { + Value::Null | Value::Object(_) | Value::Array(_) => Vec::new(), + Value::Bool(b) => vec![u8::from(*b)], + Value::Number(n) => canonical_number(n), + Value::String(s) => s.as_bytes().to_vec(), + } +} + +/// One node of a document in its stored order. +pub struct Node<'a> { + pub value: &'a Value, + pub positional: Vec, + pub parent: Option, + pub step: Step, + pub label: Option<&'a str>, + /// The nearest enclosing array element that is a container. + pub element: Option, + pub container_element: bool, +} + +/// A document's nodes in the order the page says they are stored: +/// depth-first, the root first, an object's members in `order`, an array's +/// elements in position order. +pub fn flatten(doc: &Value, order: KeyOrder) -> Vec> { + let mut flattener = Flattener { + order, + out: Vec::new(), + }; + flattener.visit(doc, Vec::new(), None, Step::Root, None, None); + flattener.out +} + +struct Flattener<'a> { + order: KeyOrder, + out: Vec>, +} + +impl<'a> Flattener<'a> { + fn visit( + &mut self, + value: &'a Value, + positional: Vec, + parent: Option, + step: Step, + label: Option<&'a str>, + element: Option, + ) { + let container_element = + matches!(step, Step::Element(_)) && matches!(value, Value::Object(_) | Value::Array(_)); + let index = self.out.len(); + self.out.push(Node { + value, + positional: positional.clone(), + parent, + step, + label, + element, + container_element, + }); + let child_element = if container_element { + Some(index) + } else { + element + }; + match value { + Value::Object(map) => { + let mut keys: Vec<&String> = map.keys().collect(); + match self.order { + KeyOrder::Utf8Bytes => keys.sort_by(|a, b| a.as_bytes().cmp(b.as_bytes())), + KeyOrder::JsonbLengthFirst => { + keys.sort_by(|a, b| (a.len(), a.as_bytes()).cmp(&(b.len(), b.as_bytes()))) + } + } + for key in keys { + let mut path = positional.clone(); + path.push(Segment::Key(key.clone())); + self.visit( + &map[key], + path, + Some(index), + Step::Member, + Some(key), + child_element, + ); + } + } + Value::Array(items) => { + for (i, item) in items.iter().enumerate() { + let mut path = positional.clone(); + path.push(Segment::Index(i as u64)); + self.visit( + item, + path, + Some(index), + Step::Element(i as u64), + None, + child_element, + ); + } + } + _ => {} + } + } +} + +/// Whether `pattern` covers `path`: `JsonPath::covers`, over segments. +fn covers(pattern: &[Segment], path: &[Segment]) -> bool { + pattern.len() == path.len() + && pattern.iter().zip(path).all(|(p, s)| match (p, s) { + (Segment::Key(a), Segment::Key(b)) => a == b, + (Segment::Wildcard, Segment::Index(_) | Segment::Wildcard) => true, + (Segment::Index(a), Segment::Index(b)) => a == b, + _ => false, + }) +} + +fn segments(path: &str) -> Vec { + JsonPath::parse(path) + .expect("the corpus names valid paths") + .segments() + .to_vec() +} + +/// A node's ciphertext length: [`AEAD_OVERHEAD`] plus its plaintext, which +/// is the kind byte, the four-byte child count, the label (a tag byte, then +/// a four-byte length and the key for a member, or the eight-byte position +/// for an element) and, for a scalar, its value (a string's bytes, a +/// number's JSON text, one byte for a boolean, nothing for `null`). A sealed +/// root's plaintext is a marker byte and the document as compact JSON. +/// +/// Where this is more precise than the page: "a number's length in digits" +/// is the length of serde_json's text for it, sign, point and exponent +/// included (`-2` is two, `1e300` five); a boolean's and a null's "length of +/// its value" are one and zero; an element's label is a fixed nine bytes, so +/// it adds nothing that varies. +pub fn ciphertext_len(node: &Node<'_>, sealed_root: bool, doc: &Value) -> usize { + if node.parent.is_none() && sealed_root { + return AEAD_OVERHEAD + 1 + serde_json::to_string(doc).expect("serializes").len(); + } + let label = match node.step { + Step::Root => 1, + Step::Member => 1 + 4 + node.label.map_or(0, str::len), + Step::Element(_) => 1 + 8, + }; + let scalar = match node.value { + Value::String(s) => s.len(), + Value::Bool(_) => 1, + Value::Number(n) => n.to_string().len(), + Value::Null | Value::Object(_) | Value::Array(_) => 0, + }; + AEAD_OVERHEAD + 1 + 4 + label + scalar +} + +fn group( + ids: &mut Ids, + nodes: &[Node<'_>], + selectors: &[SelId], + parent: Option, +) -> GroupLeakage { + let belongs = |i: usize, n: &Node<'_>| { + if n.container_element { + Some(i) + } else { + n.element + } + }; + let pairs = nodes + .iter() + .enumerate() + .filter(|(i, n)| belongs(*i, n) == parent) + .map(|(i, n)| (selectors[i], ids.value(selectors[i], n.value))) + .collect(); + let elements = nodes + .iter() + .enumerate() + .filter(|(_, n)| n.container_element && n.element == parent) + .map(|(i, _)| group(ids, nodes, selectors, Some(i))) + .collect(); + GroupLeakage { pairs, elements } +} + +fn projection(ids: &mut Ids, doc: &Value) -> GroupLeakage { + let nodes = flatten(doc, ids.clauses.key_order); + let selectors: Vec = nodes.iter().map(|n| ids.selector(&n.positional)).collect(); + group(ids, &nodes, &selectors, None) +} + +// ============================================================================= +// L +// ============================================================================= + +/// What the stored rows reveal at rest, row by row. +pub fn l_store(column: &Column, ids: &mut Ids, docs: &[Value]) -> Vec { + let ordered: Vec> = column + .ordered + .iter() + .map(|p| p.segments().to_vec()) + .collect(); + docs.iter() + .enumerate() + .map(|(r, doc)| { + let nodes = flatten(doc, ids.clauses.key_order); + let entries = nodes + .iter() + .map(|node| { + let depth = node.positional.len(); + let is_head = if column.sealed_root { + depth == 1 + } else { + depth == 0 + }; + let ordering = (ordered.iter().any(|o| covers(o, &node.positional)) + && matches!(node.value, Value::Number(_) | Value::String(_))) + .then(|| ids.ordering(node.value)); + EntryLeakage { + selector: ids.selector(&node.positional), + parent: node.parent, + step: node.step, + head: is_head.then(|| ids.head(&node.positional)), + has_parent_tag: depth > 0 && !(column.sealed_root && depth == 1), + ciphertext_len: ciphertext_len(node, column.sealed_root, doc), + ordering, + } + }) + .collect(); + RowLeakage { + sealed_root: column.sealed_root, + keyset: 0, + data_key: r, + entries, + projection: projection(ids, doc), + } + }) + .collect() +} + +/// The nodes of `nodes` that `pattern` reaches, as `JsonPath::select` finds +/// them: a key step into an array, or a position into an object, reaches +/// nothing. +fn reached(nodes: &[Node<'_>], pattern: &[Segment]) -> Vec { + (0..nodes.len()) + .filter(|&i| covers(pattern, &nodes[i].positional)) + .collect() +} + +/// Every strict descendant of node `at`: in the stored order these are the +/// positions after it up to the end of its subtree. +fn descendants(nodes: &[Node<'_>], at: usize) -> Vec { + let mut inside = vec![false; nodes.len()]; + inside[at] = true; + let mut out = Vec::new(); + for (i, node) in nodes.iter().enumerate().skip(at + 1) { + if node.parent.is_some_and(|p| inside[p]) { + inside[i] = true; + out.push(i); + } + } + out +} + +/// What one query reveals, over the stored `docs`. +pub fn l_query(column: &Column, ids: &mut Ids, docs: &[Value], query: &Query) -> QueryLeakage { + match query { + Query::Extract(path) => QueryLeakage::Extract { + selector: ids.selector(&segments(path)), + }, + Query::Subtree(path) => { + let segs = segments(path); + // The head: under the sealed root the top-level field (or + // element) the path starts with, with the root open the root. + let split = usize::from(column.sealed_root); + assert!( + segs.len() >= split && !segs[..split].contains(&Segment::Wildcard), + "under the sealed root a subtree read names a top-level node: {path}" + ); + let (head_path, rest) = segs.split_at(split); + let head = ids.head(head_path); + let mut prefix = head_path.to_vec(); + let mut positions = Vec::new(); + let mut gates = Vec::new(); + let mut chaining = true; + for segment in rest { + prefix.push(segment.clone()); + match (segment, chaining) { + (Segment::Index(i), true) => positions.push((ids.selector(&prefix), *i)), + (Segment::Index(i), false) => { + gates.push(GateLeak::Slot(*i, ids.selector(&prefix))) + } + (Segment::Key(_) | Segment::Wildcard, _) => { + chaining = false; + gates.push(GateLeak::Selector(ids.selector(&prefix))); + } + } + } + let token_path = &segs[..split + positions.len()]; + let order = ids.clauses.key_order; + let (reach, named) = docs + .iter() + .map(|doc| { + let nodes = flatten(doc, order); + let at = nodes.iter().position(|n| n.positional == token_path); + let reach = at.map(|at| descendants(&nodes, at)).unwrap_or_default(); + let named = if gates.is_empty() { + at.into_iter().collect() + } else { + reached(&nodes, &segs) + }; + (reach, named) + }) + .unzip(); + QueryLeakage::Subtree { + token: TokenName { head, positions }, + gates, + reach, + named, + } + } + Query::Contains(needle) => { + let alt = (!matches!(needle, Value::Object(_) | Value::Array(_))).then(|| { + let s = ids.selector(&[Segment::Wildcard]); + GroupLeakage { + pairs: vec![(s, ids.value(s, needle))], + elements: Vec::new(), + } + }); + QueryLeakage::Contains { + needle: projection(ids, needle), + alt, + } + } + Query::Equals(path, value) => { + let needle = segments(path) + .iter() + .rev() + .fold(value.clone(), |inner, segment| match segment { + Segment::Key(key) => { + let mut map = serde_json::Map::new(); + let _ = map.insert(key.clone(), inner); + Value::Object(map) + } + Segment::Index(_) | Segment::Wildcard => Value::Array(vec![inner]), + }); + QueryLeakage::Equals { + needle: projection(ids, &needle), + } + } + Query::Order(path, value) => QueryLeakage::Order { + selector: ids.selector(&segments(path)), + bound: ids.ordering(value), + }, + } +} + +/// `L` over a whole column: [`l_store`] for the rows, then [`l_query`] for +/// each query in turn, over one set of names. +pub fn leak(column: &Column, clauses: Clauses, docs: &[Value], queries: &[Query]) -> Leakage { + let mut ids = Ids::new(clauses); + let rows = l_store(column, &mut ids, docs); + let queries = queries + .iter() + .map(|q| l_query(column, &mut ids, docs, q)) + .collect(); + Leakage { + key_tag_len: KEY_TAG_LEN, + rows, + queries, + ordering_ranks: ids.ordering_ranks(), + } +} diff --git a/packages/stack-encrypt/tests/json_leakage/real.rs b/packages/stack-encrypt/tests/json_leakage/real.rs new file mode 100644 index 000000000..0f4816419 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/real.rs @@ -0,0 +1,104 @@ +//! The real scheme, reduced to its wire form: seal each document and derive +//! each query term through the crate's public API, under the fake key +//! source the other JSON tests use, and serialize everything with +//! `serde_json` as the EQL domain stores and sends it. +//! +//! Deliberately thin: it says which API each [`Query`] goes through and +//! nothing about what the result reveals. When the format changes, this file +//! changes only where the API does. + +use serde_json::Value; +use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath}; +use stack_encrypt::op; +use stack_encrypt::target::Json; +use stack_encrypt::{nonempty, Plan, StackCipher}; +use stack_kms::FakeDataKeySource; + +use super::observe::QueryKind; +use super::profile::{Column, Query}; + +type JsonPlan = + stack_encrypt::plan::ValuePlan>; + +/// The crate's options for a column. +pub fn options(column: &Column) -> JsonOptions { + JsonOptions { + sealed_root: column.sealed_root, + ordered: column.ordered.iter().cloned().collect(), + } +} + +fn plan(column: &Column) -> JsonPlan { + Plan::value::() + .context("users/profile") + .encrypt_with_layout(Json::new(options(column))) + .build() + .expect("the plan builds") +} + +/// The rows and query terms the database would hold and receive. +pub async fn world( + cipher: &StackCipher, + column: &Column, + docs: &[Value], + queries: &[Query], +) -> (Vec, Vec<(QueryKind, Value)>) { + let plan = plan(column); + let mut rows = Vec::with_capacity(docs.len()); + for doc in docs { + let sealed = cipher.encrypt(doc).using(&plan).await.expect("seals"); + rows.push(serde_json::to_value(&sealed).expect("serializes")); + } + let keyset = cipher.default_keyset(); + let mut sent = Vec::with_capacity(queries.len()); + for query in queries { + sent.push(match query { + Query::Extract(path) => { + let selector = keyset + .json_selector( + &JsonPath::parse(path).expect("a path"), + nonempty!("users").with("profile"), + ) + .await + .expect("derives"); + (QueryKind::Extract, serde_json::to_value(selector).unwrap()) + } + Query::Subtree(path) => { + let key = keyset + .json_subtree_key( + &JsonPath::parse(path).expect("a path"), + &options(column), + nonempty!("users").with("profile"), + ) + .await + .expect("derives"); + (QueryKind::Subtree, serde_json::to_value(&key).unwrap()) + } + Query::Contains(needle) => { + let term = cipher + .query(op::Contains, needle) + .on(&plan) + .await + .expect("derives"); + (QueryKind::Contains, serde_json::to_value(&term).unwrap()) + } + Query::Equals(path, value) => { + let term = cipher + .query(op::AtPath(op::Eq), &(path.clone(), value.clone())) + .on(&plan) + .await + .expect("derives"); + (QueryKind::Equals, serde_json::to_value(&term).unwrap()) + } + Query::Order(path, value) => { + let term = cipher + .query(op::AtPath(op::Gte), &(path.clone(), value.clone())) + .on(&plan) + .await + .expect("derives"); + (QueryKind::Order, serde_json::to_value(&term).unwrap()) + } + }); + } + (rows, sent) +} diff --git a/packages/stack-encrypt/tests/json_leakage/simulator.rs b/packages/stack-encrypt/tests/json_leakage/simulator.rs new file mode 100644 index 000000000..79d37e439 --- /dev/null +++ b/packages/stack-encrypt/tests/json_leakage/simulator.rs @@ -0,0 +1,302 @@ +//! The simulator: a wire-shaped world built from a [`Leakage`] and +//! randomness, with no plaintext and no key. +//! +//! Every keyed value is a fresh random string per opaque name (a selector +//! per [`SelId`], a value selector per [`ValueId`], a head token per +//! [`HeadId`]), so equality patterns are exactly the profile's. Ciphertexts +//! are random bytes of the stated length behind the frame's public version +//! byte. Ordering terms are random bytes laid out so that byte order is the +//! profile's rank order. +//! +//! The chain is the one place the simulator evaluates the real scheme's +//! public functions ([`super::chain`]): a token the database is sent has to +//! derive the parent tags of exactly the entries the profile says it reaches, +//! with the real hash, or the database would notice. So the simulator gives +//! every node a token (heads random, every other node chained below its +//! parent with an ordinal: an element's position, or a member's rank under a +//! keyed function the simulator samples itself), and writes the chained +//! parent tag on every entry some query's [`QueryLeakage::Subtree::reach`] +//! names. Every other parent tag is random bytes, as a tag under a token the +//! database never sees is to it. + +use std::collections::{HashMap, HashSet}; + +use base64ct::{Base64, Encoding}; +use proptest::prelude::RngCore; +use proptest::test_runner::TestRng; +use serde_json::{json, Value}; + +use super::chain::{child_token, parent_tag, simulated_rank, Tag16}; +use super::observe::QueryKind; +use super::profile::{GateLeak, GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step}; + +/// The version byte `vitaminc-encrypt` writes at the front of every +/// ciphertext: public framing, not leakage. +pub const CIPHERTEXT_FRAME: u8 = 2; + +/// A fixed-size random value per name, drawn on first use. +struct Names { + bytes: HashMap, +} + +impl Names { + fn new() -> Self { + Self { + bytes: HashMap::new(), + } + } + + fn get(&mut self, id: usize, rng: &mut TestRng) -> [u8; N] { + *self.bytes.entry(id).or_insert_with(|| random(rng)) + } +} + +fn random(rng: &mut TestRng) -> [u8; N] { + let mut out = [0; N]; + rng.fill_bytes(&mut out); + out +} + +fn random_vec(rng: &mut TestRng, len: usize) -> Vec { + let mut out = vec![0; len]; + rng.fill_bytes(&mut out); + out +} + +fn b64(bytes: &[u8]) -> String { + Base64::encode_string(bytes) +} + +struct Simulator<'a> { + leakage: &'a Leakage, + rng: &'a mut TestRng, + selectors: Names<16>, + values: Names<16>, + heads: Names<16>, + keysets: Names<16>, + ivs: Names<16>, + tags: HashMap>, + ordering: HashMap>, + rank_key: [u8; 32], +} + +/// Build the rows' wire JSON and the queries' wire JSON from `leakage` and +/// `rng` alone. +pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(QueryKind, Value)>) { + let rank_key = random(rng); + let mut sim = Simulator { + leakage, + rng, + selectors: Names::new(), + values: Names::new(), + heads: Names::new(), + keysets: Names::new(), + ivs: Names::new(), + tags: HashMap::new(), + ordering: HashMap::new(), + rank_key, + }; + let rows = leakage + .rows + .iter() + .enumerate() + .map(|(r, row)| sim.row(r, row)) + .collect(); + let queries = leakage.queries.iter().map(|q| sim.query(q)).collect(); + (rows, queries) +} + +impl Simulator<'_> { + fn selector(&mut self, id: usize) -> Tag16 { + self.selectors.get(id, self.rng) + } + + /// An ordering term of `len` bytes for class `class`: a zero byte, the + /// class's rank big-endian, then random bytes, so byte order is rank + /// order and equal classes are equal terms. + fn ordering(&mut self, class: usize, len: usize) -> Vec { + if let Some(bytes) = self.ordering.get(&class) { + assert_eq!(bytes.len(), len, "one class, one length"); + return bytes.clone(); + } + assert!(len >= 5, "a term is long enough to carry its rank"); + let rank = u32::try_from(self.leakage.ordering_ranks[class]).expect("a small world"); + let mut bytes = vec![0]; + bytes.extend_from_slice(&rank.to_be_bytes()); + bytes.extend(random_vec(self.rng, len - 5)); + let _ = self.ordering.insert(class, bytes.clone()); + bytes + } + + fn group(&mut self, group: &GroupLeakage) -> Value { + let pairs: Vec = group + .pairs + .iter() + .map(|&(s, vs)| { + let s = self.selector(s); + let vs = self.values.get(vs, self.rng); + json!({"s": b64(&s), "vs": b64(&vs)}) + }) + .collect(); + let elements: Vec = group.elements.iter().map(|g| self.group(g)).collect(); + json!({"sv": pairs, "el": elements}) + } + + /// Every entry's token and ordinal under its parent, where it has them. + fn tokens(&mut self, row: &RowLeakage) -> (Vec>, Vec>) { + let n = row.entries.len(); + let mut tokens: Vec> = vec![None; n]; + let mut ordinals: Vec> = vec![None; n]; + // A parent precedes its children only under the order clause; a + // mutant that shuffles the stored order still names parents, so the + // chain is resolved by depth rather than by position. + let depth: Vec = row + .entries + .iter() + .map(|entry| { + let mut d = 0; + let mut at = entry.parent; + while let Some(p) = at { + d += 1; + at = row.entries[p].parent; + } + d + }) + .collect(); + let mut order: Vec = (0..n).collect(); + order.sort_by_key(|&i| (depth[i], i)); + for &i in &order { + let entry = &row.entries[i]; + if let Some(head) = entry.head { + tokens[i] = Some(self.heads.get(head, self.rng)); + continue; + } + let Some(parent) = entry.parent else { continue }; + let Some(parent_token) = tokens[parent] else { + continue; + }; + let ordinal = match entry.step { + Step::Element(i) => u32::try_from(i).expect("a small array"), + Step::Member => { + let mine = self.selector(entry.selector); + let my_rank = simulated_rank(&self.rank_key, &parent_token, &mine); + let siblings: Vec = (0..n) + .filter(|&j| { + row.entries[j].parent == Some(parent) + && row.entries[j].step == Step::Member + }) + .collect(); + let below = siblings + .into_iter() + .filter(|&j| { + let theirs = self.selector(row.entries[j].selector); + simulated_rank(&self.rank_key, &parent_token, &theirs) < my_rank + }) + .count(); + u32::try_from(below).expect("a small object") + } + Step::Root => continue, + }; + ordinals[i] = Some(ordinal); + let s = self.selector(entry.selector); + tokens[i] = Some(child_token(&parent_token, &s, ordinal)); + } + (tokens, ordinals) + } + + fn row(&mut self, r: usize, row: &RowLeakage) -> Value { + let iv: [u8; 16] = self.ivs.get(row.data_key, self.rng); + let tag = self + .tags + .entry(row.data_key) + .or_insert_with(|| random_vec(self.rng, self.leakage.key_tag_len)) + .clone(); + let keyset: [u8; 16] = self.keysets.get(row.keyset, self.rng); + let mut header = vec![1, u8::from(row.sealed_root), 0, 0, 0, 0]; + header.extend_from_slice(&keyset); + header.extend_from_slice(&iv); + header.extend_from_slice(&u16::try_from(tag.len()).expect("a short tag").to_le_bytes()); + header.extend_from_slice(&tag); + + let reached: HashSet = self + .leakage + .queries + .iter() + .filter_map(|q| match q { + QueryLeakage::Subtree { reach, .. } => Some(reach[r].iter().copied()), + _ => None, + }) + .flatten() + .collect(); + let (tokens, ordinals) = self.tokens(row); + let entries: Vec = row + .entries + .iter() + .enumerate() + .map(|(i, entry)| { + let s = self.selector(entry.selector); + let mut c = vec![CIPHERTEXT_FRAME]; + c.extend(random_vec(self.rng, entry.ciphertext_len - 1)); + let mut wire = json!({"s": b64(&s), "c": base85_simd::encode(&c)}); + if entry.has_parent_tag { + let chained = entry + .parent + .and_then(|p| tokens[p]) + .zip(ordinals[i]) + .filter(|_| reached.contains(&i)); + let p = match chained { + Some((parent, o)) => parent_tag(&iv, &parent, o), + None => random(self.rng), + }; + wire["p"] = json!(b64(&p)); + } + if let Some(op) = entry.ordering { + wire["op"] = json!(b64(&self.ordering(op.class, op.len))); + } + wire + }) + .collect(); + let q = self.group(&row.projection); + json!({"h": b64(&header), "sv": entries, "q": q}) + } + + fn query(&mut self, query: &QueryLeakage) -> (QueryKind, Value) { + match query { + QueryLeakage::Extract { selector } => { + (QueryKind::Extract, json!(b64(&self.selector(*selector)))) + } + QueryLeakage::Subtree { token, gates, .. } => { + let mut t = self.heads.get(token.head, self.rng); + for &(s, i) in &token.positions { + let s = self.selector(s); + t = child_token(&t, &s, u32::try_from(i).expect("a small position")); + } + let gates: Vec = gates + .iter() + .map(|g| match *g { + GateLeak::Selector(s) => b64(&self.selector(s)), + GateLeak::Slot(n, s) => format!("#{n}:{}", b64(&self.selector(s))), + }) + .collect(); + let mut wire = json!({"t": b64(&t)}); + if !gates.is_empty() { + wire["g"] = json!(gates); + } + (QueryKind::Subtree, wire) + } + QueryLeakage::Contains { needle, alt } => { + let mut wire = self.group(needle); + if let Some(alt) = alt { + wire["alt"] = self.group(alt); + } + (QueryKind::Contains, wire) + } + QueryLeakage::Equals { needle } => (QueryKind::Equals, self.group(needle)), + QueryLeakage::Order { selector, bound } => { + let s = self.selector(*selector); + let op = self.ordering(bound.class, bound.len); + (QueryKind::Order, json!({"s": b64(&s), "op": b64(&op)})) + } + } + } +} From 839c0f6fd00c10c453d92f39282bc03c76664b92 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 18:56:50 +1100 Subject: [PATCH 02/19] test(stack-encrypt): port the json_tree leakage profile to the #1363 format The profile was written against entries {s, c, p?, op?}, member tokens carrying their ordinal, and a subtree read that sent one head token plus plaintext gates. #1363 adds a self tag x to every entry, drops the ordinal from member tokens, and has the client send the tokens of exactly the nodes a read names, wildcards expanded to a bound with a probe per expanded array. Run unchanged against it, the module did not compile (a key now holds a list of tokens), and with that patched the positive control failed on wire_fields, token_reach, token_derivability, gated_walk_results and transcripts while every mutant still read as caught, which is what a broken positive control looks like. L now states the self tag (one fresh name per entry, so unlinkable across rows and distinct within one), each read's token list with the node each token names in each row and what it reaches below, the probe flag, the budget, and the member or element rule each reached node's token takes. The simulator programs x and p with the real hash only where a sent token links them. Of the 27 distinguishers 26 are kept and gated_walk_results becomes subtree_results (with the sealed-root budget fallback); eight are new (stored_lengths, self_tag_equalities, token_counts, bound_recovery, token_hits, wildcard_positions, hit_rules, ancestor_links). The mutants go from 34 to 44: the two gate mutants are gone, four are redefined for token lists, and twelve are new, among them the pre-#1363 top-level token and member rule. All 44 are caught, the positive control passes on the handcrafted columns and 48 random ones, and string ordering terms are now modelled folded, as the code and the page have them. One finding against the page as #1363 published it: the database's member-rule-then-element-rule test on every reached node tells it whether each node below a token is an object member or an array element, which nothing stored says. The page now says so, and the_hit_rule_separates_it_from_the_real_scheme keeps the sentence load-bearing. The SQL comment that the server cannot tell a probe from any other token is not true of its place in the list or of what it finds; a_read_gives_away_its_bound_and_its_probes shows it. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql/docs/reference/json-tree-leakage.md | 2 +- .../tests/json_leakage/calibration.rs | 164 +++- .../stack-encrypt/tests/json_leakage/chain.rs | 77 +- .../tests/json_leakage/corpus.rs | 88 ++- .../tests/json_leakage/distinguishers.rs | 698 +++++++++++++----- .../stack-encrypt/tests/json_leakage/mod.rs | 190 +++-- .../tests/json_leakage/mutants.rs | 361 ++++++--- .../tests/json_leakage/observe.rs | 65 +- .../tests/json_leakage/profile.rs | 543 +++++++++----- .../stack-encrypt/tests/json_leakage/real.rs | 16 +- .../tests/json_leakage/simulator.rs | 281 ++++--- 11 files changed, 1749 insertions(+), 736 deletions(-) diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index 7ce635995..468068d76 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -36,7 +36,7 @@ A query hands the database a term derived from the plaintext it asks about, and - **Ordering at a path (`<`, `<=`, `>`, `>=`).** The path's selector and the bound's own ordering term, which compares with the stored terms exactly as they compare with each other: the database learns the bound's place in the order of the stored values at every ordered path, and whether it equals one. `ORDER BY ord_term(…)` sends no term and shows nothing the stored terms do not. - **A subtree read (`json_tree_subtree`, `json_tree_children`, `json_tree_child_count`).** A **token** for exactly the node the path names, which the client derives from the path alone. Reading `$.profile.address.city` sends that leaf's token and nothing that covers `$.profile`. No token over a top-level field or an array leaves the client unless that field or array is itself what is read. - **A wildcard** (`$.items[*].price`) is expanded by the client into one token per concrete path, up to a bound `N` (64 by default, set by the caller), plus a **probe** token for the element at index `N` of every array a wildcard expands. A probe that finds an element means the array is longer than the bound, and the read fails rather than return part of the answer. So the database learns which of the nodes exist in each row, which shows, up to `N`, which elements hold the rest of the path (each array's length, where the path ends at the element; the entry order already shows it), whether each array is longer than `N`, and the structure beneath each node it finds. It learns nothing about the elements' sibling fields. In a row where a probe finds an element, the database also walks that element's subtree, although the read then fails. Two wildcard levels send `N² + N + 1` tokens, `N + 1` of them probes, one at index `N` of each array the read expands: the `N` inner arrays and the outer one. - - **What a token is.** A capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find that node in any row (through its self tag `x`), see how the nodes beneath it nest, link each one to its parent, and derive the tokens of the nodes beneath. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. + - **What a token is.** A capability over the structure of its node's subtree in *every row of the column*: whoever holds it can find that node in any row (through its self tag `x`), see how the nodes beneath it nest, link each one to its parent, and derive the tokens of the nodes beneath. It also learns whether each node beneath is an object member or an array element, and each element's index: nothing stored says which a node is, but to derive a node's token the database tests its self tag against the member rule and then the element rule, and the one that matches says which; an element's index is the ordinal it is found at. It opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. - **The root.** A read of `$` sends the root's token, and a read of anything below `$` does not. With the sealed root on (the default) that token covers no document's structure: it finds the root entry, whose ciphertext is the document, and nothing beneath it, since the top-level fields chain from nothing. With it off, the root's token covers every row's whole document. - **`json_tree_children` and `json_tree_child_count`** take one token, of any node a path names, and send nothing else. Under the sealed root the root's token enumerates nothing (its count is NULL), for the same reason. diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index de1794295..65ac041c1 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -1,6 +1,6 @@ //! The checks: the full profile is indistinguishable from the real scheme -//! (the positive control), the page as written is not (the finding), and -//! every clause's removal is noticed (the calibration). +//! (the positive control), the page as #1363 published it is not (the +//! finding), and every clause's removal is noticed (the calibration). use proptest::test_runner::{Config, RngAlgorithm, TestCaseError, TestRng, TestRunner}; use serde_json::json; @@ -12,7 +12,7 @@ use super::corpus::{handcrafted, random_world, World}; use super::distinguishers::{self, walk_positions}; use super::mutants; use super::observe::ObservedWorld; -use super::profile::{flatten, leak, Clauses, Column, KeyOrder, Leakage}; +use super::profile::{flatten, leak, Clauses, Column, KeyOrder, Leakage, Query, StringOrder}; use super::{real, simulator}; use crate::common::stack_cipher; @@ -142,15 +142,31 @@ mod given_the_full_profile { } } -mod given_the_page_as_written { +mod given_the_page_as_published_in_1363 { use super::*; - /// The finding, held as a test: with only what the page states, a - /// distinguisher separates the simulated world from the real one, and - /// it is the ordering term's length. When the page states it, move the - /// clause into the page's profile and this test changes with it. + /// The finding, held as a test: with only what the page stated at + /// #1363's head, which says nothing of the rule a node's token takes + /// under its parent's, a distinguisher separates the simulated world + /// from the real one, and `hit_rules` is the one that names it. This + /// branch adds the sentence to the page ("whether each node beneath is + /// an object member or an array element, and each element's index"); + /// the test keeps it load-bearing. + /// + /// The rest of the set is what follows from it. A world that states no + /// rule gives every member's token by the element rule, with its keyed + /// rank as the index. But a read through a key step sends a token + /// derived by the member rule, since that is the only token a client + /// can derive from a path; so where such a read names a node below a + /// member that another read's walk reaches (`$.nested.list[*].y[*]` + /// below `$.nested.list`'s walk), the walk derives the member by the + /// element rule and then fails to recognise the named node beneath it: + /// what the walk links (`token_reach`), what the read returns + /// (`subtree_results`) and which tokens derive which + /// (`token_derivability`) all change. No world consistent with the + /// tokens a client sends hides the rule: the page has to state it. #[test] - fn the_ordering_term_length_separates_it_from_the_real_scheme() { + fn the_hit_rule_separates_it_from_the_real_scheme() { let runtime = runtime(); let cipher = runtime.block_on(stack_cipher()); let mut fired = std::collections::BTreeSet::new(); @@ -158,7 +174,7 @@ mod given_the_page_as_written { let real = real_world(&runtime, &cipher, &world); let leakage = leak( &world.column, - Clauses::page_only(), + Clauses::as_published_in_1363(), &world.docs, &world.queries, ); @@ -171,10 +187,55 @@ mod given_the_page_as_written { } assert_eq!( fired.into_iter().collect::>(), - ["ordering_term_lengths"], - "the page as written is separated from the real scheme by exactly the ordering term's length" + PAGE_AS_PUBLISHED_FIRES, + "the page as #1363 published it is separated from the real scheme by the hit rule" ); } + + /// What separates the page as #1363 published it from the real scheme. + const PAGE_AS_PUBLISHED_FIRES: &[&str] = &[ + "hit_rules", + "subtree_results", + "token_derivability", + "token_reach", + ]; +} + +mod given_strings_ordered_by_their_bytes { + use super::*; + + /// The page says strings are "ordered and compared after their case and + /// accents are folded", and that the term's length counts the folded + /// form. Read as the strings' own bytes (which the profile first + /// assumed, on a corpus of lower-case ASCII where the two agree), the + /// profile is told from the real scheme on the column whose names fold: + /// `"Carol"` and `"carol"` share a term, and `"ál"`'s term is as long + /// as `"al"`'s. + #[test] + fn the_ordering_terms_tell_it_apart() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let clauses = Clauses { + string_order: StringOrder::Bytes, + ..Clauses::default() + }; + let world = handcrafted() + .into_iter() + .find(|w| w.name == "ordered strings") + .expect("the corpus has it"); + let real = real_world(&runtime, &cipher, &world); + let leakage = leak(&world.column, clauses, &world.docs, &world.queries); + let fired: Vec<&str> = separating(&features(&real), &simulated_world(&leakage, &mut rng())) + .into_iter() + .map(|(n, _, _)| n) + .collect(); + for expected in ["ordering_term_lengths", "ordering_term_order"] { + assert!( + fired.contains(&expected), + "folding shows in {expected}: {fired:?}" + ); + } + } } mod given_the_jsonb_reading_of_the_key_order { @@ -281,14 +342,15 @@ mod given_a_mutant_profile { mod the_harness { use super::*; - /// The harness's hash chain is the crate's: walking a sealed document's - /// observed bytes from a subtree key's token finds exactly the entries - /// `JsonDocument::subtree` does, at the same depths. + /// The harness's hash chain is the crate's: reading a sealed + /// document's observed bytes with a subtree key's tokens finds exactly + /// the entries `JsonDocument::subtree` does, at the same depths, for + /// paths through members, positions and wildcards, and `$`. #[test] fn chain_is_the_crates_chain() { let runtime = runtime(); let cipher = runtime.block_on(stack_cipher()); - let doc = json!({"a": {"b": [1, {"c": 2}], "d": "e"}, "f": [[3], {"g": null}]}); + let doc = json!({"a": {"b": [1, {"c": 2}], "d": "e"}, "f": [[3], {"g": null}, [4, 5]]}); for sealed_root in [true, false] { let options = JsonOptions { sealed_root, @@ -307,26 +369,39 @@ mod the_harness { let sealed: stack_encrypt::json::JsonDocument = serde_json::from_value(rows[0].clone()).expect("a document"); let observed = ObservedWorld::from_wire(&rows, &[]); - for path in ["$.a", "$.f", "$.f[1]", "$.f[0]"] { + for (path, bound) in [ + ("$", 64), + ("$.a", 64), + ("$.a.b", 64), + ("$.a.b[1].c", 64), + ("$.f", 64), + ("$.f[1]", 64), + ("$.f[0]", 64), + ("$.f[*]", 64), + ("$.f[*].g", 64), + ("$.f[*][*]", 1), + ] { let key = runtime .block_on(async { cipher .default_keyset() - .json_subtree_key( + .json_subtree_key_with_bound( &JsonPath::parse(path).unwrap(), &options, + bound, stack_encrypt::nonempty!("users").with("profile"), ) .await }) .expect("derives"); + let tokens = key.token_bytes(); let crates: Vec<(usize, usize)> = sealed - .subtree(key.token_bytes()) + .subtree(&tokens) .iter() .map(|(j, _, d)| (*j, *d as usize)) .collect(); assert_eq!( - walk_positions(&observed.rows[0], key.token_bytes()), + walk_positions(&observed.rows[0], &tokens), crates, "{path} with the root sealed: {sealed_root}" ); @@ -338,6 +413,55 @@ mod the_harness { } } + /// What a wildcard read's token list gives away, on the real scheme: + /// its count is the bound (65 tokens, so `N = 64`, at one level; 4,161, + /// so `N = 64`, at two), the probe is the last token at its level and + /// finds the element where the path's other tokens find the leaf below + /// it, and which tokens find a node says which elements hold the rest of + /// the path. `json_tree_subtree`'s comment that "the server cannot tell + /// a probe from any other token" is true of a token's bytes and of + /// nothing else. + #[test] + fn a_read_gives_away_its_bound_and_its_probes() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let items = |n: usize| { + json!({"items": (0..n) + .map(|i| if i == 1 { json!({"sku": i}) } else { json!({"price": i, "tags": [i]}) }) + .collect::>()}) + }; + let world = World { + name: "items", + column: Column { + sealed_root: true, + ordered: Vec::new(), + }, + docs: vec![items(0), items(3), items(70)], + queries: vec![ + Query::subtree("$.items[*].price"), + Query::subtree("$.items[*].tags[*]"), + ], + }; + let observed = real_world(&runtime, &cipher, &world); + assert_eq!(distinguishers::token_counts(&observed), [65, 4161]); + // 4,161 is also one level at a bound of 4,160; which tokens find a + // node (the 65th finds `$.items[0].tags[64]`'s absence, not an + // item) is what tells the reading apart. + assert_eq!( + distinguishers::bound_recovery(&observed), + [vec![(1, 64)], vec![(1, 4160), (2, 64)]] + ); + let price = &distinguishers::wildcard_positions(&observed)[0]; + let price = price.as_ref().expect("a wildcard read"); + // No items; three, the second without a price; seventy, the + // second without a price, longer than the bound. + assert_eq!(price[0], (vec![], false, None)); + assert_eq!(price[1], (vec![0, 2], false, None)); + let mut most: Vec = (0..64).collect(); + most.retain(|&i| i != 1); + assert_eq!(price[2], (most, true, Some(true))); + } + /// The page's claim that the stored order reveals each row's shape holds /// on a column whose rows vary: the order-only attack of /// `distinguishers::depth_recovery` recovers every row's depth sequence diff --git a/packages/stack-encrypt/tests/json_leakage/chain.rs b/packages/stack-encrypt/tests/json_leakage/chain.rs index af0067f75..668bd013b 100644 --- a/packages/stack-encrypt/tests/json_leakage/chain.rs +++ b/packages/stack-encrypt/tests/json_leakage/chain.rs @@ -1,4 +1,4 @@ -//! The token chain's two public hash functions, written out from the design +//! The token chain's public hash functions, written out from the design //! rather than imported, because both sides of the harness need them on //! bytes they observe: the database's half of the walk recomputes them over //! a stored row's `iv`, a token it was sent and the selectors it holds, and @@ -7,15 +7,15 @@ //! These are the only functions of the real scheme the harness evaluates. //! Everything keyed (selectors, value selectors, head tokens, member ranks) //! stays on the client and is replaced by randomness in the simulator. -//! `the_harness_chain_is_the_crates_chain` (in `calibration.rs`) holds these -//! two to the crate's own walk, `JsonDocument::subtree`. +//! `chain_is_the_crates_chain` (in `calibration.rs`) holds them to the +//! crate's own walk, `JsonDocument::subtree`. use sha2::{Digest, Sha256}; /// A sixteen-byte tag: a selector, a value selector or a token. pub type Tag16 = [u8; 16]; -/// An eight-byte parent tag. +/// An eight-byte tag: a parent tag `p` or a self tag `x`. pub type Tag8 = [u8; 8]; fn sha256(parts: &[&[u8]]) -> [u8; 32] { @@ -28,22 +28,64 @@ fn sha256(parts: &[&[u8]]) -> [u8; 32] { out } +fn first(full: [u8; 32]) -> [u8; N] { + let mut tag = [0; N]; + tag.copy_from_slice(&full[..N]); + tag +} + /// `H8("p" ‖ iv ‖ t(parent) ‖ LE32(o))`: the parent tag of the child at /// ordinal `o` under the node whose token is `parent`. pub fn parent_tag(iv: &Tag16, parent: &Tag16, ordinal: u32) -> Tag8 { - let full = sha256(&[b"p", iv, parent, &ordinal.to_le_bytes()]); - let mut tag = [0; 8]; - tag.copy_from_slice(&full[..8]); - tag + first(sha256(&[b"p", iv, parent, &ordinal.to_le_bytes()])) } -/// `H16("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`: the token of the child at -/// ordinal `o`, whose selector is `selector`. -pub fn child_token(parent: &Tag16, selector: &Tag16, ordinal: u32) -> Tag16 { - let full = sha256(&[b"t", parent, selector, &ordinal.to_le_bytes()]); - let mut tag = [0; 16]; - tag.copy_from_slice(&full[..16]); - tag +/// `H8("x" ‖ iv ‖ t(self))`: the self tag of the node whose token is +/// `token`, by which the database finds that node in a row. +pub fn self_tag(iv: &Tag16, token: &Tag16) -> Tag8 { + first(sha256(&[b"x", iv, token])) +} + +/// `H16("t" ‖ t(parent) ‖ s(member))`: an object member's token. No +/// ordinal, so the client derives it from the path alone. +pub fn member_token(parent: &Tag16, selector: &Tag16) -> Tag16 { + first(sha256(&[b"t", parent, selector])) +} + +/// `H16("t" ‖ t(array) ‖ s(element) ‖ LE32(i))`: an array element's token, +/// `i` its index. This is the pre-#1363 chain's rule for every child, kept +/// for elements, whose ordinal was always their index. +pub fn element_token(array: &Tag16, selector: &Tag16, index: u32) -> Tag16 { + first(sha256(&[b"t", array, selector, &index.to_le_bytes()])) +} + +/// Which rule a hit's self tag matched, as the database tests them: the +/// member rule first, then the element rule with the hit's ordinal. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum Rule { + Member, + Element, +} + +/// The token the database derives for a hit found at `ordinal` under the +/// node whose token is `parent`, and the rule that gave it: the member +/// rule's if the hit's self tag `x` is that token's, else the element +/// rule's if it is that one's, else none +/// (`eql_v4_internal.json_tree_hit_token`). +pub fn hit_token( + iv: &Tag16, + parent: &Tag16, + selector: &Tag16, + x: Option<&Tag8>, + ordinal: u32, +) -> Option<(Tag16, Rule)> { + let x = x?; + let member = member_token(parent, selector); + if self_tag(iv, &member) == *x { + return Some((member, Rule::Member)); + } + let element = element_token(parent, selector, ordinal); + (self_tag(iv, &element) == *x).then_some((element, Rule::Element)) } /// The simulator's stand-in for the keyed member rank: any keyed function of @@ -51,8 +93,5 @@ pub fn child_token(parent: &Tag16, selector: &Tag16, ordinal: u32) -> Tag16 { /// is a PRF under a key the database never sees. The simulator samples its /// own `key`. pub fn simulated_rank(key: &[u8; 32], parent: &Tag16, selector: &Tag16) -> Tag16 { - let full = sha256(&[b"rank", key, parent, selector]); - let mut tag = [0; 16]; - tag.copy_from_slice(&full[..16]); - tag + first(sha256(&[b"rank", key, parent, selector])) } diff --git a/packages/stack-encrypt/tests/json_leakage/corpus.rs b/packages/stack-encrypt/tests/json_leakage/corpus.rs index 70f7ee918..3c909ecfe 100644 --- a/packages/stack-encrypt/tests/json_leakage/corpus.rs +++ b/packages/stack-encrypt/tests/json_leakage/corpus.rs @@ -45,18 +45,26 @@ fn shapes() -> Vec { } fn shape_queries(sealed_root: bool) -> Vec { - // `$.nested` is only ever read through gates, so its token is sent but - // nothing under it is named whole: what that token reaches beyond the - // paths asked is the page's subtree-read clause. `$.zeta` is read twice, - // for determinism. + // Reads below a key step (`$.nested.deep`), through positions, and + // through one and two wildcard levels, at the default bound (65 tokens) + // and at bounds the rows overflow, so that probes find elements and the + // database walks them. `$.nested` is read under a budget its rows + // overflow, so the sealed root falls back. `$.zeta` is read twice, for + // determinism. `$` is a read under either root. let mut queries = vec![ - Query::Subtree("$.nested.list".into()), - Query::Subtree("$.nested.list[1]".into()), - Query::Subtree("$.nested.list[*].x".into()), - Query::Subtree("$.zeta".into()), - Query::Subtree("$[1]".into()), - Query::Subtree("$[1][1]".into()), - Query::Subtree("$.zeta".into()), + Query::subtree("$.nested.list"), + Query::subtree("$.nested.list[1]"), + Query::subtree("$.nested.list[*].x"), + Query::subtree_with("$.nested.list[*].x", 1, 64), + Query::subtree("$.nested.deep"), + Query::subtree_with("$.nested.list[*].y[*]", 2, 64), + Query::subtree_with("$.nested", 64, 3), + Query::subtree("$.zeta"), + Query::subtree("$[1]"), + Query::subtree("$[1][1]"), + Query::subtree_with("$[*].x", 3, 64), + Query::subtree("$"), + Query::subtree("$.zeta"), Query::Extract("$.nested.list[0].x".into()), Query::Extract("$.nested.list[1].x".into()), Query::Extract("$.hello".into()), @@ -74,9 +82,8 @@ fn shape_queries(sealed_root: bool) -> Vec { ]; if !sealed_root { queries.extend([ - Query::Subtree("$".into()), - Query::Subtree("$.a.b".into()), - Query::Subtree("$[*].x".into()), + Query::subtree("$.a.b"), + Query::subtree_with("$[*][*]", 2, 64), ]); } queries @@ -84,7 +91,9 @@ fn shape_queries(sealed_root: bool) -> Vec { /// A column that orders strings, so the string half of the ordering clause /// has something to compare (a world orders one kind of leaf; see -/// [`super::profile::Ids::ordering_ranks`]). +/// [`super::profile::Ids::ordering_ranks`]). `"Carol"` and `"carol"`, and +/// `"ál"` and `"al"`, fold to one value, so their terms are equal and +/// `"ál"`'s length is that of the two bytes it folds to. fn names() -> World { World { name: "ordered strings", @@ -98,12 +107,16 @@ fn names() -> World { json!({"name": "carol", "age": 3}), json!({"name": "bob", "tags": []}), json!({"tags": ["zzzzzzzz", "b"]}), + json!({"name": "Carol", "tags": ["B", "aa", "Ab"]}), + json!({"name": "ál"}), ], queries: vec![ Query::Order("$.name".into(), json!("b")), Query::Order("$.tags[*]".into(), json!("aa")), - Query::Subtree("$.tags".into()), - Query::Subtree("$.tags[1]".into()), + Query::Order("$.name".into(), json!("AL")), + Query::subtree("$.tags"), + Query::subtree("$.tags[1]"), + Query::subtree_with("$.tags[*]", 2, 64), ], } } @@ -126,13 +139,18 @@ fn wide() -> World { ordered: Vec::new(), }, docs: vec![ - json!({"w1": object("a", 8), "w2": object("b", 8), "w3": object("c", 8)}), - json!({"w1": object("a", 5), "w3": object("c", 8)}), + json!({"w1": object("a", 8), "w2": object("b", 8), "w3": object("c", 8), + "arr": [{"k": 1}, {"k": {"z": 2}, "m": [3]}]}), + json!({"w1": object("a", 5), "w3": object("c", 8), "arr": [{"k": 4}]}), ], + // `$.arr[*].k` at a bound of one: the probe `$.arr[1]` finds the + // first row's second element, and no other read names anything in + // it, so only the probe links its subtree. queries: vec![ - Query::Subtree("$.w1".into()), - Query::Subtree("$.w2".into()), - Query::Subtree("$.w3".into()), + Query::subtree("$.w1"), + Query::subtree("$.w2"), + Query::subtree("$.w3"), + Query::subtree_with("$.arr[*].k", 1, 64), ], } } @@ -232,8 +250,9 @@ fn document() -> impl Strategy { .prop_map(numberize) } -fn query(sealed_root: bool) -> impl Strategy { - let mut subtree = vec![ +fn query() -> impl Strategy { + let subtree = vec![ + "$", "$.a", "$.list", "$.list[0]", @@ -247,12 +266,23 @@ fn query(sealed_root: bool) -> impl Strategy { "$[1].a", "$[0][1]", "$.a.tags[*]", + "$[*].a", + "$.list[*]", + "$.list[*].tags[*]", ]; - if !sealed_root { - subtree.extend(["$", "$[*].a", "$.list[*]"]); - } prop_oneof![ - 3 => prop::sample::select(subtree).prop_map(|p| Query::Subtree(p.into())), + 3 => ( + prop::sample::select(subtree), + prop::sample::select(vec![1u32, 2, 3, 64]), + prop::sample::select(vec![0u32, 2, 64]), + ) + // Two wildcard levels at the default bound send 4,161 tokens; + // the harness keeps them small, and `a_read_gives_away_its_bound` + // sends the full count once. + .prop_map(|(p, bound, budget)| { + let bound = if p.matches("[*]").count() > 1 { bound.min(3) } else { bound }; + Query::subtree_with(p, bound, budget) + }), 2 => prop::sample::select(vec!["$", "$.a", "$.a.b", "$.list[0].n", "$.list[3].n", "$.tags[*]"]) .prop_map(|p| Query::Extract(p.into())), 2 => prop_oneof![ @@ -276,7 +306,7 @@ pub fn random_world() -> impl Strategy { any::().prop_flat_map(|sealed_root| { ( prop::collection::vec(document(), 2..7), - prop::collection::vec(query(sealed_root), 4..11), + prop::collection::vec(query(), 4..11), ) .prop_map(move |(docs, queries)| World { name: "random", diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 22fdf3bfc..8311a45c1 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -7,9 +7,9 @@ //! equality pattern give the same feature whatever their bytes. A real world //! and a simulated one are compared feature by feature for equality; no //! statistics are involved. The one source of chance is a collision of -//! random bytes (an eight-byte parent tag colliding with another, say), which -//! changes a feature with probability below `n² / 2^64` for `n` tags in a -//! world, about `10^-13` at the harness's sizes. +//! random bytes (an eight-byte parent or self tag colliding with another, +//! say), which changes a feature with probability below `n² / 2^64` for `n` +//! tags in a world, about `10^-13` at the harness's sizes. //! //! What is deliberately not here, and why, is in the module docs //! (`mod.rs`, "Out of reach"). @@ -17,8 +17,10 @@ use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::hash::Hash; -use super::chain::{child_token, parent_tag, Tag16, Tag8}; -use super::observe::{ObservedGate, ObservedGroup, ObservedQuery, ObservedRow, ObservedWorld}; +use super::chain::{ + element_token, hit_token, member_token, parent_tag, self_tag, Rule, Tag16, Tag8, +}; +use super::observe::{ObservedGroup, ObservedQuery, ObservedRow, ObservedWorld}; /// One distinguisher: a name and the feature it computes, printed so that /// any two features compare as strings and a mismatch reads as a diff. @@ -47,20 +49,28 @@ registry!( ciphertext_lengths, ciphertext_length_multisets, ciphertext_framing, + stored_lengths, selector_pattern_within_rows, selector_pattern_across_rows, parent_tag_equalities, + self_tag_equalities, headers, projections, value_selector_partitions, chain_head_runs, selector_precedence, depth_recovery, + token_counts, + bound_recovery, + token_hits, + wildcard_positions, token_reach, + hit_rules, + ancestor_links, ordinal_patterns, member_ordinals_follow_key_order, token_derivability, - gated_walk_results, + subtree_results, transcripts, extraction_matches, containment_matches, @@ -180,6 +190,19 @@ pub fn ciphertext_framing(world: &ObservedWorld) -> (BTreeSet, usize) { (frames, pairs) } +/// Each row's length in characters as compact JSON, and each entry's: what +/// the stored size says, with every tag's and term's fixed width included. +/// The self tag adds 19 characters to every entry (`,"x":"` and twelve +/// base64 characters and a quote; 21 in PostgreSQL's `jsonb` text form, +/// which writes `, "x": "…"`), a constant, so it adds nothing that varies. +pub fn stored_lengths(world: &ObservedWorld) -> Vec<(usize, Vec)> { + world + .rows + .iter() + .map(|r| (r.chars, r.entries.iter().map(|e| e.chars).collect())) + .collect() +} + // ============================================================================= // At rest: equality patterns // ============================================================================= @@ -233,6 +256,40 @@ pub fn parent_tag_equalities(world: &ObservedWorld) -> Vec<((usize, usize), (usi out } +/// An eight-byte tag's place: the row, the entry, and the field (`x` or +/// `p`). +pub type TagAt = (usize, usize, char); + +/// Every pair of entries, within a row or across rows, whose self tag equals +/// another self tag or a parent tag anywhere in the world. The page: the +/// self tag "is unrelated bytes in every row and links no entry to +/// another", so this is expected empty: each is a hash of a distinct +/// token under its row's `iv`, in its own domain (`"x"`, where the parent +/// tag's is `"p"`). +pub fn self_tag_equalities(world: &ObservedWorld) -> Vec<(TagAt, TagAt)> { + let mut seen: HashMap> = HashMap::new(); + for (r, row) in world.rows.iter().enumerate() { + for (j, entry) in row.entries.iter().enumerate() { + if let Some(x) = entry.x { + seen.entry(x).or_default().push((r, j, 'x')); + } + if let Some(p) = entry.p { + seen.entry(p).or_default().push((r, j, 'p')); + } + } + } + let mut out: Vec<_> = seen + .into_values() + .filter(|at| at.len() > 1 && at.iter().any(|(_, _, f)| *f == 'x')) + .flat_map(|at| { + let first = at[0]; + at.into_iter().skip(1).map(move |other| (first, other)) + }) + .collect(); + out.sort_unstable(); + out +} + /// Each header's public fields, and the equality patterns of its keyset, /// `iv` and key tag across rows. Catches a shared `iv` (which would link /// every row's parent tags) and a keyset that varies within a column. @@ -465,61 +522,89 @@ pub fn depth_recovery(world: &ObservedWorld) -> Vec>> { } // ============================================================================= -// Under a token: the walk, with the real hash +// Under a token: the database's walk, with the real hash // ============================================================================= +/// A row's two lookups, as `eql_v4_internal.json_tree_index` builds them: +/// self tag to position and parent tag to position, the last entry winning +/// a repeated tag (`jsonb_object_agg` keeps the last value of a repeated +/// key). +struct RowIndex { + x: HashMap, + p: HashMap, +} + +fn row_index(row: &ObservedRow) -> RowIndex { + let mut index = RowIndex { + x: HashMap::new(), + p: HashMap::new(), + }; + for (j, entry) in row.entries.iter().enumerate() { + if let Some(x) = entry.x { + let _ = index.x.insert(x, j); + } + if let Some(p) = entry.p { + let _ = index.p.insert(p, j); + } + } + index +} + /// One entry a walk from a token reaches. #[derive(Clone, Copy, Debug)] struct Hit { position: usize, - /// The position of the hit's parent, or `None` for a child of the - /// token's own node, which the walk never identifies. + /// The position of the hit's parent, `None` for the node the token + /// names. parent: Option, - ordinal: u32, + /// The ordinal the hit was probed at under its parent. + ordinal: Option, depth: usize, - token: Tag16, + /// The hit's own token, when the database has one: the token sent, for + /// the node it names, or the one the member rule or the element rule + /// gives a hit whose self tag matches it. A hit with neither is returned + /// and not descended into. + token: Option, + /// The rule that gave a hit below the named node its token. + rule: Option, + /// The token of the hit's parent (for the named node, its own). parent_token: Tag16, } +/// A child as the database finds it: its position, the ordinal it was +/// probed at, and its token and the rule that gave it, if either matches. +type Child = (usize, u32, Option<(Tag16, Rule)>); + /// The children of the node whose token is `token`, as the database finds -/// them: the parent tag of ordinal `0, 1, 2, …` until the first miss. -fn children( - row: &ObservedRow, - index: &HashMap, - token: &Tag16, -) -> Vec<(usize, u32, Tag16)> { +/// them: the parent tag of ordinal `0, 1, 2, …` until the first miss, each +/// with the token the member rule or the element rule gives it. +fn children(row: &ObservedRow, index: &RowIndex, token: &Tag16) -> Vec { + let iv = &row.header.iv; let mut out = Vec::new(); for o in 0..u32::try_from(row.entries.len()).unwrap_or(u32::MAX) { - let Some(&j) = index.get(&parent_tag(&row.header.iv, token, o)) else { + let Some(&j) = index.p.get(&parent_tag(iv, token, o)) else { break; }; - out.push((j, o, child_token(token, &row.entries[j].s, o))); + let entry = &row.entries[j]; + out.push((j, o, hit_token(iv, token, &entry.s, entry.x.as_ref(), o))); } out } -fn parent_index(row: &ObservedRow) -> HashMap { - let mut index = HashMap::new(); - for (j, entry) in row.entries.iter().enumerate() { - if let Some(p) = entry.p { - let _ = index.entry(p).or_insert(j); - } - } - index -} - -/// Everything below `token` in `row`, depth first, bounded as the SQL walk -/// is: no deeper than the index seals, and no more hits than entries. -fn walk(row: &ObservedRow, token: &Tag16) -> Vec { +/// The node `token` names in `row`, found by its self tag, then everything +/// below it, depth first, bounded as the crate's walk (`JsonDocument::subtree`) +/// and the SQL's are: no deeper than the index seals, and no more hits than +/// entries. +fn walk_with(row: &ObservedRow, index: &RowIndex, token: &Tag16) -> Vec { fn go( row: &ObservedRow, - index: &HashMap, + index: &RowIndex, token: &Tag16, - parent: Option, + parent: usize, depth: usize, out: &mut Vec, ) { - if depth > 128 || out.len() >= row.entries.len() { + if out.len() >= row.entries.len() || depth > 127 { return; } for (j, o, child) in children(row, index, token) { @@ -528,57 +613,276 @@ fn walk(row: &ObservedRow, token: &Tag16) -> Vec { } out.push(Hit { position: j, - parent, - ordinal: o, + parent: Some(parent), + ordinal: Some(o), depth, - token: child, + token: child.map(|(t, _)| t), + rule: child.map(|(_, rule)| rule), parent_token: *token, }); - go(row, index, &child, Some(j), depth + 1, out); + if let Some((child, _)) = child { + go(row, index, &child, j, depth + 1, out); + } } } - let index = parent_index(row); - let mut out = Vec::new(); - go(row, &index, token, None, 1, &mut out); + let Some(&at) = index.x.get(&self_tag(&row.header.iv, token)) else { + return Vec::new(); + }; + let mut out = vec![Hit { + position: at, + parent: None, + ordinal: None, + depth: 0, + token: Some(*token), + rule: None, + parent_token: *token, + }]; + go(row, index, token, at, 1, &mut out); out } -fn subtree_queries(world: &ObservedWorld) -> Vec<(usize, Tag16, &[ObservedGate])> { +/// The subtree reads of a world: each one's place among the queries, its +/// tokens and its budget. +fn subtree_queries(world: &ObservedWorld) -> Vec<(usize, &[Tag16], u32)> { world .queries .iter() .enumerate() .filter_map(|(i, q)| match q { - ObservedQuery::Subtree { t, gates } => Some((i, *t, gates.as_slice())), + ObservedQuery::Subtree { tokens, budget } => Some((i, tokens.as_slice(), *budget)), _ => None, }) .collect() } -/// One entry a token links: its position, its parent's position (`None` -/// for a child of the token's own node) and its depth below that node. -pub type Link = (usize, Option, usize); +/// For every subtree read, every row, and every token of the read that +/// finds a node there, the walk: computed once per world and handed to the +/// features that read it. +type Walks = Vec)>>>; -/// For every subtree read and every row, what the token links: each entry -/// the walk from the token reaches, its parent among them, and its depth -/// below the token's node, found by recomputing the real parent tags over -/// the row's `iv`. Catches a token that reaches more (or less) than the -/// profile's reach clause says: the page says a token covers its node's -/// whole subtree, in every row, and not only the path asked for (#1360). -pub fn token_reach(world: &ObservedWorld) -> Vec>> { +fn walks(world: &ObservedWorld) -> Walks { + let indexes: Vec = world.rows.iter().map(row_index).collect(); subtree_queries(world) .into_iter() - .map(|(_, t, _)| { + .map(|(_, tokens, _)| { world .rows .iter() - .map(|row| { - let mut hits: Vec<_> = walk(row, &t) + .zip(&indexes) + .map(|(row, index)| { + tokens + .iter() + .enumerate() + .filter_map(|(k, t)| { + let hits = walk_with(row, index, t); + (!hits.is_empty()).then_some((k, hits)) + }) + .collect() + }) + .collect() + }) + .collect() +} + +/// How many tokens each subtree read sends: one for a path with no +/// wildcard, `N + 1` for one wildcard level, `N² + N + 1` for two. +pub fn token_counts(world: &ObservedWorld) -> Vec { + subtree_queries(world) + .into_iter() + .map(|(_, tokens, _)| tokens.len()) + .collect() +} + +/// The bound an observer recovers from a read's token count alone: every +/// `(levels, N)` with `N^levels + … + N + 1` equal to the count, up to three +/// levels (`(0, 0)` for a single token). One wildcard level at the default +/// bound sends 65 tokens, and 65 is `N + 1` for `N = 64` and nothing else; +/// two levels send 4,161, which is also one level at `N = 4160`, and which +/// tokens find nodes tells the two apart. +pub fn bound_recovery(world: &ObservedWorld) -> Vec> { + token_counts(world) + .into_iter() + .map(|count| { + let count = count as u64; + let mut out = Vec::new(); + if count == 1 { + out.push((0, 0)); + } + for levels in 1..=3u32 { + for n in 1..count { + let sum: u64 = (0..=levels).map(|k| n.pow(k)).sum(); + if sum == count { + out.push((levels, n)); + } + if sum >= count { + break; + } + } + } + out + }) + .collect() +} + +/// For every subtree read and every row, which of the read's tokens find a +/// node there (by its place in the list) and where: "which of the nodes +/// exist in each row". Catches a token that is not column-wide, a token +/// list in another order, and a token that names another node. +pub fn token_hits(world: &ObservedWorld) -> Vec>> { + walks(world) + .into_iter() + .map(|rows| { + rows.into_iter() + .map(|found| { + found + .iter() + .map(|(k, hits)| (*k, hits[0].position)) + .collect() + }) + .collect() + }) + .collect() +} + +/// What a one-level wildcard read gives away, read as one: with `N` the +/// count less one, per row, the positions below `N` whose tokens find a +/// node (which elements hold the rest of the path, `$.items[*].price`'s +/// `price`), whether the last token, the probe, finds one (the array is +/// longer than `N`), and, where both found something, whether the probe's +/// node carries a selector none of the others does (the probe names the +/// element, the others the path below it: the probe is told from the other +/// tokens by what it finds as well as by its place). `None` for a read of +/// one token. +#[allow(clippy::type_complexity)] +pub fn wildcard_positions( + world: &ObservedWorld, +) -> Vec, bool, Option)>>> { + let counts = token_counts(world); + walks(world) + .into_iter() + .zip(counts) + .map(|(rows, count)| { + (count >= 2).then(|| { + let n = count - 1; + rows.iter() + .zip(&world.rows) + .map(|(found, row)| { + let positions: Vec = + found.iter().map(|(k, _)| *k).filter(|k| *k < n).collect(); + let probe = found.iter().find(|(k, _)| *k == n); + let told = probe.filter(|_| !positions.is_empty()).map(|(_, hits)| { + let s = row.entries[hits[0].position].s; + found + .iter() + .filter(|(k, _)| *k < n) + .all(|(_, h)| row.entries[h[0].position].s != s) + }); + (positions, probe.is_some(), told) + }) + .collect() + }) + }) + .collect() +} + +/// One entry a token links: its position, its parent's position (`None` +/// for the node the token names) and its depth below that node. +pub type Link = (usize, Option, usize); + +/// What one token of a read links in one row: its place in the list and +/// its links. +pub type Reached = (usize, Vec); + +/// For every subtree read, every row and every token that finds a node, +/// what the token links: the node, each entry the walk below it reaches, +/// its parent and its depth, found by recomputing the real self and parent +/// tags over the row's `iv`. Catches a token that reaches more (or less) +/// than the profile's reach clause says: a token covers its node's whole +/// subtree in every row, and nothing above it. +pub fn token_reach(world: &ObservedWorld) -> Vec>> { + walks(world) + .into_iter() + .map(|rows| { + rows.into_iter() + .map(|found| { + found + .into_iter() + .map(|(k, hits)| { + let mut links: Vec = hits + .into_iter() + .map(|h| (h.position, h.parent, h.depth)) + .collect(); + links.sort_unstable(); + (k, links) + }) + .collect() + }) + .collect() + }) + .collect() +} + +/// A reached node's rule: the place in the list of the token whose walk +/// reached it, its position, and the rule its self tag matched. +pub type RuleAt = (usize, usize, Option); + +/// For every node a walk reaches below a token, which rule its self tag +/// matched: the member rule (its parent is an object), the element rule +/// (an array), or neither. The database runs this test on every hit +/// (`json_tree_hit_token`), so it learns each reached node's parent's kind, +/// which the stored form keeps out (the design's "no kind bit"). +pub fn hit_rules(world: &ObservedWorld) -> Vec>> { + walks(world) + .into_iter() + .map(|rows| { + rows.into_iter() + .map(|found| { + let mut rules: Vec<_> = found .into_iter() - .map(|h| (h.position, h.parent, h.depth)) + .flat_map(|(k, hits)| { + hits.into_iter() + .filter(|h| h.depth > 0) + .map(move |h| (k, h.position, h.rule)) + }) .collect(); - hits.sort_unstable(); - hits + // The walk's order follows the members' keyed ranks. + rules.sort_unstable(); + rules + }) + .collect() + }) + .collect() +} + +/// For every subtree read, every row and every token that finds a node, +/// whether the database can link that node to its parent with any token it +/// holds: every token of every read, and every token its walks derive. The +/// page: no token "over a top-level field or an array leaves the client +/// unless that field or array is itself what is read", so a node's parent +/// is linked only where some read named one of its ancestors. Catches a +/// read that sends an ancestor's token, as the gated read before #1363 sent +/// its top-level field's. +pub fn ancestor_links(world: &ObservedWorld) -> Vec>> { + let walks = walks(world); + let below: Vec> = (0..world.rows.len()) + .map(|r| { + walks + .iter() + .flat_map(|rows| rows[r].iter()) + .flat_map(|(_, hits)| hits.iter().filter(|h| h.depth > 0).map(|h| h.position)) + .collect() + }) + .collect(); + walks + .iter() + .map(|rows| { + rows.iter() + .enumerate() + .map(|(r, found)| { + found + .iter() + .map(|(k, hits)| (*k, below[r].contains(&hits[0].position))) + .collect() }) .collect() }) @@ -598,42 +902,44 @@ pub fn ordinal_patterns(world: &ObservedWorld) -> (usize, Vec<(usize, usize, usi let mut tokens = Canon::new(); let mut out_of_order = 0; let mut ranked: BTreeMap<(usize, usize, usize), BTreeSet> = BTreeMap::new(); - for (_, t, _) in subtree_queries(world) { - for row in &world.rows { - let hits = walk(row, &t); - let mut by_parent: BTreeMap<(Option, Tag16), Vec> = BTreeMap::new(); - for hit in hits { - by_parent - .entry((hit.parent, hit.parent_token)) - .or_default() - .push(hit); - } - for ((_, parent_token), mut kids) in by_parent { - kids.sort_by_key(|h| h.position); - let same = kids - .windows(2) - .all(|w| row.entries[w[0].position].s == row.entries[w[1].position].s); - if same - && kids - .iter() - .enumerate() - .any(|(i, h)| h.ordinal as usize != i) - { - out_of_order += 1; + for rows in walks(world) { + for (r, found) in rows.into_iter().enumerate() { + let row = &world.rows[r]; + for (_, hits) in found { + let mut by_parent: BTreeMap<(usize, Tag16), Vec> = BTreeMap::new(); + for hit in hits.into_iter().filter(|h| h.depth > 0) { + by_parent + .entry((hit.parent.expect("below the node"), hit.parent_token)) + .or_default() + .push(hit); } - if !same { - let parent = tokens.id(parent_token); - for a in &kids { - for b in &kids { - let (sa, sb) = ( - selectors.id(row.entries[a.position].s), - selectors.id(row.entries[b.position].s), - ); - if sa < sb { - let _ = ranked - .entry((parent, sa, sb)) - .or_default() - .insert(a.ordinal < b.ordinal); + for ((_, parent_token), mut kids) in by_parent { + kids.sort_by_key(|h| h.position); + let same = kids + .windows(2) + .all(|w| row.entries[w[0].position].s == row.entries[w[1].position].s); + if same + && kids + .iter() + .enumerate() + .any(|(i, h)| h.ordinal != Some(i as u32)) + { + out_of_order += 1; + } + if !same { + let parent = tokens.id(parent_token); + for a in &kids { + for b in &kids { + let (sa, sb) = ( + selectors.id(row.entries[a.position].s), + selectors.id(row.entries[b.position].s), + ); + if sa < sb { + let _ = ranked + .entry((parent, sa, sb)) + .or_default() + .insert(a.ordinal < b.ordinal); + } } } } @@ -664,68 +970,76 @@ pub fn ordinal_patterns(world: &ObservedWorld) -> (usize, Vec<(usize, usize, usi /// Only objects directly under a token sent are counted, since which deeper /// nodes share a derived token depends on the ranks themselves. pub fn member_ordinals_follow_key_order(world: &ObservedWorld) -> Option { + let indexes: Vec = world.rows.iter().map(row_index).collect(); let mut chance = 1.0f64; let mut all_in_order = true; let mut sent = BTreeSet::new(); - for (_, t, _) in subtree_queries(world) { - if !sent.insert(t) { - continue; - } - let mut widest = 0usize; - for row in &world.rows { - let mut kids: Vec = walk(row, &t) - .into_iter() - .filter(|h| h.parent.is_none()) - .collect(); - let distinct: BTreeSet = - kids.iter().map(|h| row.entries[h.position].s).collect(); - if kids.len() < 2 || distinct.len() != kids.len() { + for (_, tokens, _) in subtree_queries(world) { + for t in tokens { + if !sent.insert(*t) { continue; } - widest = widest.max(kids.len()); - kids.sort_by_key(|h| h.position); - if kids - .iter() - .enumerate() - .any(|(i, h)| h.ordinal as usize != i) - { - all_in_order = false; + let mut widest = 0usize; + for (row, index) in world.rows.iter().zip(&indexes) { + let mut kids: Vec = walk_with(row, index, t) + .into_iter() + .filter(|h| h.depth == 1) + .collect(); + let distinct: BTreeSet = + kids.iter().map(|h| row.entries[h.position].s).collect(); + if kids.len() < 2 || distinct.len() != kids.len() { + continue; + } + widest = widest.max(kids.len()); + kids.sort_by_key(|h| h.position); + if kids + .iter() + .enumerate() + .any(|(i, h)| h.ordinal != Some(i as u32)) + { + all_in_order = false; + } } + chance /= (1..=widest).map(|k| k as f64).product::(); } - chance /= (1..=widest).map(|k| k as f64).product::(); } (chance < 2f64.powi(-40)).then_some(all_in_order) } /// Which subtree reads' tokens the database can derive from which others': -/// `(i, j)` when the token sent by query `j` is one the walk from query -/// `i`'s token derives in some row, or is the child token of `i`'s under -/// any selector the column holds and an ordinal below 8 (a bound on what -/// the harness tries, not on what the database can). The page: a token's -/// holder can "derive the tokens of the nodes beneath". Catches tokens that -/// are not chained (a positional token unrelated to its head's) as well as -/// tokens chained where the profile says they are not. +/// `(i, j)` when a token sent by query `j` is one the walks from query +/// `i`'s tokens derive in some row, or is a child token of one of `i`'s by +/// the member rule under any selector the column holds, or by the element +/// rule at an index below 4 (a bound on what the harness tries, not on what +/// the database can). The page: a token's holder can "derive the tokens of +/// the nodes beneath". Catches tokens that are not chained (a token +/// unrelated to its path's) as well as tokens chained where the profile +/// says they are not. pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { let queries = subtree_queries(world); + let walks = walks(world); let selectors: BTreeSet = world .rows .iter() .flat_map(|r| r.entries.iter().map(|e| e.s)) .collect(); let mut out = BTreeSet::new(); - for &(i, ti, _) in &queries { - let mut derived: BTreeSet = world - .rows + for (q, &(i, tokens, _)) in queries.iter().enumerate() { + let mut derived: BTreeSet = walks[q] .iter() - .flat_map(|row| walk(row, &ti).into_iter().map(|h| h.token)) + .flatten() + .flat_map(|(_, hits)| hits.iter().filter_map(|h| h.token)) .collect(); - for s in &selectors { - for o in 0..8 { - let _ = derived.insert(child_token(&ti, s, o)); + for t in tokens { + for s in &selectors { + let _ = derived.insert(member_token(t, s)); + for o in 0..4 { + let _ = derived.insert(element_token(t, s, o)); + } } } - for &(j, tj, _) in &queries { - if i != j && derived.contains(&tj) { + for &(j, others, _) in &queries { + if i != j && others.iter().any(|t| derived.contains(t)) { let _ = out.insert((i, j)); } } @@ -733,47 +1047,50 @@ pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { out } -/// What each subtree read returns from each row, as the SQL walk computes -/// it: from the token, through each gate (a child carrying the gate's -/// selector, or the child at ordinal `n` carrying it), then each node -/// reached with everything beneath it; with no gates, everything beneath the -/// token's node. -pub fn gated_walk_results(world: &ObservedWorld) -> Vec>> { - subtree_queries(world) +/// What one subtree read returns from one row. +#[derive(Debug, PartialEq, Eq)] +pub enum Returned { + /// `(k, j, d)`: the place in the token list of the token whose node the + /// row sits in, its position and its depth below that node, ordered by + /// `k` then `j`. + Rows(Vec<(usize, usize, usize)>), + /// The root entry alone: the root is sealed, the budget is not `0`, and + /// the read found more rows than the budget. + RootAlone, +} + +/// What each subtree read returns from each row, as `json_tree_subtree` +/// computes it: for each token in turn the node it names and everything +/// beneath it, or, past the budget under the sealed root, the root entry +/// alone. The database computes the fallback from the walk it already +/// sees, so it learns nothing from it; to someone who sees only the +/// results, a row answered with its root alone had more than `budget` rows +/// under the tokens. +pub fn subtree_results(world: &ObservedWorld) -> Vec> { + let budgets: Vec = subtree_queries(world) .into_iter() - .map(|(_, t, gates)| { - world - .rows - .iter() - .map(|row| { - let index = parent_index(row); - let mut frontier: Vec<(Option, Tag16)> = vec![(None, t)]; - for gate in gates { - let mut next = Vec::new(); - for (_, token) in &frontier { - let kids = children(row, &index, token); - match gate { - ObservedGate::Selector(s) => next.extend( - kids.iter() - .filter(|(j, _, _)| row.entries[*j].s == *s) - .map(|(j, _, child)| (Some(*j), *child)), - ), - ObservedGate::Slot(n, s) => next.extend( - kids.get(*n as usize) - .filter(|(j, _, _)| row.entries[*j].s == *s) - .map(|(j, _, child)| (Some(*j), *child)), - ), - } - } - frontier = next; - } - let mut rows = Vec::new(); - for (reached, token) in frontier { - rows.extend(reached); - rows.extend(walk(row, &token).into_iter().map(|h| h.position)); + .map(|(_, _, budget)| budget) + .collect(); + walks(world) + .into_iter() + .zip(budgets) + .map(|(rows, budget)| { + rows.into_iter() + .zip(&world.rows) + .map(|(found, row)| { + let mut out: Vec<(usize, usize, usize)> = found + .into_iter() + .flat_map(|(k, hits)| { + hits.into_iter().map(move |h| (k, h.position, h.depth)) + }) + .collect(); + let sealed = row.header.flags & 1 == 1; + if sealed && budget > 0 && out.len() > budget as usize { + Returned::RootAlone + } else { + out.sort_unstable(); + Returned::Rows(out) } - rows.sort_unstable(); - rows }) .collect() }) @@ -789,17 +1106,18 @@ pub fn gated_walk_results(world: &ObservedWorld) -> Vec>> { #[allow(dead_code)] pub enum NamedQuery { Extract(usize), - Subtree(usize, Vec<(Option, usize)>), + Subtree(Vec, u32), Contains(NamedGroup, Option), Equals(NamedGroup), Order(usize, usize), } /// Every query term, its tags named across the column and the other terms: -/// which terms repeat (a query is deterministic), which selectors a term -/// shares with the stored rows and with other terms (two paths that differ -/// only in an array position send one selector), a gate's position in the -/// clear, and a needle's shape. +/// which terms repeat (a query is deterministic, and a token is the same in +/// every read that names its node), which selectors a term shares with the +/// stored rows and with other terms (two paths that differ only in an array +/// position send one selector), a read's tokens in the order sent and its +/// budget, and a needle's shape. pub fn transcripts(world: &ObservedWorld) -> Vec { let mut selectors = selector_names(world); let mut values = Canon::new(); @@ -813,16 +1131,10 @@ pub fn transcripts(world: &ObservedWorld) -> Vec { .iter() .map(|q| match q { ObservedQuery::Extract { s } => NamedQuery::Extract(selectors.id(*s)), - ObservedQuery::Subtree { t, gates } => NamedQuery::Subtree( - tokens.id(*t), - gates - .iter() - .map(|g| match g { - ObservedGate::Selector(s) => (None, selectors.id(*s)), - ObservedGate::Slot(n, s) => (Some(*n), selectors.id(*s)), - }) - .collect(), - ), + ObservedQuery::Subtree { + tokens: sent, + budget, + } => NamedQuery::Subtree(sent.iter().map(|t| tokens.id(*t)).collect(), *budget), ObservedQuery::Contains { needle, alt } => NamedQuery::Contains( name_group(needle, &mut selectors, &mut values), alt.as_ref() @@ -983,11 +1295,13 @@ pub fn ordering_matches(world: &ObservedWorld) -> Vec>> { .collect() } -/// The positions and depths a walk from `token` reaches in `row`, in the -/// walk's order: what `JsonDocument::subtree` returns. -pub fn walk_positions(row: &ObservedRow, token: &Tag16) -> Vec<(usize, usize)> { - walk(row, token) - .into_iter() +/// The positions and depths a read of `tokens` reaches in `row`, token by +/// token in the walk's order: what `JsonDocument::subtree` returns. +pub fn walk_positions(row: &ObservedRow, tokens: &[Tag16]) -> Vec<(usize, usize)> { + let index = row_index(row); + tokens + .iter() + .flat_map(|t| walk_with(row, &index, t)) .map(|h| (h.position, h.depth)) .collect() } diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 8020b9743..1212ae806 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -6,6 +6,17 @@ //! real scheme reveals anything the claim leaves out, a test here fails and //! names the observation that gave it away. //! +//! It models the format of #1363: every entry `{s, c, x, p?, op?}` with a +//! self tag `x`; a member's token derived without an ordinal and an +//! element's with its index, so the client derives any node's token from +//! its path; and a subtree read, `json_tree_subtree(doc, tokens, budget)`, +//! that sends the tokens of exactly the nodes it names, wildcards expanded +//! on the client to a bound `N` with a probe per expanded array. It was +//! first written against the format before (one head token and plaintext +//! gates per read, member tokens carrying their ordinal, no self tag) and +//! ported; `docs/plans/2026-10-25-json-tree-leakage-before-and-after.md` +//! compares the two profiles clause by clause. +//! //! # The pieces //! //! ```text @@ -27,9 +38,10 @@ //! - [`simulator`] builds a world in the wire form from a //! [`profile::Leakage`] and randomness, with no plaintext and no key: //! random bytes for every name, random ciphertexts of the stated length, -//! ordering terms laid out in the stated order, and parent tags chained with -//! the real hash below every token a query sends, as far as the profile -//! says the token reaches. +//! ordering terms laid out in the stated order, and the chain programmed +//! with the real hash where the profile says a query's token links it: +//! the self tag `x` of each node a token names, and below it every parent +//! tag and self tag the token reaches. //! - [`real`] seals the same documents and derives the same query terms //! through the crate's public API, and serializes them. //! - [`observe`] parses either world's wire JSON into one @@ -43,8 +55,10 @@ //! - [`mutants`] are profiles with one clause dropped or coarsened. //! - [`corpus`] holds the handcrafted columns and a proptest generator of //! random ones over a shared key vocabulary. -//! - [`chain`] is the token chain's two public hash functions, the only part -//! of the real scheme the harness evaluates itself. +//! - [`chain`] is the token chain's public hash functions (the parent tag, +//! the self tag, the member and element rules, and the database's test of +//! a hit against the two rules), the only part of the real scheme the +//! harness evaluates itself. //! //! # The checks ([`calibration`]) //! @@ -54,11 +68,15 @@ //! something `L` does not state: that is a finding against the page (or //! against the code), and the failure names the distinguisher and prints //! both features. -//! - **The page as written.** `L` carries one clause the page does not state, -//! marked **beyond the page** in [`profile::Clauses`]. -//! [`profile::Clauses::page_only`] turns it off, and a test holds that the -//! result is then told from the real scheme, by exactly the distinguisher -//! that sees it. +//! - **The page as #1363 published it.** `L` carries one clause the page +//! did not state when #1363 published it, marked **added to the page by +//! #1362** in [`profile::Clauses`], and this branch adds the sentence. +//! [`profile::Clauses::as_published_in_1363`] turns it off, and a test +//! holds that the result is then told from the real scheme, by the +//! distinguisher that names it and the three its absence breaks. +//! - **Two readings.** The page states the key order (UTF-8 byte order) +//! and that strings order folded; a profile read the other way on either +//! is told from the real scheme, so both sentences are load-bearing. //! - **The calibration.** Each mutant profile drops one clause. The test //! expects some distinguisher to tell each mutant's world from the real //! one, and prints a table of which (run with `--no-capture`). A mutant @@ -66,62 +84,74 @@ //! none today. This is what shows the distinguishers can see what the //! clauses describe, so that the positive control passing means something. //! - **The harness itself.** The chain functions agree with the crate's own -//! walk (`JsonDocument::subtree`), and the order-only attack in +//! walk (`JsonDocument::subtree`) for reads through members, positions, +//! wildcards and `$`; the order-only attack in //! [`distinguishers::depth_recovery`] recovers every row's depth sequence //! on a column whose rows vary, which is the page's claim that the stored -//! order reveals each row's shape. -//! -//! # What the profile found against the page -//! -//! 1. **The ordering term's length is not on the page.** A CLLW OPE term is -//! `8 × n + 1` bytes for an `n`-byte encoding: every number's term is 65 -//! bytes, and a string's is `8 × (len + 1) + 1`. So at an ordered path -//! the term says which of number or string the leaf is (the page says -//! only that its presence says "a number or a string"), and repeats the -//! string's length. `the_ordering_term_length_separates_it_from_the_real_scheme`. -//! 2. **The page contradicts itself on equal values at two paths.** "Not -//! revealed at rest: […] whether two different paths hold the same value" -//! is false once two paths are ordered: the `op` row's "leaves at two -//! ordered paths compare with each other too" includes equality, and the -//! real terms of equal values at two ordered paths are equal. `L` follows -//! the `op` row; the mutant that follows the summary sentence -//! (`ordering_terms_per_path`) is told apart by `ordering_term_order`. -//! 3. **"The sorted order of their key names" is the keys' UTF-8 byte -//! order.** `jsonb` sorts shorter keys first; read that way, the profile -//! is told apart (`given_the_jsonb_reading_of_the_key_order`). A reader -//! inferring key names from the order needs the byte order. +//! order reveals each row's shape; and on the real scheme a wildcard +//! read's token list gives its bound and its probes away +//! (`a_read_gives_away_its_bound_and_its_probes`). +//! +//! # What the profile found +//! +//! Against the page as #1363 published it: +//! +//! 1. **A walk tells members from elements.** Nothing stored says whether a +//! node is an object member or an array element (the design's decision +//! 2, "no kind bit"), but to derive a reached node's token the database +//! tests its self tag against the member rule and then the element rule +//! (`json_tree_hit_token`), so under any token it sends, it learns which +//! each node is, and so whether each parent is an object or an array. +//! Where selectors show it anyway (an array with two elements in some +//! row) this adds nothing; for a node that is an only child in every +//! row, it does. The page said a token's holder can "see how the nodes +//! beneath it nest, link each one to its parent". This branch adds the +//! sentence, and `the_hit_rule_separates_it_from_the_real_scheme` holds +//! it: without it, `hit_rules` separates the worlds, and no world +//! consistent with the tokens a client sends can hide the rule. +//! +//! Against the SQL's own comment, not the page: +//! +//! 2. **A probe is told from the other tokens.** `json_tree_subtree`'s +//! comment says "the server cannot tell a probe from any other token". +//! It cannot from the token's bytes, but the probe is the last token at +//! its level, the count gives the bound away (65 tokens is one level at +//! `N = 64` and nothing else), and a probe that finds something finds +//! the element, whose selector differs from the leaf the path's other +//! tokens find. `a_read_gives_away_its_bound_and_its_probes`. The leakage +//! page does not repeat the claim; the comment is left to #1363. +//! +//! The three findings against the page as it stood before #1363 (an +//! ordering term's length, equal values at two ordered paths, UTF-8 byte +//! order of keys) are on the page now. Their clauses stay, each with a +//! mutant or a reading the calibration catches. The profile itself was +//! wrong in one place the old corpus could not show: it ordered strings by +//! their bytes, where the code (and the page) fold case and accents; the +//! corpus now holds folded names, and the reading test holds the sentence. //! //! Where the page is vague and `L` is as precise as the code forces, the -//! clause says so: what a token's holder sees in the ordinals it probes -//! ([`profile::EntryLeakage::step`]), the needle's `alt` -//! ([`profile::QueryLeakage::Contains`]), and the fixed parts of a -//! ciphertext's length ([`profile::ciphertext_len`]). -//! -//! The two issues the profile was built to check come out as the page -//! states them. The stored order reveals each row's shape (#1357): the order -//! clause is load-bearing (`order_random_permutation`, `members_unsorted`), -//! and the depth attack works. And a subtree read's token reaches its whole -//! top-level field in every row, not only the path asked (#1360): the -//! mutant that narrows the reach to the path (`token_reaches_named_nodes_only`) -//! is told apart by `token_reach`, which walks the real parent tags from the -//! real token. +//! clause says so: a member's ordinal under a token (its keyed rank), the +//! needle's `alt` ([`profile::QueryLeakage::Contains`]), and the fixed +//! parts of a ciphertext's length ([`profile::ciphertext_len`]). +//! +//! The two issues the profile was first built to check: the stored order +//! still reveals each row's shape (#1357; the order clause is load-bearing +//! and the depth attack works), and a subtree read no longer hands over +//! its top-level field (#1360): the mutant that sends the head's token as +//! the read before #1363 did (`ancestor_token_sent`) is told apart by +//! `ancestor_links`, `token_reach` and the token counts. //! //! # When the format changes //! -//! The format is about to change (`docs/plans/2026-10-17-json-tree-index.md`, -//! "Design change (2026-10-24)"): member tokens lose their ordinal, every -//! entry gains a self tag `x`, the client expands wildcards and gates go -//! away. `L` and the real-side adapter are expected to change with it, and -//! with the leakage page; the distinguishers should mostly not. To add or -//! change a clause: +//! `L` and the real-side adapter are expected to change with the format, +//! and with the leakage page; the distinguishers mostly should not (the +//! port to #1363 kept 26 of the 27 the profile had, renamed one and added +//! eight). To add or change a clause: //! //! 1. Change the page first, or at least decide what it will say. //! 2. Change [`profile`]: a field on the leakage type, with the page's //! sentence quoted in its doc comment, computed from the plaintext. //! 3. Make the [`simulator`] produce it from that field and randomness only. -//! For `x`, that is `H8("x" ‖ iv ‖ t(node))` on every entry whose token -//! some query's reach covers, and random bytes elsewhere, exactly as `p` -//! is handled now. //! 4. Teach [`observe`] the new wire field, and [`real`] any new API. //! 5. Add a distinguisher if no existing one would see the clause, and a //! mutant that drops the clause, and check the calibration catches it. @@ -133,28 +163,35 @@ //! distinguisher that separated the worlds, with the real and the //! simulated feature. Decide whether the page or the code is wrong. Never //! widen `L` silently to make it pass: a clause the page does not state -//! goes in as **beyond the page** ([`profile::Clauses`]), with a test like -//! the one for the ordering term's length, so the gap stays visible until -//! the page closes it. +//! goes in marked as such in [`profile::Clauses`], with a test like the +//! one for the hit rule, so the gap stays visible until the page closes +//! it. //! - **Calibration fails for a mutant nothing noticed.** Either the clause //! is not load-bearing on the calibration columns (add a column that -//! exercises it, as the wide-objects column does for member ranks), or no -//! distinguisher can see it (write one). If neither is possible, list the -//! mutant in `UNNOTICED` with the reason. Do not delete the clause. +//! exercises it, as the wide-objects column does for member ranks and +//! for a probe's reach), or no distinguisher can see it (write one). If +//! neither is possible, list the mutant in `UNNOTICED` with the reason. +//! Do not delete the clause. //! - **Calibration fails for a listed mutant that is now noticed.** Remove it //! from the list: something got sharper. +//! - **Every mutant is caught and the positive control fails.** The +//! calibration means nothing until the positive control passes: a profile +//! the real scheme is already told from makes every mutant "caught". +//! That is what happened when this module first ran against #1363's +//! format. //! //! # A tension this module states and does not resolve //! //! PR #1358's read check (`JsonSubtreeKey::open`) refuses a walk's rows //! unless each subtree's rows sit at consecutive positions `j` in `sv`, which //! is what a subtree is in a depth-first stored order, and `j` is bound into -//! every ciphertext's AAD. That consecutiveness is exactly the property -//! #1357 wants gone: a stored order independent of the shape (a keyed -//! permutation of `sv`, say) would scatter a subtree's rows. The check and -//! "order independent of shape" cannot both hold as written: one of them -//! has to give, and which is the owner's decision. The design change's -//! first open question discusses the same check from the other side. +//! every ciphertext's AAD. #1363 keeps it (its decision 8: "each deeper row +//! must sit at the next consecutive `j` within its subtree"). That +//! consecutiveness is exactly the property #1357 wants gone: a stored order +//! independent of the shape (a keyed permutation of `sv`, say) would +//! scatter a subtree's rows. The check and "order independent of shape" +//! cannot both hold as written: one of them has to give, and which is the +//! owner's decision. //! //! # Out of reach //! @@ -167,17 +204,20 @@ //! - **The keyed member rank as a permutation.** Under a token, a member's //! ordinal is a keyed rank. Its relation to the stored (key) order is a //! random permutation, decidable exactly only where enough wide objects sit -//! under the tokens sent; elsewhere the feature reports nothing. For the -//! same reason, which descendants of a token share a derived token across -//! rows depends on the ranks, so no feature compares it. +//! under the tokens sent; elsewhere the feature reports nothing. //! - **The primitives.** Ciphertexts beyond their length, selectors, value -//! selectors and head tokens beyond their equality patterns: the profile -//! assumes AES-GCM and the PRFs do their job, and the simulator draws -//! random bytes for them. +//! selectors, head tokens and unlinked self tags beyond their equality +//! patterns: the profile assumes AES-GCM, the PRFs and the hash do their +//! job, and the simulator draws random bytes for them. //! - **The database's side channels.** Timing, which rows a query touches -//! beyond its result, the walk's budget fallback, and the `i` and `v` -//! envelope fields the SQL adds. One column at a time; no cross-column -//! linkage is modelled. +//! beyond its result, and the `i` and `v` envelope fields the SQL adds. +//! The budget fallback is modelled (`distinguishers::subtree_results`); +//! the database computes it from the walk it already sees. One column at a +//! time; no cross-column linkage is modelled. +//! - **`json_tree_children` and `json_tree_child_count`.** Each takes one +//! token, the token a read of the same path sends, and probes one level +//! below it, which the read's walk already does; they are not modelled as +//! queries of their own. //! - **The key service.** The fake key source mints 32-byte tags; ZeroKMS's //! tag length is its own, and a column constant either way. //! diff --git a/packages/stack-encrypt/tests/json_leakage/mutants.rs b/packages/stack-encrypt/tests/json_leakage/mutants.rs index c722e163d..dd056223d 100644 --- a/packages/stack-encrypt/tests/json_leakage/mutants.rs +++ b/packages/stack-encrypt/tests/json_leakage/mutants.rs @@ -7,7 +7,10 @@ //! //! A mutant transforms a [`Leakage`] value. It never sees the plaintext, so //! it can only coarsen what `L` already states, which is the point: the -//! mutant is "the profile without this clause". +//! mutant is "the profile without this clause". A few state a format the +//! code does not have, to show the distinguishers would see it: the +//! pre-#1363 read's top-level token (`ancestor_token_sent`), the pre-#1363 +//! member rule (`member_tokens_carry_ordinal`). use std::collections::{BTreeMap, HashMap}; @@ -15,7 +18,8 @@ use proptest::prelude::RngCore; use proptest::test_runner::TestRng; use super::profile::{ - EntryLeakage, GateLeak, GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step, AEAD_OVERHEAD, + EntryLeakage, GroupLeakage, Leakage, QueryLeakage, ReadLeak, RowLeakage, Rule, Step, TokenName, + TokenStep, AEAD_OVERHEAD, }; /// One mutant: its name, the clause it drops (a field or sentence of the @@ -73,6 +77,15 @@ fn every_group(leakage: &mut Leakage, mut f: impl FnMut(&mut GroupLeakage)) { } } +/// Every token a subtree read sends, with the read's budget. +fn every_read(leakage: &mut Leakage, mut f: impl FnMut(&mut Vec)) { + for query in &mut leakage.queries { + if let QueryLeakage::Subtree { reads, .. } = query { + f(reads); + } + } +} + /// Reorder a row's entries: `order[k]` is the old position of the entry now /// at `k`. Parents and every query's positions follow. fn reorder(leakage: &mut Leakage, r: usize, order: &[usize]) { @@ -89,13 +102,16 @@ fn reorder(leakage: &mut Leakage, r: usize, order: &[usize]) { ..old[o].clone() }) .collect(); - for query in &mut leakage.queries { - if let QueryLeakage::Subtree { reach, named, .. } = query { - for at in reach[r].iter_mut().chain(named[r].iter_mut()) { + every_read(leakage, |reads| { + for read in reads { + if let Some(at) = &mut read.hits[r] { + *at = new_at[*at]; + } + for at in &mut read.reach[r] { *at = new_at[*at]; } } - } + }); } fn children(row: &RowLeakage) -> Vec> { @@ -112,6 +128,10 @@ fn children(row: &RowLeakage) -> Vec> { // At rest // ============================================================================= +// ============================================================================= +// At rest +// ============================================================================= + fn selectors_all_distinct(l: &mut Leakage, _: &mut TestRng) { let mut fresh = Fresh::new(); for row in &mut l.rows { @@ -178,8 +198,9 @@ fn entry_count_padded(l: &mut Leakage, _: &mut TestRng) { selector: fresh.next(), parent: Some(0), step: Step::Member, - head: None, + rule: Some(Rule::Member), has_parent_tag: true, + self_tag: Some(fresh.next()), ciphertext_len: AEAD_OVERHEAD + 10, ordering: None, }); @@ -437,13 +458,55 @@ fn keyset_per_row(l: &mut Leakage, _: &mut TestRng) { } } -/// Head tokens per row rather than column-wide. -fn tokens_per_row(l: &mut Leakage, _: &mut TestRng) { +/// No entry carries a self tag. +fn self_tags_absent(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + entry.self_tag = None; + } + } +} + +/// A self tag that is not salted by the row: the same node in two rows +/// carries the same one, named by its selector and its place among the +/// row's entries with that selector. +fn self_tags_linkable_across_rows(l: &mut Leakage, _: &mut TestRng) { let mut fresh = Fresh::new(); + let mut names: HashMap<(usize, usize), usize> = HashMap::new(); for row in &mut l.rows { + let mut seen: HashMap = HashMap::new(); for entry in &mut row.entries { - if entry.head.is_some() { - entry.head = Some(fresh.next()); + let k = seen.entry(entry.selector).or_default(); + let name = *names + .entry((entry.selector, *k)) + .or_insert_with(|| fresh.next()); + *k += 1; + entry.self_tag = Some(name); + } + } +} + +/// A self tag that is not distinct per entry: the entries of a row that +/// share a selector (an array's elements) share one. +fn self_tags_per_selector(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for row in &mut l.rows { + let mut names: HashMap = HashMap::new(); + for entry in &mut row.entries { + entry.self_tag = Some(*names.entry(entry.selector).or_insert_with(|| fresh.next())); + } + } +} + +/// The member rule hidden: every member's token takes the element rule +/// with its ordinal, so the database's test of a hit's self tag never +/// says "member". What the page as #1363 published it states, which says +/// nothing of the rule ([`super::profile::Clauses::hit_rule`]). +fn hit_rule_uniform(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for entry in &mut row.entries { + if entry.rule == Some(Rule::Member) { + entry.rule = Some(Rule::Element); } } } @@ -471,12 +534,9 @@ fn queries_nondeterministic(l: &mut Leakage, _: &mut TestRng) { if seen.contains(query) { match query { QueryLeakage::Extract { selector } => *selector = fresh.next(), - QueryLeakage::Subtree { token, gates, .. } => { - token.head = fresh.next(); - for gate in gates { - match gate { - GateLeak::Selector(s) | GateLeak::Slot(_, s) => *s = fresh.next(), - } + QueryLeakage::Subtree { reads, .. } => { + for read in reads { + read.token.head = fresh.next(); } } QueryLeakage::Contains { needle, .. } | QueryLeakage::Equals { needle } => { @@ -494,90 +554,215 @@ fn queries_nondeterministic(l: &mut Leakage, _: &mut TestRng) { } } -/// No query's token links any entry: parent tags are random everywhere, as -/// if they were not chained from the tokens at all. -fn parent_tags_unchained(l: &mut Leakage, _: &mut TestRng) { - for query in &mut l.queries { - if let QueryLeakage::Subtree { reach, .. } = query { - for row in reach { - row.clear(); +/// A token finds its node in one row only, as a token salted per row +/// would: "Tokens are column-wide". +fn tokens_per_row(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| { + for read in reads { + let mut kept = false; + for (hit, reach) in read.hits.iter_mut().zip(&mut read.reach) { + if hit.is_some() { + if kept { + *hit = None; + reach.clear(); + } + kept = true; + } } } - } + }); } -/// A token links only what the read names: the nodes on the path below the -/// token's node and the subtrees of the nodes the path reaches. What the -/// page would say if the subtree read revealed no more than it asked for. -fn token_reaches_named_nodes_only(l: &mut Leakage, _: &mut TestRng) { - for q in 0..l.queries.len() { - let QueryLeakage::Subtree { reach, named, .. } = &l.queries[q] else { - continue; +/// No token finds or links anything: self and parent tags random +/// everywhere, as if they were not derived from the tokens at all. +fn tokens_find_nothing(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| { + for read in reads { + read.hits.iter_mut().for_each(|h| *h = None); + read.reach.iter_mut().for_each(Vec::clear); + } + }); +} + +/// A token finds its node and links nothing below it: what the page would +/// say if a read revealed no more than the node it names. +fn token_reaches_its_node_only(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| { + for read in reads { + read.reach.iter_mut().for_each(Vec::clear); + } + }); +} + +/// A probe that finds its element links nothing below it: what the page +/// would say without its exception ("In a row where a probe finds an +/// element, the database also walks that element's subtree"). +fn probe_reaches_nothing(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| { + for read in reads.iter_mut().filter(|r| r.probe) { + read.reach.iter_mut().for_each(Vec::clear); + } + }); +} + +/// Each read also sends the token of the head its path starts from (the +/// top-level field under the sealed root, the root with it open), which +/// links that head's whole subtree in every row: the read before #1363, +/// which sent the top-level field's token for any path below it. Its hits +/// are the heads above the read's own hits, so a row where the read finds +/// nothing gets none. +fn ancestor_token_sent(l: &mut Leakage, _: &mut TestRng) { + let rows = l.rows.clone(); + every_read(l, |reads| { + let Some(first) = reads.iter().find(|r| !r.token.steps.is_empty()).cloned() else { + return; + }; + let head = |row: &RowLeakage, mut at: usize| { + let depth = if row.sealed_root { 1 } else { 0 }; + let mut chain = vec![at]; + while let Some(p) = row.entries[at].parent { + chain.push(p); + at = p; + } + chain.iter().rev().nth(depth).copied() }; - let narrowed: Vec> = reach + let hits: Vec> = rows .iter() - .zip(named) .enumerate() - .map(|(r, (reach, named))| { - let row = &l.rows[r]; - reach + .map(|(r, row)| { + reads .iter() - .copied() + .find_map(|read| read.hits[r]) + .and_then(|h| head(row, h)) + }) + .collect(); + let reach = rows + .iter() + .zip(&hits) + .map(|(row, hit)| { + let Some(top) = *hit else { return Vec::new() }; + (0..row.entries.len()) .filter(|&i| { - named.iter().any(|&n| { - // `i` is on the path to `n`, is `n`, or is below it. - let under = |a: usize, b: usize| { - let mut at = Some(a); - while let Some(j) = at { - if j == b { - return true; - } - at = row.entries[j].parent; - } - false - }; - under(n, i) || under(i, n) - }) + let mut at = i; + while row.entries[at].has_parent_tag { + match row.entries[at].parent { + Some(p) if p == top => return true, + Some(p) => at = p, + None => return false, + } + } + false }) .collect() }) .collect(); - if let QueryLeakage::Subtree { reach, .. } = &mut l.queries[q] { - *reach = narrowed; + reads.insert( + 0, + ReadLeak { + token: TokenName { + head: first.token.head, + steps: Vec::new(), + }, + probe: false, + hits, + reach, + }, + ); + }); +} + +/// A read's token is not derived from its path: every token with a step +/// below its head is unrelated to its head's, so no token is derivable +/// from another. +fn tokens_not_chained(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + every_read(l, |reads| { + for read in reads { + if !read.token.steps.is_empty() { + read.token = TokenName { + head: fresh.next(), + steps: Vec::new(), + }; + } } - } + }); } -/// A token for a path through array positions is not chained from its -/// head's: the client would send an unrelated token. -fn position_tokens_unchained(l: &mut Leakage, _: &mut TestRng) { +/// The member rule before #1363, `H16("t" ‖ t(parent) ‖ s ‖ LE32(o))` +/// with `o` the member's keyed rank: the client cannot derive a member's +/// token from the path, so a read through a key step below its head sends a +/// token that names nothing. +fn member_tokens_carry_ordinal(l: &mut Leakage, _: &mut TestRng) { let mut fresh = Fresh::new(); - for query in &mut l.queries { - if let QueryLeakage::Subtree { token, .. } = query { - if !token.positions.is_empty() { - token.head = fresh.next(); - token.positions.clear(); + every_read(l, |reads| { + for read in reads { + if read + .token + .steps + .iter() + .any(|s| matches!(s, TokenStep::Member(_))) + { + read.token = TokenName { + head: fresh.next(), + steps: Vec::new(), + }; + read.hits.iter_mut().for_each(|h| *h = None); + read.reach.iter_mut().for_each(Vec::clear); } } - } + }); } -fn slot_positions_hidden(l: &mut Leakage, _: &mut TestRng) { - for query in &mut l.queries { - if let QueryLeakage::Subtree { gates, .. } = query { - for gate in gates { - if let GateLeak::Slot(n, _) = gate { - *n = 0; +/// An element's token without its index: every element of an array would +/// share one token. +fn element_tokens_without_index(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| { + for read in reads { + for step in &mut read.token.steps { + if let TokenStep::Element(_, i) = step { + *i = 0; } } } - } + }); +} + +/// No probe is sent: a wildcard's tokens stop at `N − 1`. +fn probes_dropped(l: &mut Leakage, _: &mut TestRng) { + every_read(l, |reads| reads.retain(|r| !r.probe)); +} + +/// The tokens are sent in an order unrelated to the paths they name. +fn token_list_shuffled(l: &mut Leakage, rng: &mut TestRng) { + every_read(l, |reads| shuffle(reads, rng)); +} + +/// The token list padded with tokens that name nothing, to the next power +/// of two: a list whose length does not give the bound away. +fn token_list_padded(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + let rows = l.rows.len(); + every_read(l, |reads| { + let target = (reads.len() + 1).next_power_of_two(); + while reads.len() < target { + reads.push(ReadLeak { + token: TokenName { + head: fresh.next(), + steps: Vec::new(), + }, + probe: false, + hits: vec![None; rows], + reach: vec![Vec::new(); rows], + }); + } + }); } -fn gates_dropped(l: &mut Leakage, _: &mut TestRng) { +/// The budget is not sent: every read goes in full. +fn budget_not_sent(l: &mut Leakage, _: &mut TestRng) { for query in &mut l.queries { - if let QueryLeakage::Subtree { gates, .. } = query { - gates.clear(); + if let QueryLeakage::Subtree { budget, .. } = query { + *budget = 0; } } } @@ -638,9 +823,12 @@ mutants!( positions_in_selectors => "EntryLeakage::selector (\"Array elements share their array's selector\")", entry_count_padded => "RowLeakage::entries (\"the number of entries\")", order_random_permutation => "RowLeakage::entries (the stored order)", - members_unsorted => "RowLeakage::entries (\"an object's members in the sorted order of their key names\")", + members_unsorted => "RowLeakage::entries (\"an object's members in the byte order of their key names\")", parent_tag_on_every_entry => "EntryLeakage::has_parent_tag", parent_tag_on_no_entry => "EntryLeakage::has_parent_tag", + self_tags_absent => "EntryLeakage::self_tag (\"the self tag on every entry\")", + self_tags_linkable_across_rows => "EntryLeakage::self_tag (\"salted by the document's iv, so it is unrelated bytes in every row\")", + self_tags_per_selector => "EntryLeakage::self_tag (\"links no entry to another\")", ciphertext_length_bucketed => "EntryLeakage::ciphertext_len", sealed_root_length_hidden => "EntryLeakage::ciphertext_len (\"the root entry's length is the document's length as JSON\")", value_selectors_all_distinct => "GroupLeakage::pairs (value selector equality)", @@ -650,20 +838,27 @@ mutants!( ordering_terms_absent => "EntryLeakage::ordering (presence)", ordering_terms_unordered => "Leakage::ordering_ranks (\"the order of that leaf's value among every row\")", ordering_terms_all_distinct => "OrderingLeak::class (\"which rows hold equal values\")", - ordering_term_length_constant => "OrderingLeak::len (beyond the page)", - ordering_terms_per_path => "OrderingLeak::class across paths (\"leaves at two ordered paths compare with each other too\")", + ordering_term_length_constant => "OrderingLeak::len (\"Its length is the leaf's length\")", + ordering_terms_per_path => "OrderingLeak::class across paths (\"two ordered paths holding equal values carry equal terms\")", member_ranks_follow_key_order => "EntryLeakage::step (a member's ordinal is its keyed rank)", + hit_rule_uniform => "EntryLeakage::rule (the member or element rule a walk's hit matches; added to the page by #1362)", sealed_flag_flipped => "RowLeakage::sealed_root", data_key_shared => "RowLeakage::data_key (\"the data key's identifier (unique to the row)\")", keyset_per_row => "RowLeakage::keyset (\"the keyset the data key was minted under\")", - tokens_per_row => "EntryLeakage::head (\"Tokens are column-wide\")", extraction_selector_fresh => "QueryLeakage::Extract (\"The selector of the path asked\")", queries_nondeterministic => "every query term is a function of the query (\"Tokens are column-wide and do not expire\")", - parent_tags_unchained => "QueryLeakage::Subtree::reach (a token links anything at all)", - token_reaches_named_nodes_only => "QueryLeakage::Subtree::reach (\"its node's subtree in every row\", not the path asked)", - position_tokens_unchained => "TokenName::positions (\"while the path goes on through array positions, the element they reach\")", - slot_positions_hidden => "GateLeak::Slot (\"with the position n in the clear\")", - gates_dropped => "QueryLeakage::Subtree::gates (\"one gate per step below it\")", + tokens_per_row => "ReadLeak::hits in every row (\"find that node in any row\")", + tokens_find_nothing => "ReadLeak::hits (a token finds its node, through its self tag x)", + token_reaches_its_node_only => "ReadLeak::reach (\"the structure of its node's subtree in every row\")", + probe_reaches_nothing => "ReadLeak::reach of a probe (\"the database also walks that element's subtree\")", + ancestor_token_sent => "ReadLeak::token (\"nothing that covers $.profile\"): the pre-#1363 top-level token", + tokens_not_chained => "TokenName::steps (\"derive the tokens of the nodes beneath\")", + member_tokens_carry_ordinal => "TokenStep::Member (a member's token has no ordinal): the pre-#1363 member rule", + element_tokens_without_index => "TokenStep::Element (an element's token carries its index)", + probes_dropped => "ReadLeak::probe (\"a probe token for the element at index N of every array a wildcard expands\")", + token_list_shuffled => "QueryLeakage::Subtree::reads (the tokens' order, which places each)", + token_list_padded => "QueryLeakage::Subtree::reads (the tokens' number, which gives N away)", + budget_not_sent => "QueryLeakage::Subtree::budget (the budget, in the clear)", needle_flat => "QueryLeakage::Contains::needle (\"its element grouping\")", needle_alt_dropped => "QueryLeakage::Contains::alt", bound_rank_random => "QueryLeakage::Order::bound (\"the bound's place in the order of the stored values\")", diff --git a/packages/stack-encrypt/tests/json_leakage/observe.rs b/packages/stack-encrypt/tests/json_leakage/observe.rs index 4a405badd..0daf114f4 100644 --- a/packages/stack-encrypt/tests/json_leakage/observe.rs +++ b/packages/stack-encrypt/tests/json_leakage/observe.rs @@ -34,10 +34,15 @@ pub struct ObservedHeader { pub struct ObservedEntry { pub s: Tag16, pub c: Vec, + /// The self tag. Every real entry carries one; `None` only from a + /// mutant, and then [`ObservedEntry::fields`] shows it. + pub x: Option, pub p: Option, pub op: Option>, /// The entry's wire keys. pub fields: Vec, + /// The entry's length in characters, as compact JSON. + pub chars: usize, } /// One group of a projection: a stored `q`, or a needle. @@ -52,6 +57,8 @@ pub struct ObservedGroup { pub struct ObservedRow { /// The document's wire keys. pub fields: Vec, + /// The document's length in characters, as compact JSON. + pub chars: usize, pub header: ObservedHeader, pub entries: Vec, pub q: ObservedGroup, @@ -63,7 +70,8 @@ pub struct ObservedRow { pub enum QueryKind { /// `->`, `json_tree_path_query`: a selector. Extract, - /// `json_tree_subtree`, `json_tree_subtree_at`: a token and gates. + /// `json_tree_subtree(doc, tokens, budget)`: a list of tokens and the + /// budget. Subtree, /// `@>`: a containment needle, with its `alt`. Contains, @@ -73,13 +81,6 @@ pub enum QueryKind { Order, } -/// One gate of a subtree read. -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum ObservedGate { - Selector(Tag16), - Slot(u32, Tag16), -} - /// One query term, as the database receives it. #[derive(Clone, Debug, PartialEq, Eq)] pub enum ObservedQuery { @@ -87,8 +88,8 @@ pub enum ObservedQuery { s: Tag16, }, Subtree { - t: Tag16, - gates: Vec, + tokens: Vec, + budget: u32, }, Contains { needle: ObservedGroup, @@ -167,9 +168,11 @@ fn entry(value: &Value) -> ObservedEntry { s: b64_16(&value["s"]), c: base85_simd::decode(value["c"].as_str().expect("`c` is a string")) .expect("`c` is base85"), + x: value.get("x").map(b64_8), p: value.get("p").map(b64_8), op: value.get("op").map(b64), fields: keys(value), + chars: chars(value), } } @@ -191,9 +194,16 @@ pub fn group(value: &Value) -> ObservedGroup { } } +fn chars(value: &Value) -> usize { + serde_json::to_string(value) + .expect("a wire value serializes") + .len() +} + fn row(value: &Value) -> ObservedRow { ObservedRow { fields: keys(value), + chars: chars(value), header: header(&b64(&value["h"])), entries: value["sv"] .as_array() @@ -205,35 +215,20 @@ fn row(value: &Value) -> ObservedRow { } } -fn gate(text: &Value) -> ObservedGate { - let text = text.as_str().expect("a gate is a string"); - match text.strip_prefix('#') { - Some(slot) => { - let (n, selector) = slot.split_once(':').expect("`#n:`"); - ObservedGate::Slot( - n.parse().expect("a position"), - b64_16(&Value::String(selector.to_owned())), - ) - } - None => ObservedGate::Selector(b64_16(&Value::String(text.to_owned()))), - } -} - fn query(kind: QueryKind, value: &Value) -> ObservedQuery { match kind { QueryKind::Extract => ObservedQuery::Extract { s: b64_16(value) }, QueryKind::Subtree => ObservedQuery::Subtree { - t: b64_16(&value["t"]), - gates: value - .get("g") - .map(|g| { - g.as_array() - .expect("`g` is a list") - .iter() - .map(gate) - .collect() - }) - .unwrap_or_default(), + tokens: value["t"] + .as_array() + .expect("`t` is a list of tokens") + .iter() + .map(b64_16) + .collect(), + budget: value["budget"] + .as_u64() + .and_then(|b| u32::try_from(b).ok()) + .expect("the budget is an integer"), }, QueryKind::Contains => ObservedQuery::Contains { needle: group(value), diff --git a/packages/stack-encrypt/tests/json_leakage/profile.rs b/packages/stack-encrypt/tests/json_leakage/profile.rs index 84e4e7940..94b525a49 100644 --- a/packages/stack-encrypt/tests/json_leakage/profile.rs +++ b/packages/stack-encrypt/tests/json_leakage/profile.rs @@ -5,33 +5,38 @@ //! query reveals on top of them. Neither takes a key or calls the crate: they //! read the documents, the column's options and the query, and they name //! every keyed value (a selector, a value selector, a token, an ordering -//! term) by an opaque integer that is equal wherever the real values are -//! equal, so the profile states equality patterns and never names. The -//! simulator ([`super::simulator`]) builds a wire-shaped world from this -//! output and its own randomness, and the distinguishers +//! term, a self tag) by an opaque integer that is equal wherever the real +//! values are equal, so the profile states equality patterns and never +//! names. The simulator ([`super::simulator`]) builds a wire-shaped world +//! from this output and its own randomness, and the distinguishers //! ([`super::distinguishers`]) compare it with the real one. //! //! Each field of the leakage types is one clause, and its doc comment quotes //! the sentence of the page it encodes. Where the page is vague the clause is //! as precise as the code forces, and says so. One clause is marked -//! **beyond the page**: the code reveals it and the page does not say so. -//! [`Clauses::page_only`] turns it off, and the calibration shows a -//! distinguisher separating that profile from the real scheme. +//! **added to the page by #1362**: the code revealed it and the page as +//! #1363 published it did not say so. [`Clauses::as_published_in_1363`] +//! turns it off, and the calibration shows a distinguisher separating that +//! profile from the real scheme. //! -//! **This module is expected to change with the format.** The design change -//! of 2026-10-24 (member tokens without ordinals, a self tag `x` on every -//! entry, wildcards expanded on the client, no gates; -//! `docs/plans/2026-10-17-json-tree-index.md`, "Design change (2026-10-24)") -//! rewrites the subtree clauses of [`QueryLeakage`] and adds a clause for -//! `x`. When it lands, this file and the real-side adapter -//! ([`super::real`]) change together with the leakage page; the simulator -//! changes where the wire does; the distinguishers should not need to. +//! **This module models the format of #1363** (`docs/plans/2026-10-17-json-tree-index.md`, +//! "Design change (2026-10-24)" and its decisions): a self tag `x` on every +//! entry, member tokens without an ordinal, element tokens with their +//! index, and a subtree read that sends the tokens of exactly the nodes it +//! names, with wildcards expanded on the client and a probe per expanded +//! array. The profile was first written against the format before it +//! (`{s, c, p?, op?}` entries, one head token and plaintext gates per read); +//! `docs/plans/2026-10-25-json-tree-leakage-before-and-after.md` records +//! what changed between the two. use std::collections::HashMap; use std::hash::Hash; use serde_json::{Number, Value}; use stack_encrypt::json::{JsonPath, Segment}; +use vitaminc_protected::Controlled; + +pub use super::chain::Rule; /// What every entry's ciphertext adds to its plaintext: the frame's version /// byte, the 96-bit random nonce and the 128-bit GCM tag @@ -44,6 +49,13 @@ pub const AEAD_OVERHEAD: usize = 1 + 12 + 16; /// service, not of any document, so it is a column constant here. pub const KEY_TAG_LEN: usize = 32; +/// The bound a wildcard expands to when a read names none +/// (`JsonSubtreeKey::DEFAULT_BOUND`). +pub const DEFAULT_BOUND: u32 = 64; + +/// The budget `json_tree_subtree` takes when the caller names none. +pub const DEFAULT_BUDGET: u32 = 64; + /// A column's configuration, which the database knows: it is in every /// header (the root option) or in the plan the queries run under. #[derive(Clone, Debug)] @@ -57,8 +69,13 @@ pub struct Column { pub enum Query { /// `->` on a path. Extract(String), - /// A subtree read of a path (`json_tree_subtree`, `json_tree_subtree_at`). - Subtree(String), + /// A subtree read of a path, `json_tree_subtree(doc, tokens, budget)`, + /// its wildcards expanded to `bound` positions each. + Subtree { + path: String, + bound: u32, + budget: u32, + }, /// Containment, `@>`, of a needle document. Contains(Value), /// Equality at a path, `AtPath`, of a scalar. @@ -67,53 +84,90 @@ pub enum Query { Order(String, Value), } -/// Which sort "the sorted order of their key names" means. The page does -/// not say; the code sorts by the keys' UTF-8 bytes. +impl Query { + /// A subtree read at the default bound and budget. + pub fn subtree(path: &str) -> Self { + Self::Subtree { + path: path.to_owned(), + bound: DEFAULT_BOUND, + budget: DEFAULT_BUDGET, + } + } + + /// A subtree read at `bound` and `budget`. + pub fn subtree_with(path: &str, bound: u32, budget: u32) -> Self { + Self::Subtree { + path: path.to_owned(), + bound, + budget, + } + } +} + +/// Which sort "the byte order of their key names' UTF-8 encodings" means. +/// The page now says it; the other reading is kept to show the sentence is +/// load-bearing. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum KeyOrder { /// Byte order of the UTF-8 key: serde_json's `Map` without /// `preserve_order`, which is what the walk iterates. Utf8Bytes, /// PostgreSQL `jsonb`'s own order: shorter keys first, then bytes. The - /// other reading a reader of the page could reasonably take. + /// other reading a reader could take of an unqualified "sorted order". JsonbLengthFirst, } -/// The clauses of the profile that are on or off as a whole, and the one -/// reading the page leaves open. The default is what the code needs; -/// [`Clauses::page_only`] is the page as written. +/// What a string's ordering term orders and compares by. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StringOrder { + /// The page: "Strings are ordered and compared after their case and + /// accents are folded", and the term's length counts "a string whose + /// folded UTF-8 form is `n` bytes". Unicode 16 NFC with case and accents + /// folded (`vitaminc_ore::text`). + Folded, + /// The string's own bytes, as the profile first assumed (its corpus had + /// only lower-case ASCII, where the two agree). + Bytes, +} + +/// The clauses of the profile that are on or off as a whole, and the two +/// readings the page could leave open. The default is what the code needs. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Clauses { - /// How "an object's members in the sorted order of their key names" + /// How "an object's members in the byte order of their key names" /// sorts. pub key_order: KeyOrder, - /// **Beyond the page.** An ordering term's length: `8 × n + 1` bytes for - /// a value encoded in `n` bytes (`clww-ore`'s `OpeClww8VariableV1`). A - /// number encodes in eight bytes, so its term is 65 bytes; a string in - /// its UTF-8 bytes and one more, so its term is `8 × (len + 1) + 1` - /// bytes and gives its length. The page says only that "a term's - /// presence at an ordered path also says the leaf is a number or a - /// string"; the length also says which, unless the string is seven bytes - /// long, and repeats the string's length that `c` already shows. Off, - /// every term is stated as one length. - pub ordering_term_length: bool, + /// How a string's ordering term orders. + pub string_order: StringOrder, + /// **Added to the page by #1362.** The rule a node's token takes under + /// its parent's, member or element, which the database learns for every + /// node a walk reaches: it tests the member rule against the hit's self + /// tag and then the element rule (`json_tree_hit_token`), so the one + /// that matches says whether the parent is an object or an array. The + /// design's decision 2 ("no kind bit") keeps it out of the stored form; + /// the walk reveals it anyway, below any token sent. Off, the profile + /// states no rule, and every node takes the element rule with its + /// ordinal, which is the coarsest world consistent with the rest. + pub hit_rule: bool, } impl Default for Clauses { fn default() -> Self { Self { key_order: KeyOrder::Utf8Bytes, - ordering_term_length: true, + string_order: StringOrder::Folded, + hit_rule: true, } } } impl Clauses { - /// The page as written: every clause beyond it off. - pub fn page_only() -> Self { + /// The page as #1363 published it (at `997c64b1e`): every clause #1362 + /// added to it off. + pub fn as_published_in_1363() -> Self { Self { - key_order: KeyOrder::Utf8Bytes, - ordering_term_length: false, + hit_rule: false, + ..Self::default() } } } @@ -123,13 +177,17 @@ impl Clauses { pub type SelId = usize; /// An opaque name for a value selector `vs`. pub type ValueId = usize; -/// An opaque name for a chain head's token. +/// An opaque name for a token that is a PRF of a path: the root's, and +/// under the sealed root each top-level field's or element's. pub type HeadId = usize; /// An opaque name for an ordering term's value: equal for equal values. pub type OrderClass = usize; +/// An opaque name for a self tag the database cannot recompute: one per +/// entry, so equal nowhere. +pub type SelfTagId = usize; /// How a node hangs off its parent, which decides its ordinal under the -/// parent's token. +/// parent's token: the `o` of its parent tag. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum Step { Root, @@ -146,8 +204,11 @@ pub struct OrderingLeak { /// which rows hold equal values." Equal values share a class; the /// classes' order is [`Leakage::ordering_ranks`]. pub class: OrderClass, - /// **Beyond the page** ([`Clauses::ordering_term_length`]): the term's - /// length in bytes. + /// "Its length is the leaf's length: one byte per bit of the value's + /// order encoding plus one, so 65 bytes for every number and + /// `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` + /// bytes." (Beyond the page when the profile was first written; on it + /// since #1363.) pub len: usize, } @@ -168,27 +229,41 @@ pub struct EntryLeakage { /// below a token a query sends; at rest the order clause /// ([`RowLeakage::entries`]) is what shows it. pub parent: Option, - /// How the node hangs off its parent, which a token's holder sees in the - /// ordinals it probes: an element's ordinal is its position. The page - /// says a token's holder can "see how the nodes nest, link each one to - /// its parent"; it does not say that it also sees each element's - /// position and each member's keyed rank. Vague; stated here as the code - /// forces it. + /// The node's ordinal under its parent, which a token's holder sees in + /// the ordinals it probes: "each element's index: […] an element's index + /// is the ordinal it is found at". A member's ordinal is its keyed rank, + /// which the page does not mention; under one token it is a fixed + /// permutation of the members' key order that the holder sees and + /// cannot read (`member_ordinals_follow_key_order`). pub step: Step, - /// The chain head this entry is, if it is one: under the sealed root a - /// top-level field ("the top-level field the path starts with"), with the - /// root open the root. A head's token is a keyed function of its path, - /// "column-wide", so every row's entry at that path is one head. - pub head: Option, + /// The rule this node's token takes under its parent's: "whether each + /// node beneath is an object member or an array element […] the + /// database tests its self tag against the member rule and then the + /// element rule, and the one that matches says which" + /// ([`Clauses::hit_rule`]). `None` for an entry that chains from no + /// parent (the root, and the top-level entries of a sealed root). + pub rule: Option, /// "Which entries carry no `p` does show: with the sealed root on, those /// are the root's direct members […] With the sealed root off, only the /// root lacks one." (The root lacks one under the sealed root too.) pub has_parent_tag: bool, + /// "`x`, the self tag on every entry: Nothing on its own. It is a hash + /// of the entry's own token salted by the document's `iv`, so it is + /// unrelated bytes in every row and links no entry to another, and since + /// every entry carries one, its presence marks none." Named per entry, + /// so no two are equal; an entry a query's token finds or reaches has + /// its real self tag instead, in [`QueryLeakage::Subtree`]. `None` would + /// be an entry without one, which only a mutant states. + pub self_tag: Option, /// "Each ciphertext's length […] for each node the length of its key name /// (for an object member) plus, for a scalar, the length of its value, /// exactly, less the AEAD's fixed overhead. […] With the sealed root on, /// the root entry's length is the document's length as JSON." The fixed - /// parts the page leaves out are written in [`ciphertext_len`]. + /// parts the page leaves out are written in [`ciphertext_len`]. The self + /// tag sits beside the ciphertext, not in it, so it changes no + /// ciphertext's length; it adds a fixed 12 base64 characters per entry, + /// 19 with its key in compact JSON, which + /// `distinguishers::stored_lengths` counts. pub ciphertext_len: usize, /// "`op`, the ordering term, only on number and string leaves at the /// paths the index orders." @@ -220,37 +295,59 @@ pub struct RowLeakage { pub keyset: usize, /// The header's key identifiers (the `iv` and the ZeroKMS tag): "the /// data key's identifier (unique to the row)". A different name in every - /// row. + /// row. The `iv` also salts every parent and self tag in the row. pub data_key: usize, /// "The number of entries", and the entries in their stored order: /// "Entries are stored in depth-first order, the root first, each node /// followed by everything beneath it, and an object's members in the - /// sorted order of their key names." The page does not say which sort; - /// the code's is the keys' UTF-8 bytes (serde_json's `Map` without - /// `preserve_order`), so that is the order here. + /// byte order of their key names' UTF-8 encodings." pub entries: Vec, /// The containment projection `q`. pub projection: GroupLeakage, } -/// A gate of a subtree read. +/// One step of a token's derivation below its head, as the client derives +/// it from the path alone. #[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum GateLeak { - /// "A gate is the step's selector". - Selector(SelId), - /// "or, for an array position, `#n:`, with the position `n` in - /// the clear." - Slot(u64, SelId), +pub enum TokenStep { + /// `H16("t" ‖ t(parent) ‖ s(member))`: a key step, by its selector. + Member(SelId), + /// `H16("t" ‖ t(array) ‖ s(element) ‖ LE32(i))`: a position step, by + /// the element selector and the position. + Element(SelId, u64), } -/// The token a subtree read sends, by name: a head, then the array positions -/// the client chains below it. +/// A token, by name: a head, then the steps the client chains below it. +/// Two reads of one concrete path send one token, in every query: "Tokens +/// are column-wide and do not expire". #[derive(Clone, Debug, PartialEq, Eq)] pub struct TokenName { pub head: HeadId, - /// Each position step after the head: the element selector and the - /// position. - pub positions: Vec<(SelId, u64)>, + pub steps: Vec, +} + +/// One token of a subtree read. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ReadLeak { + /// "A token for exactly the node the path names, which the client + /// derives from the path alone." For a wildcard, one per concrete path + /// below the bound, in document order. + pub token: TokenName, + /// "plus a probe token for the element at index `N` of every array a + /// wildcard expands", the last token at its level. The database is not + /// told which tokens are probes; it can tell from their places in the + /// list (`distinguishers::wildcard_positions`). + pub probe: bool, + /// Per row, the entry the token names, if the row has the node: "whoever + /// holds it can find that node in any row (through its self tag `x`)". + pub hits: Vec>, + /// Per row, the entries below that node the token links: "see how the + /// nodes beneath it nest, link each one to its parent, and derive the + /// tokens of the nodes beneath". Every strict descendant that chains + /// from it, which under the sealed root excludes everything below the + /// root ("the top-level fields chain from nothing"). Nothing above it: + /// "nothing that covers `$.profile`". + pub reach: Vec>, } /// What one query reveals, beyond the rows it returns. @@ -259,27 +356,18 @@ pub enum QueryLeakage { /// "Path extraction […] The selector of the path asked. A position in the /// path reads as the wildcard, so a position is not sent." Extract { selector: SelId }, - /// "A subtree read […] A token, and one gate per step below it. The token - /// is not the token of the node you asked for: it is the token of the - /// deepest node on the path the client can derive, which is the - /// top-level field the path starts with or, while the path goes on - /// through array positions, the element they reach." + /// "A subtree read […] A token for exactly the node the path names". The + /// tokens' number gives the wildcards' bound away (`N + 1` per one-level + /// wildcard, `N² + N + 1` for two levels), and which of them find a node + /// in which row gives each expanded array's length below `N`, or that it + /// is longer. Subtree { - token: TokenName, - gates: Vec, - /// Per row, the entries the token lets the database link: "A token - /// is a capability over the structure of its node's subtree in every - /// row of the column: whoever holds it can find each node under that - /// field in any row, see how the nodes nest, link each one to its - /// parent, and derive the tokens of the nodes beneath." So: every - /// strict descendant of the token's node, in every row. - reach: Vec>, - /// Per row, the nodes the path names (what the walk returns at depth - /// `0`). Not a clause of its own: the database computes it from the - /// token, the gates and the rows. Carried so a mutant profile can - /// narrow `reach` to what a read would reveal if the token covered - /// only the path asked (`token_reaches_named_nodes_only`). - named: Vec>, + reads: Vec, + /// The `budget` argument, in the clear in the call. Past it, with + /// the root sealed, the database returns the root entry alone; it + /// computes that from the walk it already sees, so the fallback + /// tells it nothing more. + budget: u32, }, /// "Containment […] The needle's own projection: the selectors of its /// paths, the value selectors of its nodes, and its element grouping." @@ -315,7 +403,8 @@ pub struct Leakage { /// The rank of each [`OrderClass`] among all of them: "the order of that /// leaf's value among every row that holds one", and the bound's "place /// in the order of the stored values at every ordered path" ("leaves at - /// two ordered paths compare with each other too"). + /// two ordered paths compare with each other too, and two ordered paths + /// holding equal values carry equal terms"). pub ordering_ranks: Vec, } @@ -343,7 +432,7 @@ impl Interner { } /// A value an ordering term orders by. Numbers order as `f64` (the term is -/// derived from `as_f64`); strings by their bytes. +/// derived from `as_f64`); strings by their order encoding's bytes. #[derive(Clone, Debug, PartialEq, Eq, Hash)] enum OrderKey { Number(u64), @@ -360,6 +449,7 @@ pub struct Ids { heads: Interner>, ordering: Interner, ordering_keys: Vec, + self_tags: usize, pub clauses: Clauses, } @@ -391,34 +481,44 @@ impl Ids { self.heads.id(positional.to_vec()) } + fn self_tag(&mut self) -> SelfTagId { + self.self_tags += 1; + self.self_tags + } + fn ordering(&mut self, value: &Value) -> OrderingLeak { - let (key, len) = match value { + let key = match value { Value::Number(n) => { let f = n.as_f64().expect("a JSON number has an f64"); assert!( f != 0.0 || f.is_sign_positive(), "the corpus has no -0.0, whose term the page does not describe" ); - (OrderKey::Number(f.to_bits()), 8 * 8 + 1) + OrderKey::Number(f.to_bits()) } - Value::String(s) => ( - OrderKey::String(s.as_bytes().to_vec()), - 8 * (s.len() + 1) + 1, - ), + Value::String(s) => OrderKey::String(match self.clauses.string_order { + // The marker, then the folded text: the order encoding. + StringOrder::Folded => vitaminc_ore::text(s) + .expect("the corpus holds assigned code points") + .risky_unwrap(), + StringOrder::Bytes => { + let mut out = vec![vitaminc_ore::VARIABLE_WIDTH_MARKER]; + out.extend_from_slice(s.as_bytes()); + out + } + }), _ => unreachable!("only numbers and strings carry a term"), }; + // One byte per bit of the order encoding, plus one. + let len = match &key { + OrderKey::Number(_) => 8 * 8 + 1, + OrderKey::String(encoded) => 8 * encoded.len() + 1, + }; let class = self.ordering.id(key.clone()); if class == self.ordering_keys.len() { self.ordering_keys.push(key); } - OrderingLeak { - class, - len: if self.clauses.ordering_term_length { - len - } else { - 65 - }, - } + OrderingLeak { class, len } } /// The rank of every ordering class. The page does not say how a @@ -678,6 +778,13 @@ fn projection(ids: &mut Ids, doc: &Value) -> GroupLeakage { group(ids, &nodes, &selectors, None) } +/// Whether the node at `depth` chains from its parent, which is whether it +/// carries a parent tag: under the sealed root the top-level entries chain +/// from nothing. +fn chains(depth: usize, sealed_root: bool) -> bool { + depth > 0 && !(sealed_root && depth == 1) +} + // ============================================================================= // L // ============================================================================= @@ -697,20 +804,21 @@ pub fn l_store(column: &Column, ids: &mut Ids, docs: &[Value]) -> Vec Rule::Member, + _ => Rule::Element, + }); EntryLeakage { selector: ids.selector(&node.positional), parent: node.parent, step: node.step, - head: is_head.then(|| ids.head(&node.positional)), - has_parent_tag: depth > 0 && !(column.sealed_root && depth == 1), + rule, + has_parent_tag, + self_tag: Some(ids.self_tag()), ciphertext_len: ciphertext_len(node, column.sealed_root, doc), ordering, } @@ -727,23 +835,14 @@ pub fn l_store(column: &Column, ids: &mut Ids, docs: &[Value]) -> Vec], pattern: &[Segment]) -> Vec { - (0..nodes.len()) - .filter(|&i| covers(pattern, &nodes[i].positional)) - .collect() -} - -/// Every strict descendant of node `at`: in the stored order these are the -/// positions after it up to the end of its subtree. -fn descendants(nodes: &[Node<'_>], at: usize) -> Vec { +/// Every strict descendant of node `at` that chains from it: in the stored +/// order these are the positions after it up to the end of its subtree. +fn descendants(nodes: &[Node<'_>], at: usize, sealed_root: bool) -> Vec { let mut inside = vec![false; nodes.len()]; inside[at] = true; let mut out = Vec::new(); for (i, node) in nodes.iter().enumerate().skip(at + 1) { - if node.parent.is_some_and(|p| inside[p]) { + if node.parent.is_some_and(|p| inside[p]) && chains(node.positional.len(), sealed_root) { inside[i] = true; out.push(i); } @@ -751,61 +850,161 @@ fn descendants(nodes: &[Node<'_>], at: usize) -> Vec { out } +/// One concrete path a read sends a token for. +struct Concrete { + path: Vec, + token: TokenName, + probe: bool, +} + +/// The concrete paths a read of `segs` sends tokens for, in the order the +/// client sends them (`subtree_key_for` and `Expansion::descend` in +/// `derive.rs`): each wildcard step expanded to the positions `0 … N − 1`, +/// each in turn, then the probe at `N`, the element itself. +fn expand(ids: &mut Ids, sealed_root: bool, segs: &[Segment], bound: u32) -> Vec { + fn descend( + ids: &mut Ids, + segs: &[Segment], + bound: u32, + at: Vec, + token: TokenName, + k: usize, + out: &mut Vec, + ) { + let Some(segment) = segs.get(k) else { + out.push(Concrete { + path: at, + token, + probe: false, + }); + return; + }; + let step = |i: u64, at: &[Segment], token: &TokenName, ids: &mut Ids| { + let mut path = at.to_vec(); + path.push(Segment::Index(i)); + let s = ids.selector(&path); + let mut token = token.clone(); + token.steps.push(TokenStep::Element(s, i)); + (path, token) + }; + match segment { + Segment::Key(key) => { + let mut path = at; + path.push(Segment::Key(key.clone())); + let s = ids.selector(&path); + let mut token = token; + token.steps.push(TokenStep::Member(s)); + descend(ids, segs, bound, path, token, k + 1, out); + } + Segment::Index(i) => { + let (path, token) = step(*i, &at, &token, ids); + descend(ids, segs, bound, path, token, k + 1, out); + } + Segment::Wildcard => { + for i in 0..u64::from(bound) { + let (path, token) = step(i, &at, &token, ids); + descend(ids, segs, bound, path, token, k + 1, out); + } + let (path, token) = step(u64::from(bound), &at, &token, ids); + out.push(Concrete { + path, + token, + probe: true, + }); + } + } + } + + let mut out = Vec::new(); + match segs.first() { + // Under the sealed root the chain starts at the heads, each a PRF + // of its path; a wildcard head is each position's head in turn. + Some(first) if sealed_root => { + let heads: Vec<(Segment, bool)> = match first { + Segment::Wildcard => (0..u64::from(bound)) + .map(|i| (Segment::Index(i), false)) + .chain(std::iter::once((Segment::Index(u64::from(bound)), true))) + .collect(), + step => vec![(step.clone(), false)], + }; + for (step, probe) in heads { + let at = vec![step]; + let token = TokenName { + head: ids.head(&at), + steps: Vec::new(), + }; + if probe { + out.push(Concrete { + path: at, + token, + probe, + }); + } else { + descend(ids, segs, bound, at, token, 1, &mut out); + } + } + } + // `$` itself, under either root, or any path from an open root. + _ => { + let token = TokenName { + head: ids.head(&[]), + steps: Vec::new(), + }; + descend(ids, segs, bound, Vec::new(), token, 0, &mut out); + } + } + out +} + /// What one query reveals, over the stored `docs`. pub fn l_query(column: &Column, ids: &mut Ids, docs: &[Value], query: &Query) -> QueryLeakage { match query { Query::Extract(path) => QueryLeakage::Extract { selector: ids.selector(&segments(path)), }, - Query::Subtree(path) => { - let segs = segments(path); - // The head: under the sealed root the top-level field (or - // element) the path starts with, with the root open the root. - let split = usize::from(column.sealed_root); - assert!( - segs.len() >= split && !segs[..split].contains(&Segment::Wildcard), - "under the sealed root a subtree read names a top-level node: {path}" - ); - let (head_path, rest) = segs.split_at(split); - let head = ids.head(head_path); - let mut prefix = head_path.to_vec(); - let mut positions = Vec::new(); - let mut gates = Vec::new(); - let mut chaining = true; - for segment in rest { - prefix.push(segment.clone()); - match (segment, chaining) { - (Segment::Index(i), true) => positions.push((ids.selector(&prefix), *i)), - (Segment::Index(i), false) => { - gates.push(GateLeak::Slot(*i, ids.selector(&prefix))) - } - (Segment::Key(_) | Segment::Wildcard, _) => { - chaining = false; - gates.push(GateLeak::Selector(ids.selector(&prefix))); - } - } - } - let token_path = &segs[..split + positions.len()]; + Query::Subtree { + path, + bound, + budget, + } => { let order = ids.clauses.key_order; - let (reach, named) = docs + let flattened: Vec>> = docs.iter().map(|d| flatten(d, order)).collect(); + let positions: Vec> = flattened .iter() - .map(|doc| { - let nodes = flatten(doc, order); - let at = nodes.iter().position(|n| n.positional == token_path); - let reach = at.map(|at| descendants(&nodes, at)).unwrap_or_default(); - let named = if gates.is_empty() { - at.into_iter().collect() - } else { - reached(&nodes, &segs) - }; - (reach, named) + .map(|nodes| { + nodes + .iter() + .enumerate() + .map(|(i, n)| (n.positional.as_slice(), i)) + .collect() }) - .unzip(); + .collect(); + let reads = expand(ids, column.sealed_root, &segments(path), *bound) + .into_iter() + .map(|concrete| { + let hits: Vec> = positions + .iter() + .map(|at| at.get(concrete.path.as_slice()).copied()) + .collect(); + let reach = hits + .iter() + .zip(&flattened) + .map(|(hit, nodes)| { + hit.map(|at| descendants(nodes, at, column.sealed_root)) + .unwrap_or_default() + }) + .collect(); + ReadLeak { + token: concrete.token, + probe: concrete.probe, + hits, + reach, + } + }) + .collect(); QueryLeakage::Subtree { - token: TokenName { head, positions }, - gates, - reach, - named, + reads, + budget: *budget, } } Query::Contains(needle) => { diff --git a/packages/stack-encrypt/tests/json_leakage/real.rs b/packages/stack-encrypt/tests/json_leakage/real.rs index 0f4816419..691f02842 100644 --- a/packages/stack-encrypt/tests/json_leakage/real.rs +++ b/packages/stack-encrypt/tests/json_leakage/real.rs @@ -63,16 +63,26 @@ pub async fn world( .expect("derives"); (QueryKind::Extract, serde_json::to_value(selector).unwrap()) } - Query::Subtree(path) => { + Query::Subtree { + path, + bound, + budget, + } => { let key = keyset - .json_subtree_key( + .json_subtree_key_with_bound( &JsonPath::parse(path).expect("a path"), &options(column), + *bound, nonempty!("users").with("profile"), ) .await .expect("derives"); - (QueryKind::Subtree, serde_json::to_value(&key).unwrap()) + // What `json_tree_subtree(doc, key.tokens(), budget)` hands + // the database: the key's wire form, `{"t": [...]}`, and the + // budget argument beside it. + let mut wire = serde_json::to_value(&key).unwrap(); + wire["budget"] = serde_json::json!(budget); + (QueryKind::Subtree, wire) } Query::Contains(needle) => { let term = cipher diff --git a/packages/stack-encrypt/tests/json_leakage/simulator.rs b/packages/stack-encrypt/tests/json_leakage/simulator.rs index 79d37e439..e28e8051c 100644 --- a/packages/stack-encrypt/tests/json_leakage/simulator.rs +++ b/packages/stack-encrypt/tests/json_leakage/simulator.rs @@ -2,33 +2,41 @@ //! randomness, with no plaintext and no key. //! //! Every keyed value is a fresh random string per opaque name (a selector -//! per [`SelId`], a value selector per [`ValueId`], a head token per -//! [`HeadId`]), so equality patterns are exactly the profile's. Ciphertexts -//! are random bytes of the stated length behind the frame's public version -//! byte. Ordering terms are random bytes laid out so that byte order is the -//! profile's rank order. +//! per [`SelId`](super::profile::SelId), a value selector per +//! [`ValueId`](super::profile::ValueId), a head token per +//! [`HeadId`](super::profile::HeadId), a self tag per +//! [`SelfTagId`](super::profile::SelfTagId)), so equality patterns are +//! exactly the profile's. Ciphertexts are random bytes of the stated length +//! behind the frame's public version byte. Ordering terms are random bytes +//! laid out so that byte order is the profile's rank order. //! //! The chain is the one place the simulator evaluates the real scheme's //! public functions ([`super::chain`]): a token the database is sent has to -//! derive the parent tags of exactly the entries the profile says it reaches, -//! with the real hash, or the database would notice. So the simulator gives -//! every node a token (heads random, every other node chained below its -//! parent with an ordinal: an element's position, or a member's rank under a -//! keyed function the simulator samples itself), and writes the chained -//! parent tag on every entry some query's [`QueryLeakage::Subtree::reach`] -//! names. Every other parent tag is random bytes, as a tag under a token the -//! database never sees is to it. +//! find the node the profile says it names, by that node's self tag, and +//! link the entries the profile says it reaches, by their parent tags and +//! self tags, with the real hash, or the database would notice. So the +//! simulator derives each token a query sends from its random head and the +//! steps the profile names (the member rule or the element rule, as the +//! client does), writes the self tag of that token on each entry the token +//! names, and below it chains every entry the token reaches: its parent tag +//! from its parent's token and its ordinal (an element's position, or a +//! member's rank under a keyed function the simulator samples itself), its +//! self tag from its own token, by the rule the profile states for it. +//! Every other parent and self tag is random bytes, as a tag under a token +//! the database never sees is to it. -use std::collections::{HashMap, HashSet}; +use std::collections::HashMap; use base64ct::{Base64, Encoding}; use proptest::prelude::RngCore; use proptest::test_runner::TestRng; use serde_json::{json, Value}; -use super::chain::{child_token, parent_tag, simulated_rank, Tag16}; +use super::chain::{ + element_token, member_token, parent_tag, self_tag, simulated_rank, Rule, Tag16, Tag8, +}; use super::observe::QueryKind; -use super::profile::{GateLeak, GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step}; +use super::profile::{GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step, TokenName, TokenStep}; /// The version byte `vitaminc-encrypt` writes at the front of every /// ciphertext: public framing, not leakage. @@ -67,6 +75,13 @@ fn b64(bytes: &[u8]) -> String { Base64::encode_string(bytes) } +/// The tags a row's entries carry where some query's token links them. +#[derive(Default)] +struct Linked { + x: HashMap, + p: HashMap, +} + struct Simulator<'a> { leakage: &'a Leakage, rng: &'a mut TestRng, @@ -75,6 +90,7 @@ struct Simulator<'a> { heads: Names<16>, keysets: Names<16>, ivs: Names<16>, + self_tags: Names<8>, tags: HashMap>, ordering: HashMap>, rank_key: [u8; 32], @@ -92,15 +108,17 @@ pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(Query heads: Names::new(), keysets: Names::new(), ivs: Names::new(), + self_tags: Names::new(), tags: HashMap::new(), ordering: HashMap::new(), rank_key, }; + let linked = sim.link(); let rows = leakage .rows .iter() - .enumerate() - .map(|(r, row)| sim.row(r, row)) + .zip(linked) + .map(|(row, linked)| sim.row(row, &linked)) .collect(); let queries = leakage.queries.iter().map(|q| sim.query(q)).collect(); (rows, queries) @@ -111,6 +129,27 @@ impl Simulator<'_> { self.selectors.get(id, self.rng) } + fn iv(&mut self, row: &RowLeakage) -> Tag16 { + self.ivs.get(row.data_key, self.rng) + } + + /// The token the client derives for `name`: its head's random token, + /// then each step by its rule. + fn token(&mut self, name: &TokenName) -> Tag16 { + let mut t = self.heads.get(name.head, self.rng); + for step in &name.steps { + t = match *step { + TokenStep::Member(s) => member_token(&t, &self.selector(s)), + TokenStep::Element(s, i) => element_token( + &t, + &self.selector(s), + u32::try_from(i).expect("a small position"), + ), + }; + } + t + } + /// An ordering term of `len` bytes for class `class`: a zero byte, the /// class's rank big-endian, then random bytes, so byte order is rank /// order and equal classes are equal terms. @@ -142,70 +181,119 @@ impl Simulator<'_> { json!({"sv": pairs, "el": elements}) } - /// Every entry's token and ordinal under its parent, where it has them. - fn tokens(&mut self, row: &RowLeakage) -> (Vec>, Vec>) { - let n = row.entries.len(); - let mut tokens: Vec> = vec![None; n]; - let mut ordinals: Vec> = vec![None; n]; - // A parent precedes its children only under the order clause; a - // mutant that shuffles the stored order still names parents, so the - // chain is resolved by depth rather than by position. - let depth: Vec = row - .entries + /// Entry `i`'s ordinal under its parent, whose token is `parent`: an + /// element's position, or a member's rank among its sibling members + /// under the simulator's own keyed rank. + fn ordinal(&mut self, row: &RowLeakage, i: usize, parent: &Tag16) -> u32 { + let entry = &row.entries[i]; + match entry.step { + Step::Element(n) => u32::try_from(n).expect("a small array"), + Step::Member => { + let mine = self.selector(entry.selector); + let my_rank = simulated_rank(&self.rank_key, parent, &mine); + let siblings: Vec = (0..row.entries.len()) + .filter(|&j| { + row.entries[j].parent == entry.parent && row.entries[j].step == Step::Member + }) + .collect(); + let below = siblings + .into_iter() + .filter(|&j| { + let theirs = self.selector(row.entries[j].selector); + simulated_rank(&self.rank_key, parent, &theirs) < my_rank + }) + .count(); + u32::try_from(below).expect("a small object") + } + Step::Root => 0, + } + } + + /// The self and parent tags every query's tokens link, row by row: + /// each token's self tag on the entry it names, and below it the chain + /// over the entries it reaches. + /// + /// Shallower named nodes first, so that where two reads link one entry + /// the one that names it last sets its tags, and every chain below it + /// is derived from that. Under the full profile the two always agree; + /// under a profile that states other rules than the code's (the page as + /// #1363 published it), this is what keeps the world consistent: a + /// node a read names keeps the token the read sends, as the database's + /// test of the member rule first then finds it. + fn link(&mut self) -> Vec { + let mut linked: Vec = self + .leakage + .rows .iter() - .map(|entry| { - let mut d = 0; - let mut at = entry.parent; - while let Some(p) = at { - d += 1; - at = row.entries[p].parent; - } - d - }) + .map(|_| Linked::default()) .collect(); - let mut order: Vec = (0..n).collect(); - order.sort_by_key(|&i| (depth[i], i)); - for &i in &order { - let entry = &row.entries[i]; - if let Some(head) = entry.head { - tokens[i] = Some(self.heads.get(head, self.rng)); - continue; + let leakage = self.leakage; + let depth = |row: &RowLeakage, mut at: usize| { + let mut d = 0; + while let Some(p) = row.entries[at].parent { + d += 1; + at = p; } - let Some(parent) = entry.parent else { continue }; - let Some(parent_token) = tokens[parent] else { + d + }; + let mut work = Vec::new(); + for query in &leakage.queries { + let QueryLeakage::Subtree { reads, .. } = query else { continue; }; - let ordinal = match entry.step { - Step::Element(i) => u32::try_from(i).expect("a small array"), - Step::Member => { - let mine = self.selector(entry.selector); - let my_rank = simulated_rank(&self.rank_key, &parent_token, &mine); - let siblings: Vec = (0..n) - .filter(|&j| { - row.entries[j].parent == Some(parent) - && row.entries[j].step == Step::Member - }) - .collect(); - let below = siblings - .into_iter() - .filter(|&j| { - let theirs = self.selector(row.entries[j].selector); - simulated_rank(&self.rank_key, &parent_token, &theirs) < my_rank - }) - .count(); - u32::try_from(below).expect("a small object") + for read in reads { + let t = self.token(&read.token); + for (r, row) in leakage.rows.iter().enumerate() { + let named = read.hits[r].map_or(0, |h| depth(row, h)); + work.push((named, work.len(), r, t, read)); } - Step::Root => continue, - }; - ordinals[i] = Some(ordinal); - let s = self.selector(entry.selector); - tokens[i] = Some(child_token(&parent_token, &s, ordinal)); + } + } + work.sort_by_key(|&(named, order, ..)| (named, order)); + for (_, _, r, t, read) in work { + let row = &leakage.rows[r]; + let iv = self.iv(row); + let mut tokens: HashMap = HashMap::new(); + if let Some(h) = read.hits[r] { + let _ = linked[r].x.insert(h, self_tag(&iv, &t)); + let _ = tokens.insert(h, t); + } + // Parents before children: the stored order puts them first, + // but a mutant may not, so by depth. + let mut reach = read.reach[r].clone(); + reach.sort_by_key(|&i| (depth(row, i), i)); + for c in reach { + let entry = &row.entries[c]; + let Some(pt) = entry.parent.and_then(|p| tokens.get(&p).copied()) else { + continue; + }; + let o = self.ordinal(row, c, &pt); + let s = self.selector(entry.selector); + let ct = match entry.rule { + Some(Rule::Member) => member_token(&pt, &s), + Some(Rule::Element) => element_token(&pt, &s, o), + None => continue, + }; + // A node a read names keeps that read's token, as the + // database's member-rule-first test then finds it. + let ct = match linked[r].x.get(&c) { + Some(x) if *x == self_tag(&iv, &member_token(&pt, &s)) => member_token(&pt, &s), + _ => { + let _ = linked[r].x.insert(c, self_tag(&iv, &ct)); + ct + } + }; + if entry.has_parent_tag { + let _ = linked[r].p.insert(c, parent_tag(&iv, &pt, o)); + } + let _ = tokens.insert(c, ct); + } } - (tokens, ordinals) + linked } - fn row(&mut self, r: usize, row: &RowLeakage) -> Value { - let iv: [u8; 16] = self.ivs.get(row.data_key, self.rng); + fn row(&mut self, row: &RowLeakage, linked: &Linked) -> Value { + let iv = self.iv(row); let tag = self .tags .entry(row.data_key) @@ -218,17 +306,6 @@ impl Simulator<'_> { header.extend_from_slice(&u16::try_from(tag.len()).expect("a short tag").to_le_bytes()); header.extend_from_slice(&tag); - let reached: HashSet = self - .leakage - .queries - .iter() - .filter_map(|q| match q { - QueryLeakage::Subtree { reach, .. } => Some(reach[r].iter().copied()), - _ => None, - }) - .flatten() - .collect(); - let (tokens, ordinals) = self.tokens(row); let entries: Vec = row .entries .iter() @@ -238,14 +315,16 @@ impl Simulator<'_> { let mut c = vec![CIPHERTEXT_FRAME]; c.extend(random_vec(self.rng, entry.ciphertext_len - 1)); let mut wire = json!({"s": b64(&s), "c": base85_simd::encode(&c)}); + if let Some(id) = entry.self_tag { + let x = match linked.x.get(&i) { + Some(x) => *x, + None => self.self_tags.get(id, self.rng), + }; + wire["x"] = json!(b64(&x)); + } if entry.has_parent_tag { - let chained = entry - .parent - .and_then(|p| tokens[p]) - .zip(ordinals[i]) - .filter(|_| reached.contains(&i)); - let p = match chained { - Some((parent, o)) => parent_tag(&iv, &parent, o), + let p: Tag8 = match linked.p.get(&i) { + Some(p) => *p, None => random(self.rng), }; wire["p"] = json!(b64(&p)); @@ -265,24 +344,12 @@ impl Simulator<'_> { QueryLeakage::Extract { selector } => { (QueryKind::Extract, json!(b64(&self.selector(*selector)))) } - QueryLeakage::Subtree { token, gates, .. } => { - let mut t = self.heads.get(token.head, self.rng); - for &(s, i) in &token.positions { - let s = self.selector(s); - t = child_token(&t, &s, u32::try_from(i).expect("a small position")); - } - let gates: Vec = gates + QueryLeakage::Subtree { reads, budget } => { + let tokens: Vec = reads .iter() - .map(|g| match *g { - GateLeak::Selector(s) => b64(&self.selector(s)), - GateLeak::Slot(n, s) => format!("#{n}:{}", b64(&self.selector(s))), - }) + .map(|read| b64(&self.token(&read.token))) .collect(); - let mut wire = json!({"t": b64(&t)}); - if !gates.is_empty() { - wire["g"] = json!(gates); - } - (QueryKind::Subtree, wire) + (QueryKind::Subtree, json!({"t": tokens, "budget": budget})) } QueryLeakage::Contains { needle, alt } => { let mut wire = self.group(needle); From fa26cb54085d3f84ec55fda9f0c2f60aa5cae588 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:01:19 +1100 Subject: [PATCH 03/19] test(stack-encrypt): derive child tokens only from tokens that find a node token_derivability derived member and element children of every token a read sends, which for a wildcard at the default bound is 65 tokens, most of them naming positions no row has, and their children name nothing either. Deriving only from tokens that find a node in some row halves the calibration's time (14.8 s to 9.6 s) with the same 44 of 44 mutants caught by the same distinguishers. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/distinguishers.rs | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 8311a45c1..35d7c97ea 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -1008,10 +1008,10 @@ pub fn member_ordinals_follow_key_order(world: &ObservedWorld) -> Option { /// Which subtree reads' tokens the database can derive from which others': /// `(i, j)` when a token sent by query `j` is one the walks from query -/// `i`'s tokens derive in some row, or is a child token of one of `i`'s by -/// the member rule under any selector the column holds, or by the element -/// rule at an index below 4 (a bound on what the harness tries, not on what -/// the database can). The page: a token's holder can "derive the tokens of +/// `i`'s tokens derive in some row, or is a child token of one of `i`'s +/// tokens that finds a node somewhere, by the member rule under any +/// selector the column holds, or by the element rule at an index below 4 +/// (bounds on what the harness tries, not on what the database can). The page: a token's holder can "derive the tokens of /// the nodes beneath". Catches tokens that are not chained (a token /// unrelated to its path's) as well as tokens chained where the profile /// says they are not. @@ -1030,7 +1030,14 @@ pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { .flatten() .flat_map(|(_, hits)| hits.iter().filter_map(|h| h.token)) .collect(); - for t in tokens { + // From the tokens that find a node in some row: one that names + // nothing anywhere (most of a wildcard's positions, on short arrays) + // derives only the tokens of paths no row has. + let found: BTreeSet = walks[q] + .iter() + .flat_map(|row| row.iter().map(|(k, _)| *k)) + .collect(); + for t in found.into_iter().map(|k| &tokens[k]) { for s in &selectors { let _ = derived.insert(member_token(t, s)); for o in 0..4 { From 09a9dfc0be3a932311d4b61ea8173866d550899c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:02:33 +1100 Subject: [PATCH 04/19] docs(plans): compare the json_tree leakage profile before and after #1363 The leakage profile in #1362 was written against the format before #1363 and has been ported to it. This records, from the code at 9a7ca372c and at the port, every clause, distinguisher and mutant of the two profiles side by side; the failures the old test showed on first meeting the new format and what each one taught; the calibration before and after; and what the database learns now that it did not (the self tag, the token count and the bound it gives away, the probes, the member-or-element rule under a token) and what it no longer learns (an ancestor's capability in every row, the gates with their plaintext positions and selectors, the head token). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- ...0-25-json-tree-leakage-before-and-after.md | 201 ++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 docs/plans/2026-10-25-json-tree-leakage-before-and-after.md diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md new file mode 100644 index 000000000..6a85bdee2 --- /dev/null +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -0,0 +1,201 @@ +# The json_tree leakage profile, before and after #1363 + +**Date:** 2026-10-25. **Pull request:** #1362 (branch `test/json-tree-leakage-profile`), now based on #1363 (`feat/json-tree-derivable-tokens`). + +This note compares two versions of one test, and through them two versions of EQL v4's encrypted JSON column, `json_tree`. It is written for someone new to both. + +## What the test is + +A `json_tree` column stores each JSON document as a list of encrypted entries, one per node of the document, so PostgreSQL can answer queries over it without the key. The database cannot read a value, but it can see a great deal of structure, and `packages/eql/docs/reference/json-tree-leakage.md` (the **leakage page**) says in prose exactly what it learns. + +PR #1362 turns that page into a test, in `packages/stack-encrypt/tests/json_leakage/`: + +- The **profile**, `L` (`profile.rs`), is the page written as code: a function of the plaintext documents and queries alone that outputs what the page says the database learns. Every secret-derived value (a selector, a token, an ordering term) is replaced by an opaque integer that is equal exactly where the real bytes are equal. Each field of its output is one **clause** of the page. +- The **simulator** (`simulator.rs`) builds a fake column and a fake query log from `L`'s output and random bytes, with no plaintext and no key. +- The **distinguishers** (`distinguishers.rs`) are computations a curious database administrator could run on what the database holds and is sent: which entries share a selector, how long each ciphertext is, what a token finds when it walks the parent tags, and so on. Each returns an exact value, so a real world and a simulated one are compared for equality. +- The **positive control** runs every distinguisher on the real scheme (documents sealed by Stack Encrypt) and on the simulation. If any distinguisher tells them apart, the real scheme reveals something the profile, and so the page, does not state. +- The **calibration** checks the test can see anything at all. Each **mutant** (`mutants.rs`) is the profile with one clause dropped or coarsened; some distinguisher must tell each mutant's simulation from the real scheme. A clause no mutant tests is a clause nobody checked. + +#1362 was written against the format as it stood before #1363. #1363 changed the format. This note records what the port of the test found, and how the leakage changed. + +## The two formats, briefly + +Terms used below: a **selector** `s` is a keyed hash of a node's path, the same in every row. A **token** is a 16-byte secret for one node that lets the database find that node and walk the nodes beneath it. A **parent tag** `p` on an entry is `H8("p" ‖ iv ‖ t(parent) ‖ LE32(o))`: eight bytes derived from the parent's token, the child's **ordinal** `o` (its index if it is an array element, a keyed rank if it is an object member) and the row's random `iv`. The database enumerates a node's children by computing the parent tag of ordinals 0, 1, 2, … until one is missing. + +**Before #1363** (the format #1362 was written for, at `9a7ca372c`): + +- Each entry was `{s, c, p?, op?}`: selector, ciphertext, parent tag (absent on the root and, under the sealed root, on the top-level fields), ordering term (only at ordered paths). +- Every child's token was `H16("t" ‖ t(parent) ‖ s(child) ‖ LE32(o))`. A member's ordinal is a rank among its siblings' keys, which the client does not know for a row it has not read, so the client could not derive the token of anything below a top-level field except through array positions. +- A subtree read therefore sent one **head token** (the top-level field's, or the root's with the root open), and one plaintext **gate** per remaining step: the step's selector, or `#n:` with the array position `n` in the clear. The database walked from the head token through the gates (`json_tree_subtree_at`). Reading `$.profile.address.city` handed the database `$.profile`'s token: a capability over that whole top-level field, in every row, for the life of the column (#1360). + +**After #1363** (the format now, at `997c64b1e`): + +- Each entry is `{s, c, x, p?, op?}`. The new **self tag** is `x = H8("x" ‖ iv ‖ t(self))`, on every entry, the root and the top-level fields included. +- A member's token is `H16("t" ‖ t(parent) ‖ s)`, with no ordinal. An element's is `H16("t" ‖ t(array) ‖ s ‖ LE32(i))`, `i` its index: this is the old rule, unchanged, since an element's ordinal always was its index. The parent tag is unchanged. So the client derives any node's token from its path alone. +- A read sends `json_tree_subtree(doc, tokens text[], budget)` with the tokens of exactly the nodes it names: one for a path with no wildcard. A wildcard is expanded by the client to positions `0 … N − 1` plus a **probe** for position `N` of every array it expands; the **bound** `N` defaults to 64, so one wildcard level sends 65 tokens and two send `N² + N + 1` = 4,161, and a read past 8,192 tokens is refused. A probe that finds an element means the array is longer than `N`, and the client fails the read rather than return part of it. Gates and `json_tree_subtree_at` are gone. +- The database finds a token's node by its self tag, probes the ordinals under it by parent tag, and for each child it finds derives the child's token by testing the child's `x` against the member rule first and then the element rule. +- Past the **budget** (default 64 rows) with the root sealed, the database returns the root entry alone, whose ciphertext is the whole document, and the client evaluates the read over it. + +## Step 1: what failed when the old test met the new format + +The test was run unchanged on the merge of #1363 into #1362 (`848940081`). The failures, in the order they appeared, were the first evidence of what changed. + +1. **It did not compile.** Two `E0308` errors in `calibration.rs`, at lines 324 and 329: `JsonSubtreeKey::token_bytes()` now returns `Vec<[u8; 16]>`, and `JsonDocument::subtree` takes `&[Tag]`. *What it taught:* a read is no longer one token; it is a list. + +2. **With a throwaway two-character fix to compile it, 1 test passed and 6 failed.** Five panicked in the parser (`observe.rs:126`, "a base64 field is a string") before any distinguisher ran: the read's wire form changed from `{"t": "", "g": [gates]}` to `{"t": ["", …]}`. The sixth, `chain_is_the_crates_chain`, compares the harness's own walk with the crate's. For `$.a` under the sealed root, the old harness found `[(6, 1), (2, 1)]` and the crate `[(1, 0), (6, 1), (2, 1), (3, 2), (4, 2), (5, 3)]` (pairs are entry position and depth). *What it taught:* the crate now returns the named node itself at depth 0, found by its self tag, which the old walk could not identify. The old harness still found `$.a`'s two children, because the parent tag and the head token are unchanged; it stopped there because both children are members and a member's token lost its ordinal. In the column-wide run below, the old harness did reach depth 2 under array elements, whose rule did not change. The one test that passed, `the_order_reveals_the_shape`, uses no query: the attack that reads each row's shape off the stored order works exactly as before. + +3. **With the parser also patched to read the first token of the list, the positive control failed on exactly five distinguishers**, on the handcrafted "shapes, sealed root" column: `wire_fields`, `token_reach`, `token_derivability`, `gated_walk_results` and `transcripts`. The random-column run failed on `wire_fields` and `transcripts`, shrunk by proptest to two empty objects under an open root and four reads of `$.a`. *What it taught:* at rest, the only thing that changed is the new `x` field (`wire_fields`). Every other stored-form feature (ciphertext lengths, selector patterns, parent-tag presence, the stored order, projections, ordering terms, headers) still agreed with the old profile. Everything else that changed is in what a query sends and what a token reaches. + +4. **The page-as-written test fired six distinguishers instead of one**: `gated_walk_results`, `ordering_term_lengths`, `token_derivability`, `token_reach`, `transcripts`, `wire_fields`, where it expected exactly `ordering_term_lengths`. The jsonb key-order test still passed. + +5. **The calibration passed: all 34 mutants "caught".** *What it taught:* this is the most important lesson of the run. A calibration only means something once the positive control passes. If the profile is already told apart from the real scheme, every mutant of it is too, and "all caught" says nothing. `mod.rs` now says so under "Reading a failure". + +## The clauses, before and after + +The "before" column is the profile at `9a7ca372c`; the "after" column is the port at `839c0f6fd` (with `fa26cb540`, which only speeds one distinguisher up). A clause is a field of the profile's output, or a switch in `Clauses`. + +| clause | before (`9a7ca372c`) | after (port) | what changed and why | +|---|---|---|---| +| `EntryLeakage::selector` | which paths, how many nodes at each, equal across rows, elements share their array's | same | unchanged | +| `EntryLeakage::parent` | each node's parent, used to place the chain | same | unchanged | +| `EntryLeakage::step` | `Root`, `Member`, `Element(i)`: decides the ordinal, and in the old chain the token too | `Root`, `Member`, `Element(i)`: decides the ordinal (the `o` of the parent tag) only | the token rule moved to `rule` | +| `EntryLeakage::rule` | (none) | `Member` or `Element` for every entry that chains, `None` for the root and sealed-root top-level entries | new: the database tests each reached node against the member rule then the element rule, and learns which. Beyond the page as #1363 published it; this branch adds it | +| `EntryLeakage::head` | the head token an entry is (top-level fields under the sealed root, the root otherwise) | removed | a head is now named only inside a token name (`TokenName::head`); nothing at rest refers to it | +| `EntryLeakage::has_parent_tag` | absent on the root and, under the sealed root, the top-level fields | same | unchanged | +| `EntryLeakage::self_tag` | (none) | a fresh opaque name per entry | new: `x` on every entry. A fresh name per entry states that it is distinct within a row and unlinkable across rows; where a sent token links an entry the simulator writes its real self tag instead | +| `EntryLeakage::ciphertext_len` | AEAD overhead (29 bytes) + kind, child count, label, scalar | same | unchanged: `x` sits beside the ciphertext. It adds a fixed 12 base64 characters per entry, 19 with its key in compact JSON (`,"x":"…"`), 21 in PostgreSQL's `jsonb` text (`, "x": "…"`); #1363 measured +9.8% on a nested document and +9.4% on a flat one | +| `EntryLeakage::ordering` / `OrderingLeak::class` | equal values share a class, ordered by `ordering_ranks`; strings ordered by their bytes | equal values share a class; strings ordered by their folded encoding (Unicode 16 NFC, case and accents folded) | the code always folded; the old corpus held only lower-case ASCII, where the two agree. The page now says "folded", and the corpus holds `"Carol"`/`"carol"` and `"ál"`/`"al"` | +| `OrderingLeak::len` | **beyond the page**: 65 bytes for a number, `8 × (len + 1) + 1` for a string | on the page: 65 for a number, `8 × (n + 1) + 1` for a string of folded length `n` | #1363's page states it; it now counts the folded length | +| `Clauses::ordering_term_length` | on/off switch for the beyond-the-page length | removed | on the page now; its mutant (`ordering_term_length_constant`) still checks it | +| `Clauses::key_order` | a reading: UTF-8 byte order vs jsonb's length-first order | same | the page now says "byte order of their key names' UTF-8 encodings"; the reading test stays to keep that sentence load-bearing | +| `Clauses::string_order` | (none) | a reading: folded vs raw bytes | new, with a test that the raw-bytes reading is told apart | +| `Clauses::hit_rule` | (none) | on/off switch for the member-or-element rule | new, the one clause beyond the page as #1363 published it | +| `RowLeakage::{sealed_root, keyset, data_key, entries, projection}` | header flag, keyset, data key unique to the row, entries in depth-first order with members in UTF-8 key order, projection `q` | same | unchanged. `data_key` also names the `iv` that salts every `p` and `x` in the row | +| `GroupLeakage::{pairs, elements}` | `{s, vs}` pairs in walk order, nested by array element | same | unchanged | +| `QueryLeakage::Extract` | the selector of the path asked | same | unchanged | +| `QueryLeakage::Subtree` | `token: TokenName{head, positions}` (one head token, chained through leading array positions), `gates` (`Selector(s)` or `Slot(n, s)`, positions in the clear), `reach` (every descendant of the token's node, in every row), `named` | `reads: Vec` and `budget` | rewritten. See the next three rows | +| `ReadLeak::token` | (one per read, see above) | `TokenName{head, steps}`, each step `Member(s)` or `Element(s, i)`: one per concrete path, in the order the client sends them | the list is what the database receives; its length gives the bound away, its order places each token | +| `ReadLeak::probe` | (none) | whether the token is a probe | new; the database is not told, and can tell (see findings) | +| `ReadLeak::hits` / `ReadLeak::reach` | `reach`: descendants of the head's node; `named`: what the gates reached | `hits`: the entry the token names in each row, if any; `reach`: every descendant of it that chains from it, in each row | a token now names exactly its node; nothing above it. Under the sealed root the root's token reaches nothing below the root | +| `QueryLeakage::Subtree::budget` | (none) | the `budget` argument | new; the fallback to the root entry is computed from it and the walk, so the database learns nothing new from it | +| `QueryLeakage::{Contains, Equals, Order}` | needle projection with `alt`; equality needle; selector and the bound's ordering term | same | unchanged | +| `Leakage::{key_tag_len, ordering_ranks}` | the key tag's length, the order of every ordering class | same | unchanged | + +## The distinguishers, before and after + +27 before, 35 after: 26 kept unchanged in meaning, 1 renamed, 1 removed (by the rename), 8 added. The walk-based ones were rewritten for the new chain, but what each measures is the same. + +| distinguisher | status | what it measures | +|---|---|---| +| `wire_fields` | kept | the wire keys of every row and entry (now `{c, op?, p?, s, x}`) | +| `entry_counts` | kept | entries per row | +| `parent_tag_presence` | kept | which entries carry `p` | +| `ordering_term_presence` | kept | which entries carry `op` | +| `ciphertext_lengths` | kept | every ciphertext's length, by position | +| `ciphertext_length_multisets` | kept | the same, order removed | +| `ciphertext_framing` | kept | the ciphertexts' version bytes, and how many pairs are equal (a deterministic cipher) | +| `selector_pattern_within_rows` | kept | which entries of a row share a selector | +| `selector_pattern_across_rows` | kept | which entries of the column share a selector | +| `parent_tag_equalities` | kept | pairs of entries sharing a `p` (expected none) | +| `headers` | kept | header fields, and equality of keyset, `iv` and key tag across rows | +| `projections` | kept | each row's `q`: grouping, pair order, selector and value-selector equality | +| `value_selector_partitions` | kept | per path, the rows grouped by value | +| `chain_head_runs` | kept | the runs after each entry with no `p`: each top-level field's subtree size | +| `selector_precedence` | kept | which of two selectors comes first across rows: sibling key order | +| `depth_recovery` | kept | an attack that reconstructs every row's depth sequence from the stored order alone | +| `token_reach` | kept | per read, row and token: the node it finds and everything its walk links, with parents and depths | +| `ordinal_patterns` | kept | elements' ordinals in position order; members ranked consistently under one token | +| `member_ordinals_follow_key_order` | kept | whether members' ordinals follow key order, reported only below a `2⁻⁴⁰` chance bound | +| `token_derivability` | kept | which reads' tokens the database can derive from which others' (now by the member rule and by the element rule at an index below 4, from the tokens that find a node in some row) | +| `transcripts` | kept | every query term with its tags named: determinism, shared selectors, needles; a read is now its token list and budget instead of a token and gates | +| `extraction_matches` | kept | which entries each `->` selector matches | +| `containment_matches` | kept | which rows each needle matches | +| `ordering_term_order` | kept | every ordering term's rank, stored and sent | +| `ordering_term_lengths` | kept | every ordering term's length | +| `ordering_matches` | kept | which entries each ordering bound holds for | +| `gated_walk_results` → `subtree_results` | renamed | what a read returns from each row: before, the walk through the gates; after, every token's node and subtree, or the root entry alone past the budget under the sealed root | +| `stored_lengths` | added | each row's and each entry's length in characters: sees `x`'s fixed 19 characters | +| `self_tag_equalities` | added | pairs of entries whose `x` equals another `x` or a `p` anywhere (expected none) | +| `token_counts` | added | how many tokens each read sends | +| `bound_recovery` | added | the `(levels, N)` an observer recovers from the count: 65 is one level at 64 and nothing else; 4,161 is two levels at 64 or one at 4,160 | +| `token_hits` | added | per read and row, which tokens (by place in the list) find a node, and where | +| `wildcard_positions` | added | for a one-level wildcard read: per row, which positions below `N` hold the rest of the path, whether the probe found an element, and whether the probe's find carries a selector the others do not | +| `hit_rules` | added | for every node a walk reaches, which rule its `x` matched: member, element, or neither | +| `ancestor_links` | added | whether the database can link each node a read names to its parent with any token it holds: the "no ancestor capability" check | + +## The mutants, before and after + +34 before, 44 after: 28 kept unchanged, 4 redefined for token lists, 2 removed, 12 added. + +| mutant | status | clause it drops | +|---|---|---| +| `selectors_all_distinct`, `selectors_per_row_only`, `positions_in_selectors` | kept | selector equality; column-wide; array positions folded | +| `entry_count_padded` | kept | the number of entries | +| `order_random_permutation`, `members_unsorted` | kept | the stored order; members in key order | +| `parent_tag_on_every_entry`, `parent_tag_on_no_entry` | kept | which entries lack `p` | +| `ciphertext_length_bucketed`, `sealed_root_length_hidden` | kept | ciphertext length | +| `value_selectors_all_distinct`, `value_selectors_path_only` | kept | value-selector equality | +| `projection_flat`, `projection_shuffled` | kept | element grouping and pair order in `q` | +| `ordering_terms_absent`, `ordering_terms_unordered`, `ordering_terms_all_distinct`, `ordering_term_length_constant`, `ordering_terms_per_path` | kept | `op` presence, order, equality, length, equality across paths | +| `member_ranks_follow_key_order` | kept | a member's ordinal is a keyed rank | +| `sealed_flag_flipped`, `data_key_shared`, `keyset_per_row` | kept | the header | +| `extraction_selector_fresh`, `queries_nondeterministic` | kept | `->` sends the stored selector; queries are deterministic | +| `needle_flat`, `needle_alt_dropped`, `bound_rank_random` | kept | needle grouping, `alt`, the ordering bound's rank | +| `tokens_per_row` | redefined | before: head tokens per row; after: a token finds its node in one row only | +| `parent_tags_unchained` → `tokens_find_nothing` | redefined | a token finds and links nothing | +| `token_reaches_named_nodes_only` → `token_reaches_its_node_only` | redefined | before: the head token reaches only the path asked (the narrower reach #1360 wanted); after: a token finds its node and links nothing below it | +| `position_tokens_unchained` → `tokens_not_chained` | redefined | a token with a step below its head is unrelated to its head's | +| `slot_positions_hidden`, `gates_dropped` | removed | gates no longer exist | +| `self_tags_absent`, `self_tags_linkable_across_rows`, `self_tags_per_selector` | added | `x` on every entry; salted by the row; distinct per entry | +| `hit_rule_uniform` | added | the member-or-element rule (the page fix) | +| `probe_reaches_nothing` | added | a probe that finds an element links its subtree (the page's exception) | +| `ancestor_token_sent` | added | no ancestor's token: the pre-#1363 read, which also sent the top-level field's token | +| `member_tokens_carry_ordinal` | added | a member's token has no ordinal: the pre-#1363 member rule, under which reads through a key step find nothing | +| `element_tokens_without_index` | added | an element's token carries its index | +| `probes_dropped`, `token_list_shuffled`, `token_list_padded` | added | a probe per expanded array; the list's order; the list's length | +| `budget_not_sent` | added | the budget, in the clear | + +## Calibration, before and after + +| | before (the profile at `9a7ca372c`, on the old format) | first run on the new format (step 1) | after the port (`839c0f6fd`, `fa26cb540`) | +|---|---|---|---| +| tests | 7, all passing, about 3 s (the PR measured 6.3 s for the binary) | did not compile; with throwaway patches 1 of 7 passing | 9, all passing, 11.3 s for the binary: the calibration 9.6 s, every other test under 0.5 s | +| positive control, handcrafted columns | 4 columns, no distinguisher separates | fails: 5 distinguishers separate | 4 columns, no distinguisher separates | +| positive control, random columns | 48 cases pass | fails on the first case (`wire_fields`, `transcripts`) | 48 cases pass | +| mutants caught | 34 of 34 | 34 of 34, meaningless (the control failed) | 44 of 44 | +| page-as-written test | exactly `ordering_term_lengths` separates | 6 separate | renamed `the_hit_rule_separates_it_from_the_real_scheme`: exactly `hit_rules`, `subtree_results`, `token_derivability`, `token_reach` | +| `member_ordinals_follow_key_order` | decided on the wide-objects column, `2⁻⁴⁰` bound | not reached | decided on the wide-objects column (`Some(false)`: members' ordinals do not follow key order), same bound | + +The ported handcrafted columns exercise every new mechanism: one-level reads at the default bound (65 tokens) and at bounds the rows overflow, a two-level read (7 tokens at a bound of 2), probes that find elements in every column (3, 3, 1 and 1 probe hits), a probe whose element no other read names (so `probe_reaches_nothing` is load-bearing), `$` under both root options, a read below a key step (`$.nested.deep`, which before #1363 needed a gate), and a read past its budget under the sealed root (3 rows answered with the root alone). The random columns draw bounds from {1, 2, 3, 64} and budgets from {0, 2, 64}. + +Two tests are new. `given_strings_ordered_by_their_bytes` holds the page's folding sentence: read as raw bytes, the profile is told apart by `ordering_term_lengths` and `ordering_term_order`. `a_read_gives_away_its_bound_and_its_probes` runs `$.items[*].price` and `$.items[*].tags[*]` on the real scheme over rows with 0, 3 and 70 items: the counts are 65 and 4,161, the bound is recovered, and in the 70-item row the database sees positions 0 and 2 … 63 holding a price, the probe finding an element, and the probe's find carrying a different selector from the others. + +## What the port found + +**One finding against the page as #1363 published it, fixed on this branch.** Nothing stored says whether a node is an object member or an array element (the design's decision 2, "no kind bit"). But to derive the token of each node it reaches, the database tests the node's self tag against the member rule and then the element rule, and the one that matches says which. So below any token it is sent, the database learns whether each node is a member or an element, and therefore whether each parent is an object or an array; for an element it also learns the index, which is the ordinal it was found at. Where selectors already show it (an array that has two elements in some row) this adds nothing; for a node that is an only child in every row, it does. The page said a token's holder can "see how the nodes beneath it nest, link each one to its parent, and derive the tokens of the nodes beneath". This branch adds: "It also learns whether each node beneath is an object member or an array element, and each element's index". `the_hit_rule_separates_it_from_the_real_scheme` shows the sentence is needed: without it, `hit_rules` separates the worlds, and so do the three features that break when a world tries to hide the rule while still matching the tokens a client sends (a read through a key step always sends a member-rule token). + +**One inaccuracy in a SQL comment, reported, not changed.** `json_tree_subtree`'s comment in `packages/eql/src/v3/json_tree/functions.sql` says "the server cannot tell a probe from any other token". It cannot from a token's bytes. But the probe is always the last token at its level, the count gives the bound away, and a probe that finds something finds the element, whose selector differs from that of the leaf the path's other tokens find. The leakage page does not repeat the claim. The comment is #1363's to correct. + +**One error in the old profile, which its corpus could not show.** It ordered strings by their raw bytes. The code folds case and accents before ordering, so `"Carol"` and `"carol"` share a term, and the term's length counts the folded form. The old corpus held only lower-case ASCII, where the two agree. The port models folding, the corpus now holds names that fold, and a test holds the page's sentence. + +**The old page's three findings are closed.** The ordering term's length, equal values at two ordered paths sharing a term, and UTF-8 byte order of keys are all on the page now. Each keeps a mutant or a reading that the calibration catches. + +**The tension stands.** #1358's read check, kept by #1363 (decision 8), refuses a subtree's rows unless they sit at consecutive positions `j`. That consecutiveness is what #1357 wants gone from the stored order. Both cannot hold as written; it is the owner's decision. + +## What the database learns now that it did not before + +- **The self tag, at rest: nothing.** Every entry carries 8 more bytes (12 base64 characters; 19 more characters per entry in compact JSON, 21 in `jsonb` text; about a tenth of a document's size). The tag is salted by the row's random `iv`, so no two are equal in a column (`self_tag_equalities` is empty in both worlds), and since every entry has one, its presence marks none. +- **The self tag, under a token: the named node itself.** Before, a head token let the database find the head's children, not the head; it never identified the node a token belonged to (the old walk recorded its children's parent as unknown). Now a token finds its node directly, by `x`, in every row that has it. +- **The token count, and with it the bound.** A read with no wildcard sends 1 token. One wildcard level sends `N + 1` (65 at the default), two send `N² + N + 1` (4,161), and 8,192 is the cap. 65 identifies `N = 64` uniquely; 4,161 is either two levels at 64 or one at 4,160, and which tokens find nodes settles it. +- **The probes.** Per row, whether each expanded array is longer than `N`, and, where a probe finds an element, that element's whole subtree, including the siblings of the path asked for, although the read then fails. The probe is identifiable by its place in the list and by what it finds. +- **Which elements hold the rest of the path.** For `$.items[*].price`, per row, which positions below `N` have a `price`. Before, the gated walk returned the same rows, so the result was visible; now each position also has its own token, column-wide. +- **Member or element, for every reached node** (the finding above). +- **The budget**, in the clear. The fallback to the root entry is computed by the database from the walk it already sees, so it learns nothing more from it; someone who sees only the results learns, from a row answered with its root alone, that the read found more than `budget` rows there. + +## What the database no longer learns + +- **An ancestor's capability in every row.** Before, any read below a top-level field sent that field's token, a capability over the whole field's structure in every row for the life of the column, and under an open root every read sent the root's token, the whole column's shape. Now no token over any node above the one read leaves the client unless that node is itself read. `ancestor_links` is the check: in the ported worlds a named node's parent is linked only where another read named one of its ancestors, and the mutant that sends the pre-#1363 head token is caught by it, by `token_reach` and by the token counts. +- **The gates.** Before, a read sent the selector of every step below the head, and every array position in plaintext (`#n:`), whether or not a row had the path. Now no selector is sent with a read: the database learns the selector of the node read only from the entries the token finds. Array positions travel only inside tokens; the database still infers a found node's position from the stored order (#1357), and, for a wildcard, from the token's place in the list. +- **The head token.** Reading `$.profile.address.city` sends the city leaf's token and nothing that covers `$.profile`. Under the sealed root the root's token, sent only for a read of `$`, finds the root entry and nothing beneath it, since the top-level fields carry no parent tag. + +Everything at rest other than `x` is unchanged: shape through the stored order, parent structure, sibling key order, selector equality across rows, value-selector equality, ciphertext lengths, ordering terms with their lengths and their equality across ordered paths, and the header. From 64821b08e8db8c9e1f759b4e59d703c8b7e63cfb Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:17:18 +1100 Subject: [PATCH 05/19] docs(plans): lead the leakage before-and-after note with the net The note's tables grow from the old profile to the new one, and read as a worsening to a reader who stops there; the owner read them that way. The conclusion was in the last two sections. It now opens the note: the growth measures the test, and the format discloses less, with the ancestor capability and the gates gone and the additions small. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../2026-10-25-json-tree-leakage-before-and-after.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index 6a85bdee2..7b8cce389 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -18,6 +18,15 @@ PR #1362 turns that page into a test, in `packages/stack-encrypt/tests/json_leak #1362 was written against the format as it stood before #1363. #1363 changed the format. This note records what the port of the test found, and how the leakage changed. +## The net, first + +The tables below grow from the old profile to the new one: more clauses, more distinguishers, more mutants. That measures the test, not the scheme. A longer profile means more of what the database sees is written down and held by a test. Measured as leakage, the format after #1363 discloses less, and less where it matters: + +- A read no longer sends any token above the node it asks for. Before, a read of `$.profile.address.city` sent `$.profile`'s token, a capability over that whole field in every row of the column, for the life of the column. Now it sends the leaf's token and nothing that covers a node above it. The `ancestor_links` distinguisher exists to hold this. +- The gates are gone. Each key step and array position below the head used to reach the server in the clear, as a selector or `#n:selector`. + +What is new is small. The self tag `x` is a fresh unlinkable name per entry that at rest says only that entries are distinct, which the entry count already said; a token the client sends links it, as a token always linked the parent tag. The member-or-element rule and the element index at the nodes a walk reaches were already disclosed by the gated walk's ordinals and `#n:` gates, but the page had not said so, and now does. A wildcard read's token count, bound and probes are a smaller form of what the plaintext gates showed. The last two sections set each item out. + ## The two formats, briefly Terms used below: a **selector** `s` is a keyed hash of a node's path, the same in every row. A **token** is a 16-byte secret for one node that lets the database find that node and walk the nodes beneath it. A **parent tag** `p` on an entry is `H8("p" ‖ iv ‖ t(parent) ‖ LE32(o))`: eight bytes derived from the parent's token, the child's **ordinal** `o` (its index if it is an array element, a keyed rank if it is an object member) and the row's random `iv`. The database enumerates a node's children by computing the parent tag of ordinals 0, 1, 2, … until one is missing. From a386b9c4fe228e1040e40fc16983b1bb6156ce0f Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:19:39 +1100 Subject: [PATCH 06/19] docs(plans): say which kind of absence each (none) in the clause table is An old-profile clause marked (none) was either a thing the old format did not have, or a thing it had that the old profile failed to model; the table did not say which, and the owner asked. Each row now says, and a paragraph above the table names the method's limit: a clause the profile and the distinguishers both omit is invisible to the calibration. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- ...026-10-25-json-tree-leakage-before-and-after.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index 7b8cce389..d73b38edd 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -63,30 +63,32 @@ The test was run unchanged on the merge of #1363 into #1362 (`848940081`). The f The "before" column is the profile at `9a7ca372c`; the "after" column is the port at `839c0f6fd` (with `fa26cb540`, which only speeds one distinguisher up). A clause is a field of the profile's output, or a switch in `Clauses`. +A "(none)" in the before column means the old profile had no such clause, for one of two reasons, and the row says which: *no such thing* (the old format had nothing to state: `x`, the hit rule as a rule, probes) or *a gap in the old profile* (the old format disclosed it and the old test did not model it, so no distinguisher could see it: string folding, whose corpus was lower-case ASCII, and the budget, which the old `json_tree_subtree` also had). The second kind is the method's limit: the calibration proves the page accounts for everything the distinguishers measure, not everything the database sees, and a clause both omit is invisible to it. Each new distinguisher is a way of finding one. + | clause | before (`9a7ca372c`) | after (port) | what changed and why | |---|---|---|---| | `EntryLeakage::selector` | which paths, how many nodes at each, equal across rows, elements share their array's | same | unchanged | | `EntryLeakage::parent` | each node's parent, used to place the chain | same | unchanged | | `EntryLeakage::step` | `Root`, `Member`, `Element(i)`: decides the ordinal, and in the old chain the token too | `Root`, `Member`, `Element(i)`: decides the ordinal (the `o` of the parent tag) only | the token rule moved to `rule` | -| `EntryLeakage::rule` | (none) | `Member` or `Element` for every entry that chains, `None` for the root and sealed-root top-level entries | new: the database tests each reached node against the member rule then the element rule, and learns which. Beyond the page as #1363 published it; this branch adds it | +| `EntryLeakage::rule` | (none: no such thing) | `Member` or `Element` for every entry that chains, `None` for the root and sealed-root top-level entries | new: the database tests each reached node against the member rule then the element rule, and learns which. Beyond the page as #1363 published it; this branch adds it | | `EntryLeakage::head` | the head token an entry is (top-level fields under the sealed root, the root otherwise) | removed | a head is now named only inside a token name (`TokenName::head`); nothing at rest refers to it | | `EntryLeakage::has_parent_tag` | absent on the root and, under the sealed root, the top-level fields | same | unchanged | -| `EntryLeakage::self_tag` | (none) | a fresh opaque name per entry | new: `x` on every entry. A fresh name per entry states that it is distinct within a row and unlinkable across rows; where a sent token links an entry the simulator writes its real self tag instead | +| `EntryLeakage::self_tag` | (none: no such thing) | a fresh opaque name per entry | new: `x` on every entry. A fresh name per entry states that it is distinct within a row and unlinkable across rows; where a sent token links an entry the simulator writes its real self tag instead | | `EntryLeakage::ciphertext_len` | AEAD overhead (29 bytes) + kind, child count, label, scalar | same | unchanged: `x` sits beside the ciphertext. It adds a fixed 12 base64 characters per entry, 19 with its key in compact JSON (`,"x":"…"`), 21 in PostgreSQL's `jsonb` text (`, "x": "…"`); #1363 measured +9.8% on a nested document and +9.4% on a flat one | | `EntryLeakage::ordering` / `OrderingLeak::class` | equal values share a class, ordered by `ordering_ranks`; strings ordered by their bytes | equal values share a class; strings ordered by their folded encoding (Unicode 16 NFC, case and accents folded) | the code always folded; the old corpus held only lower-case ASCII, where the two agree. The page now says "folded", and the corpus holds `"Carol"`/`"carol"` and `"ál"`/`"al"` | | `OrderingLeak::len` | **beyond the page**: 65 bytes for a number, `8 × (len + 1) + 1` for a string | on the page: 65 for a number, `8 × (n + 1) + 1` for a string of folded length `n` | #1363's page states it; it now counts the folded length | | `Clauses::ordering_term_length` | on/off switch for the beyond-the-page length | removed | on the page now; its mutant (`ordering_term_length_constant`) still checks it | | `Clauses::key_order` | a reading: UTF-8 byte order vs jsonb's length-first order | same | the page now says "byte order of their key names' UTF-8 encodings"; the reading test stays to keep that sentence load-bearing | -| `Clauses::string_order` | (none) | a reading: folded vs raw bytes | new, with a test that the raw-bytes reading is told apart | -| `Clauses::hit_rule` | (none) | on/off switch for the member-or-element rule | new, the one clause beyond the page as #1363 published it | +| `Clauses::string_order` | (none: a gap in the old profile) | a reading: folded vs raw bytes | new, with a test that the raw-bytes reading is told apart | +| `Clauses::hit_rule` | (none: no such thing) | on/off switch for the member-or-element rule | new, the one clause beyond the page as #1363 published it | | `RowLeakage::{sealed_root, keyset, data_key, entries, projection}` | header flag, keyset, data key unique to the row, entries in depth-first order with members in UTF-8 key order, projection `q` | same | unchanged. `data_key` also names the `iv` that salts every `p` and `x` in the row | | `GroupLeakage::{pairs, elements}` | `{s, vs}` pairs in walk order, nested by array element | same | unchanged | | `QueryLeakage::Extract` | the selector of the path asked | same | unchanged | | `QueryLeakage::Subtree` | `token: TokenName{head, positions}` (one head token, chained through leading array positions), `gates` (`Selector(s)` or `Slot(n, s)`, positions in the clear), `reach` (every descendant of the token's node, in every row), `named` | `reads: Vec` and `budget` | rewritten. See the next three rows | | `ReadLeak::token` | (one per read, see above) | `TokenName{head, steps}`, each step `Member(s)` or `Element(s, i)`: one per concrete path, in the order the client sends them | the list is what the database receives; its length gives the bound away, its order places each token | -| `ReadLeak::probe` | (none) | whether the token is a probe | new; the database is not told, and can tell (see findings) | +| `ReadLeak::probe` | (none: no such thing) | whether the token is a probe | new; the database is not told, and can tell (see findings) | | `ReadLeak::hits` / `ReadLeak::reach` | `reach`: descendants of the head's node; `named`: what the gates reached | `hits`: the entry the token names in each row, if any; `reach`: every descendant of it that chains from it, in each row | a token now names exactly its node; nothing above it. Under the sealed root the root's token reaches nothing below the root | -| `QueryLeakage::Subtree::budget` | (none) | the `budget` argument | new; the fallback to the root entry is computed from it and the walk, so the database learns nothing new from it | +| `QueryLeakage::Subtree::budget` | (none: a gap in the old profile; the old read had a budget too) | the `budget` argument | new; the fallback to the root entry is computed from it and the walk, so the database learns nothing new from it | | `QueryLeakage::{Contains, Equals, Order}` | needle projection with `alt`; equality needle; selector and the bound's ordering term | same | unchanged | | `Leakage::{key_tag_len, ordering_ranks}` | the key tag's length, the order of every ordering class | same | unchanged | From d6f02a413dde606e80c7cec9c5d402606f8b416d Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:20:11 +1100 Subject: [PATCH 07/19] docs(plans): an absent old-profile clause reads 'not measured', not '(none)' '(none)' read as 'no leakage'. The cell says what is true of the old test: it did not measure the clause, with the reason beside it. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- ...026-10-25-json-tree-leakage-before-and-after.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index d73b38edd..caf9f8690 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -63,32 +63,32 @@ The test was run unchanged on the merge of #1363 into #1362 (`848940081`). The f The "before" column is the profile at `9a7ca372c`; the "after" column is the port at `839c0f6fd` (with `fa26cb540`, which only speeds one distinguisher up). A clause is a field of the profile's output, or a switch in `Clauses`. -A "(none)" in the before column means the old profile had no such clause, for one of two reasons, and the row says which: *no such thing* (the old format had nothing to state: `x`, the hit rule as a rule, probes) or *a gap in the old profile* (the old format disclosed it and the old test did not model it, so no distinguisher could see it: string folding, whose corpus was lower-case ASCII, and the budget, which the old `json_tree_subtree` also had). The second kind is the method's limit: the calibration proves the page accounts for everything the distinguishers measure, not everything the database sees, and a clause both omit is invisible to it. Each new distinguisher is a way of finding one. +"Not measured" in the before column means the old profile had no such clause, for one of two reasons, and the row says which: the old format had no such thing to state (`x`, the hit rule as a rule, probes), or the old format disclosed it and the old test did not model it, so no distinguisher could see it (string folding, whose corpus was lower-case ASCII, and the budget, which the old `json_tree_subtree` also had). The second kind is the method's limit: the calibration proves the page accounts for everything the distinguishers measure, not everything the database sees, and a clause both omit is invisible to it. Each new distinguisher is a way of finding one. | clause | before (`9a7ca372c`) | after (port) | what changed and why | |---|---|---|---| | `EntryLeakage::selector` | which paths, how many nodes at each, equal across rows, elements share their array's | same | unchanged | | `EntryLeakage::parent` | each node's parent, used to place the chain | same | unchanged | | `EntryLeakage::step` | `Root`, `Member`, `Element(i)`: decides the ordinal, and in the old chain the token too | `Root`, `Member`, `Element(i)`: decides the ordinal (the `o` of the parent tag) only | the token rule moved to `rule` | -| `EntryLeakage::rule` | (none: no such thing) | `Member` or `Element` for every entry that chains, `None` for the root and sealed-root top-level entries | new: the database tests each reached node against the member rule then the element rule, and learns which. Beyond the page as #1363 published it; this branch adds it | +| `EntryLeakage::rule` | not measured (the old format had no such thing) | `Member` or `Element` for every entry that chains, `None` for the root and sealed-root top-level entries | new: the database tests each reached node against the member rule then the element rule, and learns which. Beyond the page as #1363 published it; this branch adds it | | `EntryLeakage::head` | the head token an entry is (top-level fields under the sealed root, the root otherwise) | removed | a head is now named only inside a token name (`TokenName::head`); nothing at rest refers to it | | `EntryLeakage::has_parent_tag` | absent on the root and, under the sealed root, the top-level fields | same | unchanged | -| `EntryLeakage::self_tag` | (none: no such thing) | a fresh opaque name per entry | new: `x` on every entry. A fresh name per entry states that it is distinct within a row and unlinkable across rows; where a sent token links an entry the simulator writes its real self tag instead | +| `EntryLeakage::self_tag` | not measured (the old format had no such thing) | a fresh opaque name per entry | new: `x` on every entry. A fresh name per entry states that it is distinct within a row and unlinkable across rows; where a sent token links an entry the simulator writes its real self tag instead | | `EntryLeakage::ciphertext_len` | AEAD overhead (29 bytes) + kind, child count, label, scalar | same | unchanged: `x` sits beside the ciphertext. It adds a fixed 12 base64 characters per entry, 19 with its key in compact JSON (`,"x":"…"`), 21 in PostgreSQL's `jsonb` text (`, "x": "…"`); #1363 measured +9.8% on a nested document and +9.4% on a flat one | | `EntryLeakage::ordering` / `OrderingLeak::class` | equal values share a class, ordered by `ordering_ranks`; strings ordered by their bytes | equal values share a class; strings ordered by their folded encoding (Unicode 16 NFC, case and accents folded) | the code always folded; the old corpus held only lower-case ASCII, where the two agree. The page now says "folded", and the corpus holds `"Carol"`/`"carol"` and `"ál"`/`"al"` | | `OrderingLeak::len` | **beyond the page**: 65 bytes for a number, `8 × (len + 1) + 1` for a string | on the page: 65 for a number, `8 × (n + 1) + 1` for a string of folded length `n` | #1363's page states it; it now counts the folded length | | `Clauses::ordering_term_length` | on/off switch for the beyond-the-page length | removed | on the page now; its mutant (`ordering_term_length_constant`) still checks it | | `Clauses::key_order` | a reading: UTF-8 byte order vs jsonb's length-first order | same | the page now says "byte order of their key names' UTF-8 encodings"; the reading test stays to keep that sentence load-bearing | -| `Clauses::string_order` | (none: a gap in the old profile) | a reading: folded vs raw bytes | new, with a test that the raw-bytes reading is told apart | -| `Clauses::hit_rule` | (none: no such thing) | on/off switch for the member-or-element rule | new, the one clause beyond the page as #1363 published it | +| `Clauses::string_order` | not measured (the old format had it; the old profile did not model it) | a reading: folded vs raw bytes | new, with a test that the raw-bytes reading is told apart | +| `Clauses::hit_rule` | not measured (the old format had no such thing) | on/off switch for the member-or-element rule | new, the one clause beyond the page as #1363 published it | | `RowLeakage::{sealed_root, keyset, data_key, entries, projection}` | header flag, keyset, data key unique to the row, entries in depth-first order with members in UTF-8 key order, projection `q` | same | unchanged. `data_key` also names the `iv` that salts every `p` and `x` in the row | | `GroupLeakage::{pairs, elements}` | `{s, vs}` pairs in walk order, nested by array element | same | unchanged | | `QueryLeakage::Extract` | the selector of the path asked | same | unchanged | | `QueryLeakage::Subtree` | `token: TokenName{head, positions}` (one head token, chained through leading array positions), `gates` (`Selector(s)` or `Slot(n, s)`, positions in the clear), `reach` (every descendant of the token's node, in every row), `named` | `reads: Vec` and `budget` | rewritten. See the next three rows | | `ReadLeak::token` | (one per read, see above) | `TokenName{head, steps}`, each step `Member(s)` or `Element(s, i)`: one per concrete path, in the order the client sends them | the list is what the database receives; its length gives the bound away, its order places each token | -| `ReadLeak::probe` | (none: no such thing) | whether the token is a probe | new; the database is not told, and can tell (see findings) | +| `ReadLeak::probe` | not measured (the old format had no such thing) | whether the token is a probe | new; the database is not told, and can tell (see findings) | | `ReadLeak::hits` / `ReadLeak::reach` | `reach`: descendants of the head's node; `named`: what the gates reached | `hits`: the entry the token names in each row, if any; `reach`: every descendant of it that chains from it, in each row | a token now names exactly its node; nothing above it. Under the sealed root the root's token reaches nothing below the root | -| `QueryLeakage::Subtree::budget` | (none: a gap in the old profile; the old read had a budget too) | the `budget` argument | new; the fallback to the root entry is computed from it and the walk, so the database learns nothing new from it | +| `QueryLeakage::Subtree::budget` | not measured (the old read had a budget too; the old profile did not model it) | the `budget` argument | new; the fallback to the root entry is computed from it and the walk, so the database learns nothing new from it | | `QueryLeakage::{Contains, Equals, Order}` | needle projection with `alt`; equality needle; selector and the bound's ordering term | same | unchanged | | `Leakage::{key_tag_len, ordering_ranks}` | the key tag's length, the order of every ordering class | same | unchanged | From 75096006248aadb3dc3dd0d80582565048ddf7a7 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:45:25 +1100 Subject: [PATCH 08/19] test(stack-encrypt): hold the ordering terms' shared prefixes in the leakage profile A CLLW OPE term is a carry byte, then one byte per plaintext bit: the PRF of the bits before it, plus the bit at weight 128, plus a carry of zero or one from the right. Two terms therefore agree to within one at every byte before the first bit their encodings differ in, and differ by 127 to 129 there. The database reads the exact common-prefix length of every pair of values off the stored terms; for numbers that is their signs and binary exponents. The page states the disclosure ("the term also shows how long that common prefix is"), but the profile listed it as out of reach on the ground that it was not an exact function of the plaintexts. That holds for the byte prefix, which carries move, and not for the bit prefix. Its simulator drew terms with no prefix structure at all, so a profile that left it out passed. The profile now states the bit LCP of every two classes adjacent in rank (Leakage::ordering_prefixes, from vitaminc_ore's encodings). The simulator builds CLLW-shaped terms under its own key over codes that have exactly those prefixes, and ordering_term_prefixes reads them. The mutant ordering_prefixes_hidden is caught by that distinguisher alone. The page's sentence says bits, and what they mean for a number and a string. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql/docs/reference/json-tree-leakage.md | 2 +- .../stack-encrypt/tests/json_leakage/chain.rs | 34 +++++- .../tests/json_leakage/distinguishers.rs | 42 +++++++ .../stack-encrypt/tests/json_leakage/mod.rs | 38 ++++-- .../tests/json_leakage/mutants.rs | 8 ++ .../tests/json_leakage/profile.rs | 68 ++++++++++- .../tests/json_leakage/simulator.rs | 114 ++++++++++++++---- 7 files changed, 272 insertions(+), 34 deletions(-) diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index af1f8b22f..f5226d06f 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -22,7 +22,7 @@ Someone who holds the column and no key, such as a database administrator, a bac | `p`, the parent tag below the chain's heads | Nothing beyond the entry order on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes, and it does not say which entry is the parent. Which entries carry no `p` does show: with the sealed root on, those are the root's direct members, and since everything beneath a member follows it, **the node count of each top-level field's subtree** can be read off a single row exactly. With the sealed root off, only the root lacks one. | | `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | -| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bytes (for a string, its first characters) get terms that share leading bytes, so the term also shows how long that common prefix is. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | +| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | | `h`, the header | The keyset the data key was minted under, the data key's identifier (unique to the row), and whether the root is sealed. `v` and `i` are the version and the table and column identifier every EQL value carries. | Not revealed at rest: any value's or key name's text, and, except through the ordering terms at the paths the index orders, whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the byte order of sibling key names. diff --git a/packages/stack-encrypt/tests/json_leakage/chain.rs b/packages/stack-encrypt/tests/json_leakage/chain.rs index 668bd013b..fabfdeb27 100644 --- a/packages/stack-encrypt/tests/json_leakage/chain.rs +++ b/packages/stack-encrypt/tests/json_leakage/chain.rs @@ -4,9 +4,11 @@ //! a stored row's `iv`, a token it was sent and the selectors it holds, and //! the simulator programs its random tags with them. //! -//! These are the only functions of the real scheme the harness evaluates. -//! Everything keyed (selectors, value selectors, head tokens, member ranks) -//! stays on the client and is replaced by randomness in the simulator. +//! These are the only functions of the real scheme the harness evaluates, +//! with the layout of a CLLW OPE term ([`cllw_ope`]), which the simulator +//! fills under a key of its own. Everything keyed (selectors, value +//! selectors, head tokens, member ranks, the OPE key) stays on the client +//! and is replaced by randomness in the simulator. //! `chain_is_the_crates_chain` (in `calibration.rs`) holds them to the //! crate's own walk, `JsonDocument::subtree`. @@ -88,6 +90,32 @@ pub fn hit_token( (self_tag(iv, &element) == *x).then_some((element, Rule::Element)) } +/// A CLLW OPE term over `bits` under `key`, with the structure +/// `cllw_ore::encrypt_ope_bits` gives it: a leading carry byte, then one +/// byte per bit, a keyed function of the bits before it plus the bit at +/// weight 128, the sums carried from right to left. The keyed function is +/// a stand-in (the real one is a keyed BLAKE3 under the column's key, which +/// the database never sees); the layout is public, and it is what makes two +/// terms agree to within one at every byte before the first bit their +/// plaintexts differ in. +pub fn cllw_ope(key: &[u8; 32], bits: &[bool]) -> Vec { + let mut out = Vec::with_capacity(bits.len() + 1); + out.push(0); + let mut prefix: Vec = Vec::with_capacity(bits.len()); + for &bit in bits { + out.push(sha256(&[b"ope", key, &prefix])[0]); + prefix.push(u8::from(bit)); + } + let mut carry = 0u16; + for (byte, &bit) in out[1..].iter_mut().rev().zip(bits.iter().rev()) { + let sum = u16::from(*byte) + (u16::from(bit) << 7) + carry; + *byte = sum as u8; + carry = sum >> 8; + } + out[0] = carry as u8; + out +} + /// The simulator's stand-in for the keyed member rank: any keyed function of /// the parent's token and the member's selector will do, since the real one /// is a PRF under a key the database never sees. The simulator samples its diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 35d7c97ea..2f06200e0 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -75,6 +75,7 @@ registry!( extraction_matches, containment_matches, ordering_term_order, + ordering_term_prefixes, ordering_term_lengths, ordering_matches, ); @@ -1251,6 +1252,47 @@ pub fn ordering_term_order(world: &ObservedWorld) -> (Vec>>, V ) } +/// For every two distinct ordering terms in the world, stored and sent, by +/// their ranks in byte order, how many leading plaintext bits their values' +/// encodings share, read off the terms alone. A CLLW OPE term is a carry +/// byte, then one byte per plaintext bit: a keyed function of the bits +/// before it, plus the bit at weight 128, plus a carry of zero or one from +/// the byte to its right. So two terms differ by at most one, modulo 256, at +/// every byte before the first bit their plaintexts differ in, and by 127 to +/// 129 at that byte; the count is the bytes before it (after the carry +/// byte), or the shorter term's bit count when one encoding is a prefix of +/// the other. Exact: no carry can make a shared bit's bytes differ by more +/// than one, and none can bring a differing bit's within one. The page: the +/// term "shows how long that common prefix is". Of a number, the first bit +/// is its sign and the next eleven its exponent's, so this sorts numbers +/// into signs and binary orders of magnitude without any bound sent. +pub fn ordering_term_prefixes(world: &ObservedWorld) -> Vec<(usize, usize, usize)> { + let mut all: Vec<&[u8]> = world + .rows + .iter() + .flat_map(|r| r.entries.iter().filter_map(|e| e.op.as_deref())) + .collect(); + all.extend(world.queries.iter().filter_map(|q| match q { + ObservedQuery::Order { op, .. } => Some(op.as_slice()), + _ => None, + })); + all.sort_unstable(); + all.dedup(); + let shared = |a: &[u8], b: &[u8]| { + let bits = a.len().min(b.len()).saturating_sub(1); + (1..=bits) + .find(|&k| !matches!(a[k].wrapping_sub(b[k]), 0 | 1 | 255)) + .map_or(bits, |k| k - 1) + }; + let mut out = Vec::new(); + for (i, a) in all.iter().enumerate() { + for (j, b) in all.iter().enumerate().skip(i + 1) { + out.push((i, j, shared(a, b))); + } + } + out +} + /// Every ordering term's length, stored and sent. #[allow(clippy::type_complexity)] pub fn ordering_term_lengths(world: &ObservedWorld) -> (Vec>>, Vec) { diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 1212ae806..048565456 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -38,7 +38,9 @@ //! - [`simulator`] builds a world in the wire form from a //! [`profile::Leakage`] and randomness, with no plaintext and no key: //! random bytes for every name, random ciphertexts of the stated length, -//! ordering terms laid out in the stated order, and the chain programmed +//! ordering terms with CLLW's layout under a key of its own, over codes +//! that have the stated order, lengths and shared prefixes and nothing +//! else of the values, and the chain programmed //! with the real hash where the profile says a query's token links it: //! the self tag `x` of each node a token names, and below it every parent //! tag and self tag the token reaches. @@ -129,6 +131,18 @@ //! their bytes, where the code (and the page) fold case and accents; the //! corpus now holds folded names, and the reading test holds the sentence. //! +//! The profile also left out, until review, a disclosure the page states: +//! the ordering terms' shared prefixes. A CLLW OPE term spends a byte per +//! plaintext bit, so two terms agree to within one up to the first bit their +//! values' encodings differ in, and the database reads that bit off the +//! stored terms exactly; for numbers it is their signs and binary exponents. +//! The profile had called it "not an exact function of the plaintexts" +//! (true of the byte prefix, which carries move, and not of the bit), and +//! its simulator drew terms with no prefix structure, so nothing could see +//! the omission. It is a clause now ([`profile::Leakage::ordering_prefixes`], +//! [`distinguishers::ordering_term_prefixes`], the mutant +//! `ordering_prefixes_hidden`), and the page says bits, not bytes. +//! //! Where the page is vague and `L` is as precise as the code forces, the //! clause says so: a member's ordinal under a token (its keyed rank), the //! needle's `alt` ([`profile::QueryLeakage::Contains`]), and the fixed @@ -195,12 +209,22 @@ //! //! # Out of reach //! -//! - **The inside of an ordering term.** CLLW OPE terms share leading bytes -//! when the encoded values do (the page says so), with a carry that can -//! move a byte by one. The simulator reproduces a term's presence, length, -//! equality and order, not its prefix structure, and no distinguisher -//! looks at it: the prefix a pair of terms shares is not an exact function -//! of the plaintexts, so it is a statistical question. +//! - **The carries inside an ordering term.** A CLLW OPE term's bytes before +//! the first bit two values differ in agree to within one, a carry from +//! the right that depends on the key. Where the terms part is exact and +//! is in the profile ([`profile::Leakage::ordering_prefixes`], +//! [`distinguishers::ordering_term_prefixes`]); which shared bytes differ +//! by one is not a function of the plaintexts, and no distinguisher reads +//! it. +//! - **A path ordered over numbers and strings both.** The profile refuses +//! such a column ([`profile::Ids::ordering_ranks`]), and the page does not +//! say what it shows. Measured on this crate: a number's term and a +//! string's are under different keys (the order space's salt names its +//! domain), so how one compares with the other is set by the column's key +//! and the two values' leading bits, not by any order between a number +//! and a string. In the column measured every string sorted below every +//! number, and a bound of either kind compares with the stored terms of +//! both. Stating it needs the page to say it first. //! - **The keyed member rank as a permutation.** Under a token, a member's //! ordinal is a keyed rank. Its relation to the stored (key) order is a //! random permutation, decidable exactly only where enough wide objects sit diff --git a/packages/stack-encrypt/tests/json_leakage/mutants.rs b/packages/stack-encrypt/tests/json_leakage/mutants.rs index dd056223d..1ed19bbe9 100644 --- a/packages/stack-encrypt/tests/json_leakage/mutants.rs +++ b/packages/stack-encrypt/tests/json_leakage/mutants.rs @@ -423,6 +423,13 @@ fn ordering_term_length_constant(l: &mut Leakage, _: &mut TestRng) { } } +/// The terms show their order and lengths and no shared prefix beyond what +/// those force: what the page would say without "the term also shows how +/// long that common prefix is". +fn ordering_prefixes_hidden(l: &mut Leakage, _: &mut TestRng) { + l.ordering_prefixes.clear(); +} + /// A member's ordinal is its place in the stored (key) order rather than a /// keyed rank: what the chain would be without decision 11. fn member_ranks_follow_key_order(l: &mut Leakage, _: &mut TestRng) { @@ -840,6 +847,7 @@ mutants!( ordering_terms_all_distinct => "OrderingLeak::class (\"which rows hold equal values\")", ordering_term_length_constant => "OrderingLeak::len (\"Its length is the leaf's length\")", ordering_terms_per_path => "OrderingLeak::class across paths (\"two ordered paths holding equal values carry equal terms\")", + ordering_prefixes_hidden => "Leakage::ordering_prefixes (\"the term also shows how long that common prefix is\")", member_ranks_follow_key_order => "EntryLeakage::step (a member's ordinal is its keyed rank)", hit_rule_uniform => "EntryLeakage::rule (the member or element rule a walk's hit matches; added to the page by #1362)", sealed_flag_flipped => "RowLeakage::sealed_root", diff --git a/packages/stack-encrypt/tests/json_leakage/profile.rs b/packages/stack-encrypt/tests/json_leakage/profile.rs index 94b525a49..f071fa461 100644 --- a/packages/stack-encrypt/tests/json_leakage/profile.rs +++ b/packages/stack-encrypt/tests/json_leakage/profile.rs @@ -29,7 +29,7 @@ //! `docs/plans/2026-10-25-json-tree-leakage-before-and-after.md` records //! what changed between the two. -use std::collections::HashMap; +use std::collections::{BTreeMap, HashMap}; use std::hash::Hash; use serde_json::{Number, Value}; @@ -406,6 +406,21 @@ pub struct Leakage { /// two ordered paths compare with each other too, and two ordered paths /// holding equal values carry equal terms"). pub ordering_ranks: Vec, + /// For each two classes next to each other in rank order, `(lower, + /// higher)`, how many leading bits their order encodings share: "two + /// values whose encodings share leading bits […] get terms that share + /// leading bytes, […] so the term also shows how long that common + /// prefix is". CLLW OPE writes one byte per bit, the PRF of the bits + /// before it plus the bit at weight 128, with a carry from the right, so + /// two terms agree to within one at every byte before the first bit + /// their encodings differ in, and differ by about 128 there + /// (`distinguishers::ordering_term_prefixes`). The common prefix of any + /// two classes is the least of the adjacent ones between them, so the + /// adjacent pairs are the whole clause. For a number the encoding is its + /// `f64`'s order encoding, so its first bit is the sign and the next + /// eleven the exponent's; for a string, the marker byte and then the + /// folded text. + pub ordering_prefixes: BTreeMap<(OrderClass, OrderClass), usize>, } // ============================================================================= @@ -439,6 +454,36 @@ enum OrderKey { String(Vec), } +impl OrderKey { + /// The bytes the term is CLLW OPE over, one term byte per bit: a + /// number's `f64` order encoding (`vitaminc_ore::order` at `F64`), and a + /// string's marker and folded text as [`Ids::ordering`] keeps them. + fn encoding(&self) -> Vec { + match self { + Self::Number(bits) => vitaminc_ore::order( + vitaminc_ore::OrderTarget::F64, + None, + vitaminc_ore::Physical::Float(vitaminc_ore::Float::F64(f64::from_bits(*bits))), + ) + .expect("a finite f64 has an order encoding") + .bytes() + .risky_ref() + .clone(), + Self::String(encoded) => encoded.clone(), + } + } +} + +/// How many leading bits `a` and `b` share. +fn common_bits(a: &[u8], b: &[u8]) -> usize { + a.iter() + .zip(b) + .position(|(x, y)| x != y) + .map_or(8 * a.len().min(b.len()), |k| { + 8 * k + (a[k] ^ b[k]).leading_zeros() as usize + }) +} + /// The names shared by [`l_store`] and [`l_query`]: one id space per kind of /// keyed value, so a query's selector is the stored selector's id when the /// real bytes are equal. @@ -554,6 +599,23 @@ impl Ids { } ranks } + + /// The leading bits each two classes next to each other in `ranks`' + /// order share ([`Leakage::ordering_prefixes`]). + pub fn ordering_prefixes(&self, ranks: &[usize]) -> BTreeMap<(OrderClass, OrderClass), usize> { + let mut by_rank: Vec = (0..ranks.len()).collect(); + by_rank.sort_by_key(|&class| ranks[class]); + by_rank + .windows(2) + .map(|pair| { + let shared = common_bits( + &self.ordering_keys[pair[0]].encoding(), + &self.ordering_keys[pair[1]].encoding(), + ); + ((pair[0], pair[1]), shared) + }) + .collect() + } } // ============================================================================= @@ -1052,10 +1114,12 @@ pub fn leak(column: &Column, clauses: Clauses, docs: &[Value], queries: &[Query] .iter() .map(|q| l_query(column, &mut ids, docs, q)) .collect(); + let ordering_ranks = ids.ordering_ranks(); Leakage { key_tag_len: KEY_TAG_LEN, rows, queries, - ordering_ranks: ids.ordering_ranks(), + ordering_prefixes: ids.ordering_prefixes(&ordering_ranks), + ordering_ranks, } } diff --git a/packages/stack-encrypt/tests/json_leakage/simulator.rs b/packages/stack-encrypt/tests/json_leakage/simulator.rs index e28e8051c..863a7cf5a 100644 --- a/packages/stack-encrypt/tests/json_leakage/simulator.rs +++ b/packages/stack-encrypt/tests/json_leakage/simulator.rs @@ -7,8 +7,10 @@ //! [`HeadId`](super::profile::HeadId), a self tag per //! [`SelfTagId`](super::profile::SelfTagId)), so equality patterns are //! exactly the profile's. Ciphertexts are random bytes of the stated length -//! behind the frame's public version byte. Ordering terms are random bytes -//! laid out so that byte order is the profile's rank order. +//! behind the frame's public version byte. Ordering terms have CLLW OPE's +//! public layout under a key the simulator draws, over codes built from the +//! profile's ranks, lengths and shared prefixes, so they compare, repeat +//! and share leading bytes exactly as the profile says. //! //! The chain is the one place the simulator evaluates the real scheme's //! public functions ([`super::chain`]): a token the database is sent has to @@ -25,7 +27,7 @@ //! Every other parent and self tag is random bytes, as a tag under a token //! the database never sees is to it. -use std::collections::HashMap; +use std::collections::{BTreeMap, HashMap}; use base64ct::{Base64, Encoding}; use proptest::prelude::RngCore; @@ -33,10 +35,12 @@ use proptest::test_runner::TestRng; use serde_json::{json, Value}; use super::chain::{ - element_token, member_token, parent_tag, self_tag, simulated_rank, Rule, Tag16, Tag8, + cllw_ope, element_token, member_token, parent_tag, self_tag, simulated_rank, Rule, Tag16, Tag8, }; use super::observe::QueryKind; -use super::profile::{GroupLeakage, Leakage, QueryLeakage, RowLeakage, Step, TokenName, TokenStep}; +use super::profile::{ + GroupLeakage, Leakage, OrderingLeak, QueryLeakage, RowLeakage, Step, TokenName, TokenStep, +}; /// The version byte `vitaminc-encrypt` writes at the front of every /// ciphertext: public framing, not leakage. @@ -100,6 +104,7 @@ struct Simulator<'a> { /// `rng` alone. pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(QueryKind, Value)>) { let rank_key = random(rng); + let ordering = ordering_terms(leakage, &random(rng)); let mut sim = Simulator { leakage, rng, @@ -110,7 +115,7 @@ pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(Query ivs: Names::new(), self_tags: Names::new(), tags: HashMap::new(), - ordering: HashMap::new(), + ordering, rank_key, }; let linked = sim.link(); @@ -124,6 +129,82 @@ pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(Query (rows, queries) } +/// Every ordering class's term: a CLLW OPE term ([`cllw_ope`]) under `key` +/// over a code of the class's bit length, the codes built in rank order from +/// the profile's shared prefixes alone ([`successor`]). So the terms +/// compare as the ranks do, are equal where the classes are, are as long as +/// the profile says, and share leading bytes exactly as far as the profile +/// says the values' encodings share leading bits. +fn ordering_terms(leakage: &Leakage, key: &[u8; 32]) -> HashMap> { + let mut bits: BTreeMap = BTreeMap::new(); + let mut note = |op: &OrderingLeak| { + let _ = bits.insert(op.class, op.len - 1); + }; + for row in &leakage.rows { + for op in row.entries.iter().filter_map(|e| e.ordering.as_ref()) { + note(op); + } + } + for query in &leakage.queries { + if let QueryLeakage::Order { bound, .. } = query { + note(bound); + } + } + let mut classes: Vec = bits.keys().copied().collect(); + classes.sort_by_key(|&class| (leakage.ordering_ranks[class], class)); + let mut terms = HashMap::new(); + let mut previous: Option<(usize, Vec)> = None; + for class in classes { + let n = bits[&class]; + let code = match &previous { + None => vec![false; n], + Some((below, code)) => successor( + code, + n, + leakage.ordering_prefixes.get(&(*below, class)).copied(), + ), + }; + let _ = terms.insert(class, cllw_ope(key, &code)); + previous = Some((class, code)); + } + terms +} + +/// The least `n`-bit code above `below` that shares `shared` leading bits +/// with it. Codes built this way, in rank order from all zeros, share +/// exactly the prefixes the profile states, which is what any set of real +/// encodings does: at the bit where two neighbours part, the lower one has a +/// zero, which is where the code puts its one. A profile that states no +/// prefix (`None`, a mutant's) gets the latest branch that still sorts +/// above `below`; one that states an impossible prefix gets the nearest +/// branch before it, so a mutant's world still builds. +fn successor(below: &[bool], n: usize, shared: Option) -> Vec { + let m = below.len(); + let extends = match shared { + Some(l) => l >= m && m < n, + None => m < n, + }; + if extends { + let mut code = below.to_vec(); + code.resize(n, false); + return code; + } + let last = shared + .unwrap_or(n) + .min(n.saturating_sub(1)) + .min(m.saturating_sub(1)); + let mut code: Vec = match (0..=last).rev().find(|&i| !below[i]) { + Some(at) => { + let mut code = below[..at].to_vec(); + code.push(true); + code + } + None => below[..n.min(m)].to_vec(), + }; + code.resize(n, false); + code +} + impl Simulator<'_> { fn selector(&mut self, id: usize) -> Tag16 { self.selectors.get(id, self.rng) @@ -150,21 +231,12 @@ impl Simulator<'_> { t } - /// An ordering term of `len` bytes for class `class`: a zero byte, the - /// class's rank big-endian, then random bytes, so byte order is rank - /// order and equal classes are equal terms. - fn ordering(&mut self, class: usize, len: usize) -> Vec { - if let Some(bytes) = self.ordering.get(&class) { - assert_eq!(bytes.len(), len, "one class, one length"); - return bytes.clone(); - } - assert!(len >= 5, "a term is long enough to carry its rank"); - let rank = u32::try_from(self.leakage.ordering_ranks[class]).expect("a small world"); - let mut bytes = vec![0]; - bytes.extend_from_slice(&rank.to_be_bytes()); - bytes.extend(random_vec(self.rng, len - 5)); - let _ = self.ordering.insert(class, bytes.clone()); - bytes + /// The ordering term of class `class`, `len` bytes long: built once for + /// every class by [`ordering_terms`]. + fn ordering(&self, class: usize, len: usize) -> Vec { + let bytes = &self.ordering[&class]; + assert_eq!(bytes.len(), len, "one class, one length"); + bytes.clone() } fn group(&mut self, group: &GroupLeakage) -> Value { From 076a9212e810966ee11ad7d614225765f8eea662 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:46:43 +1100 Subject: [PATCH 09/19] test(stack-encrypt): compare the leakage harness's values across kinds Every feature named selectors, value selectors, tokens, keysets and ivs in a space of its own, and self_tag_equalities compared x only with x and p. A token equal to its path's selector, a value selector equal to a selector, or a self tag equal to the front of a token would be visible to the database, and no feature would change. cross_kind_equalities reports every pair of kinds that share a value: the sixteen-byte values with each other, and the eight-byte tags against the first eight bytes of each. It is empty on the real scheme and in the simulator. No mutant can state such a world, since the profile names each kind on its own, so a harness test shows it fires: replacing one read's token with a stored selector gives exactly ("s", "token"). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/calibration.rs | 36 +++++++++- .../tests/json_leakage/distinguishers.rs | 71 +++++++++++++++++++ 2 files changed, 106 insertions(+), 1 deletion(-) diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index 65ac041c1..e2f4cb260 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -11,7 +11,7 @@ use stack_kms::FakeDataKeySource; use super::corpus::{handcrafted, random_world, World}; use super::distinguishers::{self, walk_positions}; use super::mutants; -use super::observe::ObservedWorld; +use super::observe::{ObservedQuery, ObservedWorld}; use super::profile::{flatten, leak, Clauses, Column, KeyOrder, Leakage, Query, StringOrder}; use super::{real, simulator}; use crate::common::stack_cipher; @@ -462,6 +462,40 @@ mod the_harness { assert_eq!(price[2], (most, true, Some(true))); } + /// `distinguishers::cross_kind_equalities` can fire: it is empty on the + /// real scheme, and a world in which one read's token is a stored + /// selector, which every feature naming each kind on its own would miss, + /// is told apart by it. No mutant can state that world, since the + /// profile names each kind in a space of its own; this is the clause's + /// check. + #[test] + fn cross_kind_equalities_see_a_token_that_is_a_selector() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let world = &handcrafted()[0]; + let mut observed = real_world(&runtime, &cipher, world); + assert!( + distinguishers::cross_kind_equalities(&observed).is_empty(), + "no value is shared across kinds in the real scheme: {:?}", + distinguishers::cross_kind_equalities(&observed) + ); + let selector = observed.rows[0].entries[1].s; + let read = observed + .queries + .iter_mut() + .find_map(|q| match q { + ObservedQuery::Subtree { tokens, .. } => Some(tokens), + _ => None, + }) + .expect("the corpus sends a subtree read"); + read[0] = selector; + assert_eq!( + distinguishers::cross_kind_equalities(&observed), + [("s", "token")].into_iter().collect(), + "a token equal to a stored selector is seen" + ); + } + /// The page's claim that the stored order reveals each row's shape holds /// on a column whose rows vary: the order-only attack of /// `distinguishers::depth_recovery` recovers every row's depth sequence diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 2f06200e0..0c83fca0e 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -54,6 +54,7 @@ registry!( selector_pattern_across_rows, parent_tag_equalities, self_tag_equalities, + cross_kind_equalities, headers, projections, value_selector_partitions, @@ -291,6 +292,76 @@ pub fn self_tag_equalities(world: &ObservedWorld) -> Vec<(TagAt, TagAt)> { out } +/// Which kinds of value share a value with another kind, anywhere in the +/// world: the sixteen-byte ones (a selector `s`, stored or sent, a value +/// selector `vs`, a token, a keyset, an `iv`) with each other, and an +/// eight-byte tag (`x`, `p`) with the first eight bytes of any of them. +/// Every other feature names each kind in its own space, so a token equal +/// to its path's selector, or a value selector equal to a selector, would +/// leave them all unchanged; the database would see it. Expected empty: +/// each kind is a keyed hash or a PRF output in a domain of its own, or +/// random. Equalities within a kind are the other features' business, and +/// `self_tag_equalities` compares `x` with `p`. +pub fn cross_kind_equalities(world: &ObservedWorld) -> BTreeSet<(&'static str, &'static str)> { + let mut wide: HashMap> = HashMap::new(); + let mut add = |kind: &'static str, value: Tag16| { + let _ = wide.entry(value).or_default().insert(kind); + }; + fn add_group(group: &ObservedGroup, add: &mut dyn FnMut(&'static str, Tag16)) { + let mut pairs = Vec::new(); + pairs_of(group, &mut pairs); + for (s, vs) in pairs { + add("s", s); + add("vs", vs); + } + } + for row in &world.rows { + add("keyset", row.header.keyset); + add("iv", row.header.iv); + for entry in &row.entries { + add("s", entry.s); + } + add_group(&row.q, &mut add); + } + for query in &world.queries { + match query { + ObservedQuery::Extract { s } | ObservedQuery::Order { s, .. } => add("s", *s), + ObservedQuery::Subtree { tokens, .. } => tokens.iter().for_each(|t| add("token", *t)), + ObservedQuery::Contains { needle, alt } => { + add_group(needle, &mut add); + if let Some(alt) = alt { + add_group(alt, &mut add); + } + } + ObservedQuery::Equals { needle } => add_group(needle, &mut add), + } + } + let mut out = BTreeSet::new(); + let mut fronts: HashMap> = HashMap::new(); + for (value, kinds) in &wide { + for a in kinds { + for b in kinds { + if a < b { + let _ = out.insert((*a, *b)); + } + } + } + let mut front = Tag8::default(); + front.copy_from_slice(&value[..8]); + fronts.entry(front).or_default().extend(kinds); + } + for row in &world.rows { + for entry in &row.entries { + for (kind, tag) in [("x", entry.x), ("p", entry.p)] { + for other in tag.and_then(|t| fronts.get(&t)).into_iter().flatten() { + let _ = out.insert((kind, *other)); + } + } + } + } + out +} + /// Each header's public fields, and the equality patterns of its keyset, /// `iv` and key tag across rows. Catches a shared `iv` (which would link /// every row's parent tags) and a keyset that varies within a column. From e52fe64f30d6b88773464fd4bb7f435fa3718a40 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:47:35 +1100 Subject: [PATCH 10/19] test(stack-encrypt): check the wire field names of every query term and projection wire_fields covered the keys of rows and entries only. The parser reads t, budget, sv, el, alt, s and op out of a query term and the projection, and drops anything else unread, so a term that also carried its bound or its path would have left every feature unchanged. term_fields reports every key path inside each row's q and each query term as sent. needle_alt_dropped, needle_flat and projection_flat are now caught by it as well as by their own features. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/distinguishers.rs | 12 ++++++ .../tests/json_leakage/observe.rs | 37 +++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 0c83fca0e..c77678ff8 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -43,6 +43,7 @@ macro_rules! registry { registry!( wire_fields, + term_fields, entry_counts, parent_tag_presence, ordering_term_presence, @@ -130,6 +131,17 @@ pub fn wire_fields(world: &ObservedWorld) -> Vec<(Vec, Vec>) .collect() } +/// Every key path inside each row's projection `q` and inside each query +/// term as it is sent: catches a term field the profile does not account +/// for, which the parser (`observe`) would otherwise drop unread, as +/// [`wire_fields`] does for the rows and entries. +pub fn term_fields(world: &ObservedWorld) -> (Vec>, Vec>) { + ( + world.rows.iter().map(|r| r.q_fields.clone()).collect(), + world.query_fields.clone(), + ) +} + /// How many entries each row has: the node count. pub fn entry_counts(world: &ObservedWorld) -> Vec { world.rows.iter().map(|r| r.entries.len()).collect() diff --git a/packages/stack-encrypt/tests/json_leakage/observe.rs b/packages/stack-encrypt/tests/json_leakage/observe.rs index 0daf114f4..a5998d82a 100644 --- a/packages/stack-encrypt/tests/json_leakage/observe.rs +++ b/packages/stack-encrypt/tests/json_leakage/observe.rs @@ -62,6 +62,8 @@ pub struct ObservedRow { pub header: ObservedHeader, pub entries: Vec, pub q: ObservedGroup, + /// Every key path inside `q` ([`key_paths`]). + pub q_fields: Vec, } /// Which SQL surface a query term was sent to. The database knows which @@ -110,6 +112,9 @@ pub enum ObservedQuery { pub struct ObservedWorld { pub rows: Vec, pub queries: Vec, + /// Every key path inside each query term's wire form ([`key_paths`]), + /// which the parser above reads only in part. + pub query_fields: Vec>, } impl ObservedWorld { @@ -119,6 +124,7 @@ impl ObservedWorld { Self { rows: rows.iter().map(row).collect(), queries: queries.iter().map(|(kind, q)| query(*kind, q)).collect(), + query_fields: queries.iter().map(|(_, q)| key_paths(q)).collect(), } } } @@ -136,6 +142,36 @@ fn b64_8(text: &Value) -> Tag8 { b64(text).try_into().expect("an eight-byte tag") } +/// Every key path in `value`, sorted: `k` for a key at the top, `k.j` below +/// it, and `k[].j` below an array's elements. What a field the parser does +/// not read would show in. +pub fn key_paths(value: &Value) -> Vec { + fn walk(value: &Value, at: &str, out: &mut std::collections::BTreeSet) { + match value { + Value::Object(map) => { + for (key, child) in map { + let path = if at.is_empty() { + key.clone() + } else { + format!("{at}.{key}") + }; + let _ = out.insert(path.clone()); + walk(child, &path, out); + } + } + Value::Array(items) => { + for item in items { + walk(item, &format!("{at}[]"), out); + } + } + _ => {} + } + } + let mut out = std::collections::BTreeSet::new(); + walk(value, "", &mut out); + out.into_iter().collect() +} + fn keys(value: &Value) -> Vec { value .as_object() @@ -212,6 +248,7 @@ fn row(value: &Value) -> ObservedRow { .map(entry) .collect(), q: group(&value["q"]), + q_fields: key_paths(&value["q"]), } } From 6b384632061b6affab69ff4a29d1fb7f5d344c6c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:49:25 +1100 Subject: [PATCH 11/19] test(stack-encrypt): let token_derivability try the element rule past index 3 The database derives an element's token from its array's at any index. The feature tried indices 0 to 3 only, so a read whose tokens all sat at index 4 or more and found nothing (`$.tags[10]` beside `$.tags`, in a column whose arrays are shorter) read as unlinked. A profile that unchained such a token would have passed it. It now also tries every index up to the longest token list sent or 64, whichever is more, under the selector of the children each found token's walks reach: the one selector an element under it can carry, so the check stays cheap. The ordered-strings column reads `$.tags[10]`, and a harness test holds that the real scheme's tokens link it to `$.tags`. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/calibration.rs | 27 +++++++++++++++ .../tests/json_leakage/corpus.rs | 3 ++ .../tests/json_leakage/distinguishers.rs | 33 +++++++++++++++++-- 3 files changed, 60 insertions(+), 3 deletions(-) diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index e2f4cb260..b4b714764 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -496,6 +496,33 @@ mod the_harness { ); } + /// A token for a position past every row's array finds nothing, and the + /// database still links it to the array's token by the element rule: + /// `token_derivability` tries indices up to the longest token list + /// sent or 64, not only the first few. + #[test] + fn a_position_past_every_array_is_linked_to_its_array() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let world = handcrafted() + .into_iter() + .find(|w| w.name == "ordered strings") + .expect("the corpus has it"); + let at = |path: &str| { + world + .queries + .iter() + .position(|q| matches!(q, Query::Subtree { path: p, .. } if p == path)) + .expect("the column reads it") + }; + let observed = real_world(&runtime, &cipher, &world); + let links = distinguishers::token_derivability(&observed); + assert!( + links.contains(&(at("$.tags"), at("$.tags[10]"))), + "`$.tags[10]`'s token derives from `$.tags`'s: {links:?}" + ); + } + /// The page's claim that the stored order reveals each row's shape holds /// on a column whose rows vary: the order-only attack of /// `distinguishers::depth_recovery` recovers every row's depth sequence diff --git a/packages/stack-encrypt/tests/json_leakage/corpus.rs b/packages/stack-encrypt/tests/json_leakage/corpus.rs index 3c909ecfe..7d91348e5 100644 --- a/packages/stack-encrypt/tests/json_leakage/corpus.rs +++ b/packages/stack-encrypt/tests/json_leakage/corpus.rs @@ -117,6 +117,9 @@ fn names() -> World { Query::subtree("$.tags"), Query::subtree("$.tags[1]"), Query::subtree_with("$.tags[*]", 2, 64), + // Past every row's array: found nowhere, and linked to the + // read of `$.tags` by the element rule alone. + Query::subtree("$.tags[10]"), ], } } diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index c77678ff8..68ec93547 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -1094,8 +1094,10 @@ pub fn member_ordinals_follow_key_order(world: &ObservedWorld) -> Option { /// `(i, j)` when a token sent by query `j` is one the walks from query /// `i`'s tokens derive in some row, or is a child token of one of `i`'s /// tokens that finds a node somewhere, by the member rule under any -/// selector the column holds, or by the element rule at an index below 4 -/// (bounds on what the harness tries, not on what the database can). The page: a token's holder can "derive the tokens of +/// selector the column holds, by the element rule at an index below 4, or +/// by the element rule under the selector of the children its walks reach +/// at any index up to the longest token list sent or 64 (bounds on what the +/// harness tries, not on what the database can). The page: a token's holder can "derive the tokens of /// the nodes beneath". Catches tokens that are not chained (a token /// unrelated to its path's) as well as tokens chained where the profile /// says they are not. @@ -1107,6 +1109,12 @@ pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { .iter() .flat_map(|r| r.entries.iter().map(|e| e.s)) .collect(); + let longest = queries + .iter() + .map(|(_, tokens, _)| u32::try_from(tokens.len()).unwrap_or(u32::MAX)) + .max() + .unwrap_or(0) + .max(64); let mut out = BTreeSet::new(); for (q, &(i, tokens, _)) in queries.iter().enumerate() { let mut derived: BTreeSet = walks[q] @@ -1121,7 +1129,7 @@ pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { .iter() .flat_map(|row| row.iter().map(|(k, _)| *k)) .collect(); - for t in found.into_iter().map(|k| &tokens[k]) { + for t in found.iter().map(|&k| &tokens[k]) { for s in &selectors { let _ = derived.insert(member_token(t, s)); for o in 0..4 { @@ -1129,6 +1137,25 @@ pub fn token_derivability(world: &ObservedWorld) -> BTreeSet<(usize, usize)> { } } } + // And by the element rule at every index up to the longest list + // sent or 64, whichever is more, under the one selector an element of the found node can + // carry: its children's, wherever a walk from it reaches one. This + // links a read of `$.tags[10]` to one of `$.tags` in a column whose + // arrays are all shorter. + for (r, rows) in walks[q].iter().enumerate() { + for (k, hits) in rows { + let kids: BTreeSet = hits + .iter() + .filter(|h| h.depth == 1) + .map(|h| world.rows[r].entries[h.position].s) + .collect(); + for s in &kids { + for o in 0..=longest { + let _ = derived.insert(element_token(&tokens[*k], s, o)); + } + } + } + } for &(j, others, _) in &queries { if i != j && others.iter().any(|t| derived.contains(t)) { let _ = out.insert((i, j)); From 7ca5b0df2c7e5f5b15978ac6b62b8042535455e5 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:50:10 +1100 Subject: [PATCH 12/19] test(stack-encrypt): give the leakage profile's four unchecked clauses a mutant each Four clauses had no mutant, so nothing showed the calibration could see them: the key tag's length, an ordering bound's selector, an element's ordinal being its index (element_tokens_without_index changes the names of tokens sent, not the stored chain), and each node's parent. key_tag_shortened is caught by headers and stored_lengths, order_selector_fresh by transcripts and ordering_matches, element_ordinals_reversed by ordinal_patterns among others, and parents_lifted by token_reach and parent_tag_equalities among others. Also drops mutants.rs's repeated "At rest" banner. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/mutants.rs | 55 +++++++++++++++++-- 1 file changed, 51 insertions(+), 4 deletions(-) diff --git a/packages/stack-encrypt/tests/json_leakage/mutants.rs b/packages/stack-encrypt/tests/json_leakage/mutants.rs index 1ed19bbe9..29d1b5bb9 100644 --- a/packages/stack-encrypt/tests/json_leakage/mutants.rs +++ b/packages/stack-encrypt/tests/json_leakage/mutants.rs @@ -128,10 +128,6 @@ fn children(row: &RowLeakage) -> Vec> { // At rest // ============================================================================= -// ============================================================================= -// At rest -// ============================================================================= - fn selectors_all_distinct(l: &mut Leakage, _: &mut TestRng) { let mut fresh = Fresh::new(); for row in &mut l.rows { @@ -207,6 +203,43 @@ fn entry_count_padded(l: &mut Leakage, _: &mut TestRng) { } } +/// Every node below depth one hangs off its grandparent instead of its +/// parent: the shape a token's walk links is not the document's. +fn parents_lifted(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + let parents: Vec> = row.entries.iter().map(|e| e.parent).collect(); + for entry in &mut row.entries { + if let Some(grandparent) = entry.parent.and_then(|p| parents[p]) { + entry.parent = Some(grandparent); + } + } + } +} + +/// An array's elements sit at their positions reversed under its token, +/// so an element's ordinal is not its index. +fn element_ordinals_reversed(l: &mut Leakage, _: &mut TestRng) { + for row in &mut l.rows { + for siblings in children(row) { + let elements: Vec = siblings + .into_iter() + .filter(|&k| matches!(row.entries[k].step, Step::Element(_))) + .collect(); + let n = elements.len() as u64; + for k in elements { + if let Step::Element(i) = row.entries[k].step { + row.entries[k].step = Step::Element(n - 1 - i); + } + } + } + } +} + +/// A key tag of another length than the key service's. +fn key_tag_shortened(l: &mut Leakage, _: &mut TestRng) { + l.key_tag_len = 16; +} + fn order_random_permutation(l: &mut Leakage, rng: &mut TestRng) { for r in 0..l.rows.len() { let mut order: Vec = (1..l.rows[r].entries.len()).collect(); @@ -532,6 +565,16 @@ fn extraction_selector_fresh(l: &mut Leakage, _: &mut TestRng) { } } +/// An ordering bound's selector unrelated to the stored one of its path. +fn order_selector_fresh(l: &mut Leakage, _: &mut TestRng) { + let mut fresh = Fresh::new(); + for query in &mut l.queries { + if let QueryLeakage::Order { selector, .. } = query { + *selector = fresh.next(); + } + } +} + /// A query that repeats an earlier one sends unrelated terms: queries are /// not deterministic. fn queries_nondeterministic(l: &mut Leakage, _: &mut TestRng) { @@ -829,6 +872,8 @@ mutants!( selectors_per_row_only => "EntryLeakage::selector (\"the same in every row\")", positions_in_selectors => "EntryLeakage::selector (\"Array elements share their array's selector\")", entry_count_padded => "RowLeakage::entries (\"the number of entries\")", + parents_lifted => "EntryLeakage::parent (each node's parent, which a token's walk links)", + element_ordinals_reversed => "EntryLeakage::step (\"an element's index is the ordinal it is found at\")", order_random_permutation => "RowLeakage::entries (the stored order)", members_unsorted => "RowLeakage::entries (\"an object's members in the byte order of their key names\")", parent_tag_on_every_entry => "EntryLeakage::has_parent_tag", @@ -853,7 +898,9 @@ mutants!( sealed_flag_flipped => "RowLeakage::sealed_root", data_key_shared => "RowLeakage::data_key (\"the data key's identifier (unique to the row)\")", keyset_per_row => "RowLeakage::keyset (\"the keyset the data key was minted under\")", + key_tag_shortened => "Leakage::key_tag_len (the header's key tag, a column constant)", extraction_selector_fresh => "QueryLeakage::Extract (\"The selector of the path asked\")", + order_selector_fresh => "QueryLeakage::Order::selector (\"The path's selector\")", queries_nondeterministic => "every query term is a function of the query (\"Tokens are column-wide and do not expire\")", tokens_per_row => "ReadLeak::hits in every row (\"find that node in any row\")", tokens_find_nothing => "ReadLeak::hits (a token finds its node, through its self tag x)", From faff23345a3319216678cfcf9a01404b82ed7278 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:52:37 +1100 Subject: [PATCH 13/19] test(stack-encrypt): widen the leakage corpus to the clauses it never exercised Several clauses the positive control should hold were never tested. No ordered path held a null, a boolean or a container, so "op only on number and string leaves" was never checked. No ordered number was past 2^53, where 9007199254740992 and 9007199254740993 share a term under two value selectors. The random columns held only lower-case ASCII strings and never ordered strings, so folding and multi-byte lengths showed up in one handcrafted column alone. The random generator now holds big integers, floats, and strings that fold or are wider in bytes than in characters. It keeps non-scalars at the ordered paths, and orders numbers or strings by a coin toss (one kind per world, as the profile requires). It runs 256 cases instead of 48, in about three seconds. A small handcrafted column orders numbers of both signs and many magnitudes, the two big integers, and leaves that carry no term, which also gives the shared-prefix clause something to bite on. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/calibration.rs | 7 +- .../tests/json_leakage/corpus.rs | 202 ++++++++++++++---- .../stack-encrypt/tests/json_leakage/mod.rs | 2 +- 3 files changed, 161 insertions(+), 50 deletions(-) diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index b4b714764..f3b52517c 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -106,14 +106,15 @@ mod given_the_full_profile { } } - /// The positive control on random columns. Deterministic: the runner's - /// own generator is seeded, and so is every simulation. + /// The positive control on 256 random columns, about three seconds. + /// Deterministic: the runner's own generator is seeded, and so is every + /// simulation. #[test] fn no_distinguisher_separates_random_worlds() { let runtime = runtime(); let cipher = runtime.block_on(stack_cipher()); let config = Config { - cases: 48, + cases: 256, failure_persistence: None, rng_algorithm: RngAlgorithm::ChaCha, ..Config::default() diff --git a/packages/stack-encrypt/tests/json_leakage/corpus.rs b/packages/stack-encrypt/tests/json_leakage/corpus.rs index 7d91348e5..76de6630e 100644 --- a/packages/stack-encrypt/tests/json_leakage/corpus.rs +++ b/packages/stack-encrypt/tests/json_leakage/corpus.rs @@ -124,6 +124,43 @@ fn names() -> World { } } +/// A column that orders numbers of both signs and many magnitudes, so the +/// terms' shared prefixes (sign, then exponent) have something to show; two +/// integers past 2^53 that round to one `f64`, so one term under two value +/// selectors; and leaves at the ordered path that carry no term (a null, a +/// boolean, an empty object, an array, which holds its own leaf below the +/// ordered path). +fn numbers() -> World { + World { + name: "ordered numbers", + column: Column { + sealed_root: true, + ordered: ordered(&["$.v"]), + }, + docs: vec![ + json!({"v": 1}), + json!({"v": -2}), + json!({"v": 3}), + json!({"v": 9_007_199_254_740_993u64}), + json!({"v": 9_007_199_254_740_992u64}), + json!({"v": 2.5}), + json!({"v": 1e300}), + json!({"v": -1e300}), + json!({"v": 0.001}), + json!({"v": null}), + json!({"v": true}), + json!({"v": {}}), + json!({"v": [3]}), + ], + queries: vec![ + Query::Order("$.v".into(), json!(0)), + Query::Order("$.v".into(), json!(9_007_199_254_740_992u64)), + Query::Order("$.v".into(), json!(-1)), + Query::subtree("$.v"), + ], + } +} + /// A column with three wide top-level objects, each read whole, so that /// whether members' ordinals follow their keys' order is decidable rather /// than a coin toss (see `distinguishers::member_ordinals_follow_key_order`). @@ -158,8 +195,8 @@ fn wide() -> World { } } -/// The handcrafted worlds: the shapes under each root option, and a column -/// of ordered strings. +/// The handcrafted worlds: the shapes under each root option, a column of +/// ordered strings, one of ordered numbers, and one of wide objects. pub fn handcrafted() -> Vec { vec![ World { @@ -181,6 +218,7 @@ pub fn handcrafted() -> Vec { queries: shape_queries(false), }, names(), + numbers(), wide(), ] } @@ -191,12 +229,26 @@ pub fn handcrafted() -> Vec { const KEYS: &[&str] = &["a", "b", "c", "list", "n", "name", "tags", "zz"]; +/// The strings the random columns hold: short, sharing prefixes, and some +/// that fold to one another (`"X"` and `"x"`, `"é"` and `"e"`) or are wider +/// in bytes than in characters. +const STRINGS: &[&str] = &["x", "yy", "x", "zzz", "ab", "", "X", "é", "e", "Ab"]; + fn leaf() -> impl Strategy { prop_oneof![ - Just(Value::Null), - any::().prop_map(Value::Bool), - (-3i64..6).prop_map(Value::from), - prop::sample::select(vec!["x", "yy", "x", "zzz", "ab", ""]).prop_map(Value::from), + 1 => Just(Value::Null), + 1 => any::().prop_map(Value::Bool), + 4 => (-3i64..6).prop_map(Value::from), + // Past 2^53, where two integers can round to one f64 and so share + // an ordering term while their value selectors differ; and floats. + 1 => prop::sample::select(vec![ + json!(9_007_199_254_740_992u64), + json!(9_007_199_254_740_993u64), + json!(2.5), + json!(-0.5), + json!(1e300), + ]), + 4 => prop::sample::select(STRINGS).prop_map(Value::from), ] } @@ -223,37 +275,92 @@ fn value() -> impl Strategy { }) } -/// Every value under the key `n` made a number, since the random columns -/// order `$.n` and `$.list[*].n` and a world orders one kind of leaf. -fn numberize(value: Value) -> Value { - match value { - Value::Object(map) => Value::Object( - map.into_iter() - .map(|(k, v)| { - let v = if k == "n" && !v.is_number() { - Value::from(i64::try_from(v.to_string().len() % 7).unwrap()) - } else { - numberize(v) - }; - (k, v) +/// Which kind of leaf a random column orders: a world orders one (see +/// [`super::profile::Ids::ordering_ranks`]). +#[derive(Clone, Copy, Debug)] +enum Ordered { + /// `$.n` and `$.list[*].n`. + Numbers, + /// `$.name` and `$.tags[*]`. + Strings, +} + +impl Ordered { + fn paths(self) -> Vec { + match self { + Self::Numbers => ordered(&["$.n", "$.list[*].n"]), + Self::Strings => ordered(&["$.name", "$.tags[*]"]), + } + } + + /// `value` with every leaf an ordered path could reach made the kind + /// this column orders: a string under the key `n` a number, or a number + /// under the key `name` or directly in an array under `tags` a string. + /// Everything else stays, `null`s, booleans and containers at the + /// ordered paths included, which carry no term. + fn fix(self, value: Value) -> Value { + let to_number = |v: &Value| Value::from(i64::try_from(v.to_string().len() % 7).unwrap()); + let to_string = |v: &Value| Value::from(STRINGS[v.to_string().len() % STRINGS.len()]); + match (self, value) { + (_, Value::Object(map)) => Value::Object( + map.into_iter() + .map(|(k, v)| { + let v = match (self, k.as_str(), v) { + (Self::Numbers, "n", v) if v.is_string() => to_number(&v), + (Self::Strings, "name", v) if v.is_number() => to_string(&v), + (Self::Strings, "tags", Value::Array(items)) => Value::Array( + items + .into_iter() + .map(|item| { + if item.is_number() { + to_string(&item) + } else { + self.fix(item) + } + }) + .collect(), + ), + (_, _, v) => self.fix(v), + }; + (k, v) + }) + .collect(), + ), + (_, Value::Array(items)) => { + Value::Array(items.into_iter().map(|item| self.fix(item)).collect()) + } + (_, other) => other, + } + } + + /// An ordering bound of this kind at one of this kind's paths. + fn query(self) -> BoxedStrategy { + match self { + Self::Numbers => (prop::sample::select(vec!["$.n", "$.list[*].n"]), leaf()) + .prop_filter_map("a number bound", |(p, v)| { + v.is_number().then(|| Query::Order(p.into(), v)) }) - .collect(), - ), - Value::Array(items) => Value::Array(items.into_iter().map(numberize).collect()), - other => other, + .boxed(), + Self::Strings => ( + prop::sample::select(vec!["$.name", "$.tags[*]"]), + prop::sample::select(STRINGS), + ) + .prop_map(|(p, v)| Query::Order(p.into(), Value::from(v))) + .boxed(), + } } } -fn document() -> impl Strategy { +fn document(kind: Ordered) -> impl Strategy { prop_oneof![ 8 => object(value(), 0..5), 1 => prop::collection::vec(value(), 0..3).prop_map(Value::Array), 1 => leaf(), ] - .prop_map(numberize) + .prop_map(move |v| kind.fix(v)) } -fn query() -> impl Strategy { +fn query(kind: Ordered) -> impl Strategy { let subtree = vec![ "$", "$.a", @@ -294,31 +401,34 @@ fn query() -> impl Strategy { prop::collection::vec(leaf(), 1..3).prop_map(Value::Array), object(prop::collection::vec(object(leaf(), 1..2), 1..2).prop_map(Value::Array), 1..2), ] - .prop_map(|v| Query::Contains(numberize(v))), + .prop_map(move |v| Query::Contains(kind.fix(v))), 1 => (prop::sample::select(vec!["$.a", "$.tags[0]", "$.list[*].b"]), prop_oneof![(-3i64..6).prop_map(Value::from), prop::sample::select(vec!["x", "ab"]).prop_map(Value::from)]) .prop_map(|(p, v)| Query::Equals(p.into(), v)), - 1 => (prop::sample::select(vec!["$.n", "$.list[*].n"]), -3i64..6) - .prop_map(|(p, v)| Query::Order(p.into(), Value::from(v))), + 1 => kind.query(), ] } -/// A random column: a root option, two to six documents, and four to ten -/// queries, some of which repeat. +/// A random column: a root option, the kind of leaf it orders, two to six +/// documents, and four to ten queries, some of which repeat. pub fn random_world() -> impl Strategy { - any::().prop_flat_map(|sealed_root| { - ( - prop::collection::vec(document(), 2..7), - prop::collection::vec(query(), 4..11), - ) - .prop_map(move |(docs, queries)| World { - name: "random", - column: Column { - sealed_root, - ordered: ordered(&["$.n", "$.list[*].n"]), - }, - docs, - queries, - }) - }) + ( + any::(), + prop_oneof![Just(Ordered::Numbers), Just(Ordered::Strings)], + ) + .prop_flat_map(|(sealed_root, kind)| { + ( + prop::collection::vec(document(kind), 2..7), + prop::collection::vec(query(kind), 4..11), + ) + .prop_map(move |(docs, queries)| World { + name: "random", + column: Column { + sealed_root, + ordered: kind.paths(), + }, + docs, + queries, + }) + }) } diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 048565456..20af3302f 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -66,7 +66,7 @@ //! //! - **The positive control.** The simulator, run from the full profile, is //! told from the real scheme by no distinguisher, on the handcrafted -//! columns and on 48 random ones. If this fails, the real scheme reveals +//! columns and on 256 random ones. If this fails, the real scheme reveals //! something `L` does not state: that is a finding against the page (or //! against the code), and the failure names the distinguisher and prints //! both features. From 981360faf3400b0f3260dccfcfd74eedfb5b5f5a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:54:10 +1100 Subject: [PATCH 14/19] test(stack-encrypt): say #1363 corrected the probe comment, and give assertions messages After the merge of #1363, json_tree_subtree's comment says the server "can pick it out"; mod.rs and the doc comment of a_read_gives_away_its_bound_and_its_probes still quoted the old claim as standing. Both now say it was corrected. The test's assert_eq! calls gain the messages the rust-testing convention asks for, and mod.rs names the two harness tests that show features no mutant can reach do fire. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../tests/json_leakage/calibration.rs | 29 ++++++++++++++----- .../stack-encrypt/tests/json_leakage/mod.rs | 11 ++++--- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index f3b52517c..bc8fc27c3 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -419,9 +419,9 @@ mod the_harness { /// so `N = 64`, at two), the probe is the last token at its level and /// finds the element where the path's other tokens find the leaf below /// it, and which tokens find a node says which elements hold the rest of - /// the path. `json_tree_subtree`'s comment that "the server cannot tell - /// a probe from any other token" is true of a token's bytes and of - /// nothing else. + /// the path. `json_tree_subtree`'s comment once said "the server + /// cannot tell a probe from any other token", which is true of a token's + /// bytes and of nothing else; #1363 corrected it. #[test] fn a_read_gives_away_its_bound_and_its_probes() { let runtime = runtime(); @@ -444,23 +444,36 @@ mod the_harness { ], }; let observed = real_world(&runtime, &cipher, &world); - assert_eq!(distinguishers::token_counts(&observed), [65, 4161]); + assert_eq!( + distinguishers::token_counts(&observed), + [65, 4161], + "one wildcard level sends N + 1 tokens, two send N² + N + 1" + ); // 4,161 is also one level at a bound of 4,160; which tokens find a // node (the 65th finds `$.items[0].tags[64]`'s absence, not an // item) is what tells the reading apart. assert_eq!( distinguishers::bound_recovery(&observed), - [vec![(1, 64)], vec![(1, 4160), (2, 64)]] + [vec![(1, 64)], vec![(1, 4160), (2, 64)]], + "the count alone gives the bound" ); let price = &distinguishers::wildcard_positions(&observed)[0]; let price = price.as_ref().expect("a wildcard read"); // No items; three, the second without a price; seventy, the // second without a price, longer than the bound. - assert_eq!(price[0], (vec![], false, None)); - assert_eq!(price[1], (vec![0, 2], false, None)); + assert_eq!(price[0], (vec![], false, None), "no items, no hits"); + assert_eq!( + price[1], + (vec![0, 2], false, None), + "three items, the second without a price" + ); let mut most: Vec = (0..64).collect(); most.retain(|&i| i != 1); - assert_eq!(price[2], (most, true, Some(true))); + assert_eq!( + price[2], + (most, true, Some(true)), + "seventy items: the probe finds the element, under another selector" + ); } /// `distinguishers::cross_kind_equalities` can fire: it is empty on the diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 20af3302f..79bc4fa14 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -92,7 +92,10 @@ //! on a column whose rows vary, which is the page's claim that the stored //! order reveals each row's shape; and on the real scheme a wildcard //! read's token list gives its bound and its probes away -//! (`a_read_gives_away_its_bound_and_its_probes`). +//! (`a_read_gives_away_its_bound_and_its_probes`). Two features no mutant +//! can reach are shown to fire: `cross_kind_equalities` on a world whose +//! token is a stored selector, and `token_derivability` on a read of a +//! position past every row's array. //! //! # What the profile found //! @@ -115,13 +118,13 @@ //! Against the SQL's own comment, not the page: //! //! 2. **A probe is told from the other tokens.** `json_tree_subtree`'s -//! comment says "the server cannot tell a probe from any other token". +//! comment said "the server cannot tell a probe from any other token". //! It cannot from the token's bytes, but the probe is the last token at //! its level, the count gives the bound away (65 tokens is one level at //! `N = 64` and nothing else), and a probe that finds something finds //! the element, whose selector differs from the leaf the path's other -//! tokens find. `a_read_gives_away_its_bound_and_its_probes`. The leakage -//! page does not repeat the claim; the comment is left to #1363. +//! tokens find. `a_read_gives_away_its_bound_and_its_probes`. #1363 +//! corrected the comment, which now says the server "can pick it out". //! //! The three findings against the page as it stood before #1363 (an //! ordering term's length, equal values at two ordered paths, UTF-8 byte From 5455ae0d85917e3301b9aedf494745187e12ef9e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 19:54:10 +1100 Subject: [PATCH 15/19] docs(plans): correct the leakage note's net, and record the review The net paragraph said the member-or-element rule "was already disclosed by the gated walk's ordinals and #n: gates". Before #1363 every child's token took one rule, and an only child's ordinal is 0 either way, so a node that is an only child in every row had its kind disclosed by nothing. The note's own finding said so. The paragraph now says what was and was not disclosed, and that the net rests on the ancestor token and the gates. The SQL-comment paragraph records that #1363 corrected the comment, and a closing section records what the review added: the ordering terms' shared prefixes, which both profiles missed, the new checks, mutants and corpus, and what is left to the owner. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- ...10-25-json-tree-leakage-before-and-after.md | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index caf9f8690..990706713 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -25,7 +25,7 @@ The tables below grow from the old profile to the new one: more clauses, more di - A read no longer sends any token above the node it asks for. Before, a read of `$.profile.address.city` sent `$.profile`'s token, a capability over that whole field in every row of the column, for the life of the column. Now it sends the leaf's token and nothing that covers a node above it. The `ancestor_links` distinguisher exists to hold this. - The gates are gone. Each key step and array position below the head used to reach the server in the clear, as a selector or `#n:selector`. -What is new is small. The self tag `x` is a fresh unlinkable name per entry that at rest says only that entries are distinct, which the entry count already said; a token the client sends links it, as a token always linked the parent tag. The member-or-element rule and the element index at the nodes a walk reaches were already disclosed by the gated walk's ordinals and `#n:` gates, but the page had not said so, and now does. A wildcard read's token count, bound and probes are a smaller form of what the plaintext gates showed. The last two sections set each item out. +What is new is small. The self tag `x` is a fresh unlinkable name per entry that at rest says only that entries are distinct, which the entry count already said; a token the client sends links it, as a token always linked the parent tag. At the nodes a walk reaches, the element index was already disclosed by the gated walk's ordinals. Whether a node is a member or an element was disclosed only where selectors showed it, in an array that has two elements in some row. It is new for a node that is an only child in every row: before #1363 every child's token took one rule, and an only child's ordinal is 0 either way. The page now says so. That walk also covers fewer nodes than the old one, because no read sends a token above the node it names. A wildcard read's token count, bound and probes are a smaller form of what the plaintext gates showed. The last two sections set each item out. ## The two formats, briefly @@ -185,7 +185,7 @@ Two tests are new. `given_strings_ordered_by_their_bytes` holds the page's foldi **One finding against the page as #1363 published it, fixed on this branch.** Nothing stored says whether a node is an object member or an array element (the design's decision 2, "no kind bit"). But to derive the token of each node it reaches, the database tests the node's self tag against the member rule and then the element rule, and the one that matches says which. So below any token it is sent, the database learns whether each node is a member or an element, and therefore whether each parent is an object or an array; for an element it also learns the index, which is the ordinal it was found at. Where selectors already show it (an array that has two elements in some row) this adds nothing; for a node that is an only child in every row, it does. The page said a token's holder can "see how the nodes beneath it nest, link each one to its parent, and derive the tokens of the nodes beneath". This branch adds: "It also learns whether each node beneath is an object member or an array element, and each element's index". `the_hit_rule_separates_it_from_the_real_scheme` shows the sentence is needed: without it, `hit_rules` separates the worlds, and so do the three features that break when a world tries to hide the rule while still matching the tokens a client sends (a read through a key step always sends a member-rule token). -**One inaccuracy in a SQL comment, reported, not changed.** `json_tree_subtree`'s comment in `packages/eql/src/v3/json_tree/functions.sql` says "the server cannot tell a probe from any other token". It cannot from a token's bytes. But the probe is always the last token at its level, the count gives the bound away, and a probe that finds something finds the element, whose selector differs from that of the leaf the path's other tokens find. The leakage page does not repeat the claim. The comment is #1363's to correct. +**One inaccuracy in a SQL comment, reported, and corrected by #1363.** `json_tree_subtree`'s comment in `packages/eql/src/v3/json_tree/functions.sql` said "the server cannot tell a probe from any other token". It cannot from a token's bytes. But the probe is always the last token at its level, the count gives the bound away, and a probe that finds something finds the element, whose selector differs from that of the leaf the path's other tokens find. The leakage page does not repeat the claim. #1363 corrected the comment in `b2a20abb9`: it now says the server "can pick it out". **One error in the old profile, which its corpus could not show.** It ordered strings by their raw bytes. The code folds case and accents before ordering, so `"Carol"` and `"carol"` share a term, and the term's length counts the folded form. The old corpus held only lower-case ASCII, where the two agree. The port models folding, the corpus now holds names that fold, and a test holds the page's sentence. @@ -209,4 +209,16 @@ Two tests are new. `given_strings_ordered_by_their_bytes` holds the page's foldi - **The gates.** Before, a read sent the selector of every step below the head, and every array position in plaintext (`#n:`), whether or not a row had the path. Now no selector is sent with a read: the database learns the selector of the node read only from the entries the token finds. Array positions travel only inside tokens; the database still infers a found node's position from the stored order (#1357), and, for a wildcard, from the token's place in the list. - **The head token.** Reading `$.profile.address.city` sends the city leaf's token and nothing that covers `$.profile`. Under the sealed root the root's token, sent only for a read of `$`, finds the root entry and nothing beneath it, since the top-level fields carry no parent tag. -Everything at rest other than `x` is unchanged: shape through the stored order, parent structure, sibling key order, selector equality across rows, value-selector equality, ciphertext lengths, ordering terms with their lengths and their equality across ordered paths, and the header. +Everything at rest other than `x` is unchanged: shape through the stored order, parent structure, sibling key order, selector equality across rows, value-selector equality, ciphertext lengths, ordering terms with their lengths, their equality across ordered paths and their shared prefixes, and the header. + +## After review + +A review of the port (2026-10-25) found one disclosure that both profiles missed, and a set of gaps in the method. None of them changes the comparison above, since the missed disclosure is the same before and after #1363. + +- **The ordering terms' shared prefixes.** A CLLW OPE term is a carry byte, then one byte per plaintext bit (the PRF of the bits before it, plus the bit at weight 128, plus a carry). So two terms agree to within one up to the first bit their values' encodings differ in, and differ by about 128 there. The database reads that bit off the stored terms exactly. For numbers it is their signs and binary exponents. The page stated the disclosure in bytes, and the profile had listed it as out of reach ("not an exact function of the plaintexts", which is true of the byte prefix and not of the bit). Its simulator drew terms with no prefix structure, so nothing could see the omission. It is now a clause (`Leakage::ordering_prefixes`), with a distinguisher (`ordering_term_prefixes`) and a mutant (`ordering_prefixes_hidden`), and the simulator builds CLLW-shaped terms. The page's sentence now says bits, and what they mean for a number and a string. +- **New checks on the method.** `cross_kind_equalities` compares values across kinds (a token equal to a selector, say), which every other feature names apart. `term_fields` checks the field names inside `q` and every query term, which the parser would otherwise drop unread. `token_derivability` tries the element rule up to index 64 or the longest token list, not only below 4. +- **Four clauses with no mutant** now have one: `key_tag_shortened`, `order_selector_fresh`, `element_ordinals_reversed`, `parents_lifted`. +- **The corpus.** The random columns hold big integers, floats, strings that fold, and non-scalars at ordered paths. They order numbers or strings, and run 256 cases instead of 48. A fifth handcrafted column orders numbers of both signs and many magnitudes, and holds two integers past 2⁵³ that share a term. +- **Left to the owner:** a path ordered over numbers and strings both. Their terms are under different keys, so how they compare is set by the column key, and the profile refuses such a column; `mod.rs` records what was measured. Also, a number's equality under its term is equality as an `f64`, which the page does not say. + +After the review: 38 distinguishers, 49 mutants, 11 tests. From 905791dfc0b9d0db45ba45115d0515c7cb567f27 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 20:46:05 +1100 Subject: [PATCH 16/19] test(stack-encrypt): state what a path ordered over numbers and strings shows The profile refused a column whose ordered path held numbers in some rows and strings in others, and the page said nothing about one. The owner's decision: the page states it, and so does the profile. vitaminc_ore's order space salts its key with the domain, so a number's term and a string's are under different keys. A CLLW term begins with a carry byte and then the key's constant byte plus 128 for a first bit of 1. Under one key, then, every string compares the same way with every negative number and the same way with every non-negative one: the strings sort as one block, which the key places below the numbers, between the negative and non-negative ones, or above them. That is three placements, not the two of a single coin. Within each kind the order is the values'. The terms also show which leaves are numbers and which are strings. The profile now ranks classes within their kind, states each class's kind, and leaves the placement to the key (Leakage::kind_order). The simulator puts each kind's terms under a key of its own and draws key pairs until the strings fall at the placement asked. It refuses pairs whose front bytes tie to within a carry (about one in forty), under which a few pairs compare by later keyed bytes, a case no clause states. The real column's key picked one placement and the simulator cannot know which, so the calibration simulates each and counts worlds as told apart only when none agrees. The mutant mixed_kinds_jsonb_order fixes jsonb's numbers-first order. It is caught by ordering_term_order, ordering_matches and ordering_term_prefixes, because the test key puts the strings below the numbers; one world cannot tell a key that chose numbers-first from jsonb. A handcrafted column and a third kind of random column order numbers and strings at one path, with bounds of both kinds. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql/docs/reference/json-tree-leakage.md | 2 +- .../tests/json_leakage/calibration.rs | 75 ++++++--- .../stack-encrypt/tests/json_leakage/chain.rs | 13 ++ .../tests/json_leakage/corpus.rs | 42 +++-- .../tests/json_leakage/distinguishers.rs | 10 +- .../stack-encrypt/tests/json_leakage/mod.rs | 25 ++- .../tests/json_leakage/mutants.rs | 24 ++- .../tests/json_leakage/profile.rs | 154 ++++++++++++------ .../tests/json_leakage/simulator.rs | 138 ++++++++++++---- 9 files changed, 345 insertions(+), 138 deletions(-) diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index f5226d06f..5ace4d7fc 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -22,7 +22,7 @@ Someone who holds the column and no key, such as a database administrator, a bac | `p`, the parent tag below the chain's heads | Nothing beyond the entry order on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes, and it does not say which entry is the parent. Which entries carry no `p` does show: with the sealed root on, those are the root's direct members, and since everything beneath a member follows it, **the node count of each top-level field's subtree** can be read off a single row exactly. With the sealed root off, only the root lacks one. | | `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | -| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | +| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column for each kind of leaf, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. At an ordered path whose values are numbers in some rows and strings in others, the order between a number and a string depends on the column key and is not jsonb's (jsonb sorts numbers before strings); within each kind the order is the values'. The two kinds' terms are under different keys, so the terms also show which leaves are numbers and which are strings, and the strings sort together, as one block that the key places below the numbers, between the negative and the non-negative ones, or above them. By default no path is ordered; see the options below. | | `h`, the header | The keyset the data key was minted under, the data key's identifier (unique to the row), and whether the root is sealed. `v` and `i` are the version and the table and column identifier every EQL value carries. | Not revealed at rest: any value's or key name's text, and, except through the ordering terms at the paths the index orders, whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the byte order of sibling key names. diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index bc8fc27c3..17548a24b 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -2,6 +2,8 @@ //! (the positive control), the page as #1363 published it is not (the //! finding), and every clause's removal is noticed (the calibration). +use std::collections::BTreeSet; + use proptest::test_runner::{Config, RngAlgorithm, TestCaseError, TestRng, TestRunner}; use serde_json::json; use stack_encrypt::json::{JsonOptions, JsonPath}; @@ -12,7 +14,10 @@ use super::corpus::{handcrafted, random_world, World}; use super::distinguishers::{self, walk_positions}; use super::mutants; use super::observe::{ObservedQuery, ObservedWorld}; -use super::profile::{flatten, leak, Clauses, Column, KeyOrder, Leakage, Query, StringOrder}; +use super::profile::{ + flatten, leak, Clauses, Column, KeyOrder, KindOrder, Leakage, OrderKind, Placement, Query, + QueryLeakage, StringOrder, +}; use super::{real, simulator}; use crate::common::stack_cipher; @@ -44,9 +49,25 @@ fn real_world( ObservedWorld::from_wire(&rows, &queries) } -fn simulated_world(leakage: &Leakage, rng: &mut TestRng) -> ObservedWorld { - let (rows, queries) = simulator::simulate(leakage, rng); - ObservedWorld::from_wire(&rows, &queries) +/// The placements of the strings' terms among the numbers' a simulation +/// tries: every one a key could pick where the profile leaves it to the key +/// and the column orders both kinds, since the real column's key picked one +/// and the simulator cannot know which; otherwise only the profile's own. +fn placements(leakage: &Leakage) -> Vec> { + let kinds: BTreeSet = leakage + .rows + .iter() + .flat_map(|r| r.entries.iter().filter_map(|e| e.ordering.map(|o| o.kind))) + .chain(leakage.queries.iter().filter_map(|q| match q { + QueryLeakage::Order { bound, .. } => Some(bound.kind), + _ => None, + })) + .collect(); + if leakage.kind_order == KindOrder::ByKey && kinds.len() == 2 { + Placement::ALL.into_iter().map(Some).collect() + } else { + vec![None] + } } /// Every distinguisher's feature of `world`, by name. @@ -57,16 +78,28 @@ fn features(world: &ObservedWorld) -> Vec<(&'static str, String)> { .collect() } -/// The distinguishers whose features differ between two worlds, each with -/// the two features. +/// The distinguishers whose features differ between the real world and +/// the simulation of `leakage`, each with the two features: of the +/// simulations at every placement [`placements`] tries, the one fewest +/// distinguishers separate. So the worlds count as told apart only when no +/// placement a key could pick makes them agree. fn separating( real: &[(&'static str, String)], - simulated: &ObservedWorld, + leakage: &Leakage, + rng: &mut TestRng, ) -> Vec<(&'static str, String, String)> { - real.iter() - .zip(features(simulated)) - .filter_map(|((name, a), (_, b))| (*a != b).then(|| (*name, a.clone(), b))) - .collect() + placements(leakage) + .into_iter() + .map(|placement| { + let (rows, queries) = simulator::simulate(leakage, rng, placement); + let simulated = ObservedWorld::from_wire(&rows, &queries); + real.iter() + .zip(features(&simulated)) + .filter_map(|((name, a), (_, b))| (*a != b).then(|| (*name, a.clone(), b))) + .collect::>() + }) + .min_by_key(Vec::len) + .expect("at least one placement") } fn report(separated: &[(&'static str, String, String)]) -> String { @@ -95,8 +128,7 @@ mod given_the_full_profile { &world.docs, &world.queries, ); - let simulated = simulated_world(&leakage, &mut rng()); - let separated = separating(&features(&real), &simulated); + let separated = separating(&features(&real), &leakage, &mut rng()); assert!( separated.is_empty(), "{}: the real scheme reveals more than L states:\n{}", @@ -128,8 +160,7 @@ mod given_the_full_profile { &world.docs, &world.queries, ); - let simulated = simulated_world(&leakage, &mut rng()); - let separated = separating(&features(&real), &simulated); + let separated = separating(&features(&real), &leakage, &mut rng()); if separated.is_empty() { Ok(()) } else { @@ -179,9 +210,8 @@ mod given_the_page_as_published_in_1363 { &world.docs, &world.queries, ); - let simulated = simulated_world(&leakage, &mut rng()); fired.extend( - separating(&features(&real), &simulated) + separating(&features(&real), &leakage, &mut rng()) .into_iter() .map(|(n, _, _)| n), ); @@ -226,7 +256,7 @@ mod given_strings_ordered_by_their_bytes { .expect("the corpus has it"); let real = real_world(&runtime, &cipher, &world); let leakage = leak(&world.column, clauses, &world.docs, &world.queries); - let fired: Vec<&str> = separating(&features(&real), &simulated_world(&leakage, &mut rng())) + let fired: Vec<&str> = separating(&features(&real), &leakage, &mut rng()) .into_iter() .map(|(n, _, _)| n) .collect(); @@ -259,7 +289,7 @@ mod given_the_jsonb_reading_of_the_key_order { let world = &handcrafted()[0]; let real = real_world(&runtime, &cipher, world); let leakage = leak(&world.column, clauses, &world.docs, &world.queries); - let fired: Vec<&str> = separating(&features(&real), &simulated_world(&leakage, &mut rng())) + let fired: Vec<&str> = separating(&features(&real), &leakage, &mut rng()) .into_iter() .map(|(n, _, _)| n) .collect(); @@ -307,8 +337,11 @@ mod given_a_mutant_profile { for (real, leakage) in &worlds { let mut mutated = leakage.clone(); (mutant.apply)(&mut mutated, &mut rng); - let simulated = simulated_world(&mutated, &mut rng); - fired.extend(separating(real, &simulated).into_iter().map(|(n, _, _)| n)); + fired.extend( + separating(real, &mutated, &mut rng) + .into_iter() + .map(|(n, _, _)| n), + ); } let expected_unnoticed = UNNOTICED.iter().any(|(name, _)| *name == mutant.name); if fired.is_empty() != expected_unnoticed { diff --git a/packages/stack-encrypt/tests/json_leakage/chain.rs b/packages/stack-encrypt/tests/json_leakage/chain.rs index fabfdeb27..375b72aa3 100644 --- a/packages/stack-encrypt/tests/json_leakage/chain.rs +++ b/packages/stack-encrypt/tests/json_leakage/chain.rs @@ -116,6 +116,19 @@ pub fn cllw_ope(key: &[u8; 32], bits: &[bool]) -> Vec { out } +/// How many leading plaintext bits two CLLW OPE terms show their values +/// share: the bytes after the carry byte before the first that differs by +/// more than one modulo 256, or the shorter term's bit count when there is +/// none. A shared bit's bytes differ only by a carry, zero or one; the first +/// differing bit's by 127 to 129. Terms under two keys share none, but where +/// a front byte ties to within a carry. +pub fn shared_bits(a: &[u8], b: &[u8]) -> usize { + let bits = a.len().min(b.len()).saturating_sub(1); + (1..=bits) + .find(|&k| !matches!(a[k].wrapping_sub(b[k]), 0 | 1 | 255)) + .map_or(bits, |k| k - 1) +} + /// The simulator's stand-in for the keyed member rank: any keyed function of /// the parent's token and the member's selector will do, since the real one /// is a PRF under a key the database never sees. The simulator samples its diff --git a/packages/stack-encrypt/tests/json_leakage/corpus.rs b/packages/stack-encrypt/tests/json_leakage/corpus.rs index 76de6630e..143148ba8 100644 --- a/packages/stack-encrypt/tests/json_leakage/corpus.rs +++ b/packages/stack-encrypt/tests/json_leakage/corpus.rs @@ -89,9 +89,8 @@ fn shape_queries(sealed_root: bool) -> Vec { queries } -/// A column that orders strings, so the string half of the ordering clause -/// has something to compare (a world orders one kind of leaf; see -/// [`super::profile::Ids::ordering_ranks`]). `"Carol"` and `"carol"`, and +/// A column that orders strings only, so the string half of the ordering +/// clause has something to compare on its own. `"Carol"` and `"carol"`, and /// `"ál"` and `"al"`, fold to one value, so their terms are equal and /// `"ál"`'s length is that of the two bytes it folds to. fn names() -> World { @@ -124,15 +123,17 @@ fn names() -> World { } } -/// A column that orders numbers of both signs and many magnitudes, so the +/// A column whose one ordered path holds numbers in some rows and strings in +/// others, so the strings' terms have a place among the numbers' that the +/// column's key picks; numbers of both signs and many magnitudes, so the /// terms' shared prefixes (sign, then exponent) have something to show; two /// integers past 2^53 that round to one `f64`, so one term under two value /// selectors; and leaves at the ordered path that carry no term (a null, a /// boolean, an empty object, an array, which holds its own leaf below the -/// ordered path). +/// ordered path). Bounds of both kinds compare with the terms of both. fn numbers() -> World { World { - name: "ordered numbers", + name: "numbers and strings at one ordered path", column: Column { sealed_root: true, ordered: ordered(&["$.v"]), @@ -151,9 +152,14 @@ fn numbers() -> World { json!({"v": true}), json!({"v": {}}), json!({"v": [3]}), + json!({"v": "carol"}), + json!({"v": "Carol"}), + json!({"v": "al"}), + json!({"v": ""}), ], queries: vec![ Query::Order("$.v".into(), json!(0)), + Query::Order("$.v".into(), json!("b")), Query::Order("$.v".into(), json!(9_007_199_254_740_992u64)), Query::Order("$.v".into(), json!(-1)), Query::subtree("$.v"), @@ -196,7 +202,8 @@ fn wide() -> World { } /// The handcrafted worlds: the shapes under each root option, a column of -/// ordered strings, one of ordered numbers, and one of wide objects. +/// ordered strings, one ordering numbers and strings at one path, and one +/// of wide objects. pub fn handcrafted() -> Vec { vec![ World { @@ -275,20 +282,23 @@ fn value() -> impl Strategy { }) } -/// Which kind of leaf a random column orders: a world orders one (see -/// [`super::profile::Ids::ordering_ranks`]). +/// Which kind of leaf a random column orders: numbers, strings, or both at +/// one path. #[derive(Clone, Copy, Debug)] enum Ordered { /// `$.n` and `$.list[*].n`. Numbers, /// `$.name` and `$.tags[*]`. Strings, + /// `$.n` and `$.list[*].n`, holding numbers in some rows and strings in + /// others. + Mixed, } impl Ordered { fn paths(self) -> Vec { match self { - Self::Numbers => ordered(&["$.n", "$.list[*].n"]), + Self::Numbers | Self::Mixed => ordered(&["$.n", "$.list[*].n"]), Self::Strings => ordered(&["$.name", "$.tags[*]"]), } } @@ -302,6 +312,7 @@ impl Ordered { let to_number = |v: &Value| Value::from(i64::try_from(v.to_string().len() % 7).unwrap()); let to_string = |v: &Value| Value::from(STRINGS[v.to_string().len() % STRINGS.len()]); match (self, value) { + (Self::Mixed, value) => value, (_, Value::Object(map)) => Value::Object( map.into_iter() .map(|(k, v)| { @@ -347,6 +358,11 @@ impl Ordered { ) .prop_map(|(p, v)| Query::Order(p.into(), Value::from(v))) .boxed(), + Self::Mixed => (prop::sample::select(vec!["$.n", "$.list[*].n"]), leaf()) + .prop_filter_map("a number or string bound", |(p, v)| { + (v.is_number() || v.is_string()).then(|| Query::Order(p.into(), v)) + }) + .boxed(), } } } @@ -414,7 +430,11 @@ fn query(kind: Ordered) -> impl Strategy { pub fn random_world() -> impl Strategy { ( any::(), - prop_oneof![Just(Ordered::Numbers), Just(Ordered::Strings)], + prop_oneof![ + Just(Ordered::Numbers), + Just(Ordered::Strings), + Just(Ordered::Mixed) + ], ) .prop_flat_map(|(sealed_root, kind)| { ( diff --git a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs index 68ec93547..3f3abd8d2 100644 --- a/packages/stack-encrypt/tests/json_leakage/distinguishers.rs +++ b/packages/stack-encrypt/tests/json_leakage/distinguishers.rs @@ -18,7 +18,7 @@ use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::hash::Hash; use super::chain::{ - element_token, hit_token, member_token, parent_tag, self_tag, Rule, Tag16, Tag8, + element_token, hit_token, member_token, parent_tag, self_tag, shared_bits, Rule, Tag16, Tag8, }; use super::observe::{ObservedGroup, ObservedQuery, ObservedRow, ObservedWorld}; @@ -1388,16 +1388,10 @@ pub fn ordering_term_prefixes(world: &ObservedWorld) -> Vec<(usize, usize, usize })); all.sort_unstable(); all.dedup(); - let shared = |a: &[u8], b: &[u8]| { - let bits = a.len().min(b.len()).saturating_sub(1); - (1..=bits) - .find(|&k| !matches!(a[k].wrapping_sub(b[k]), 0 | 1 | 255)) - .map_or(bits, |k| k - 1) - }; let mut out = Vec::new(); for (i, a) in all.iter().enumerate() { for (j, b) in all.iter().enumerate().skip(i + 1) { - out.push((i, j, shared(a, b))); + out.push((i, j, shared_bits(a, b))); } } out diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 79bc4fa14..8a17a2ba3 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -219,15 +219,22 @@ //! [`distinguishers::ordering_term_prefixes`]); which shared bytes differ //! by one is not a function of the plaintexts, and no distinguisher reads //! it. -//! - **A path ordered over numbers and strings both.** The profile refuses -//! such a column ([`profile::Ids::ordering_ranks`]), and the page does not -//! say what it shows. Measured on this crate: a number's term and a -//! string's are under different keys (the order space's salt names its -//! domain), so how one compares with the other is set by the column's key -//! and the two values' leading bits, not by any order between a number -//! and a string. In the column measured every string sorted below every -//! number, and a bound of either kind compares with the stored terms of -//! both. Stating it needs the page to say it first. +//! - **A near-tie between the two kinds' keys.** At a path ordered over +//! numbers and strings both, the profile states where the column's key +//! puts the strings' terms among the numbers' ([`profile::Placement`]): +//! as one block, below, between the negative and non-negative numbers, or +//! above. That holds because each kind's terms begin with that kind's +//! key's constant byte (plus 128 for a first bit of 1, with a carry). For +//! about one pair of keys in forty, the two kinds' front bytes tie to +//! within a carry, and then a few pairs of a number and a string compare +//! by their later, keyed bytes. No clause states that case; the simulator +//! refuses such keys, and the test key is not one. Since the real key +//! picked one placement and the simulator cannot know which, the +//! calibration simulates each and counts two worlds as told apart only +//! when none agrees, which is also why one world cannot tell "the key +//! chose numbers first" from jsonb's fixed order: the mutant that states +//! jsonb's (`mixed_kinds_jsonb_order`) is caught because the test key put +//! the strings below the numbers. //! - **The keyed member rank as a permutation.** Under a token, a member's //! ordinal is a keyed rank. Its relation to the stored (key) order is a //! random permutation, decidable exactly only where enough wide objects sit diff --git a/packages/stack-encrypt/tests/json_leakage/mutants.rs b/packages/stack-encrypt/tests/json_leakage/mutants.rs index 29d1b5bb9..661c971c7 100644 --- a/packages/stack-encrypt/tests/json_leakage/mutants.rs +++ b/packages/stack-encrypt/tests/json_leakage/mutants.rs @@ -18,8 +18,8 @@ use proptest::prelude::RngCore; use proptest::test_runner::TestRng; use super::profile::{ - EntryLeakage, GroupLeakage, Leakage, QueryLeakage, ReadLeak, RowLeakage, Rule, Step, TokenName, - TokenStep, AEAD_OVERHEAD, + EntryLeakage, GroupLeakage, KindOrder, Leakage, OrderKind, Placement, QueryLeakage, ReadLeak, + RowLeakage, Rule, Step, TokenName, TokenStep, AEAD_OVERHEAD, }; /// One mutant: its name, the clause it drops (a field or sentence of the @@ -408,20 +408,21 @@ fn ordering_terms_all_distinct(l: &mut Leakage, rng: &mut TestRng) { /// two ordered paths compare with each other too"). The order is kept, and /// ties across paths are broken by path. fn ordering_terms_per_path(l: &mut Leakage, _: &mut TestRng) { - let mut keyed: BTreeMap<(usize, usize), usize> = BTreeMap::new(); + let mut keyed: BTreeMap<(OrderKind, usize, usize), usize> = BTreeMap::new(); for row in &l.rows { for entry in &row.entries { if let Some(op) = &entry.ordering { - let _ = keyed.insert((l.ordering_ranks[op.class], entry.selector), 0); + let _ = keyed.insert((op.kind, l.ordering_ranks[op.class], entry.selector), 0); } } } for query in &l.queries { if let QueryLeakage::Order { selector, bound } = query { - let _ = keyed.insert((l.ordering_ranks[bound.class], *selector), 0); + let _ = keyed.insert((bound.kind, l.ordering_ranks[bound.class], *selector), 0); } } - // Classes in (rank, path) order, so a class's index is its new rank. + // Classes in (kind, rank, path) order, so a class's index orders it + // within its kind. for (class, slot) in keyed.values_mut().enumerate() { *slot = class; } @@ -429,13 +430,13 @@ fn ordering_terms_per_path(l: &mut Leakage, _: &mut TestRng) { for row in &mut l.rows { for entry in &mut row.entries { if let Some(op) = &mut entry.ordering { - op.class = keyed[&(old_ranks[op.class], entry.selector)]; + op.class = keyed[&(op.kind, old_ranks[op.class], entry.selector)]; } } } for query in &mut l.queries { if let QueryLeakage::Order { selector, bound } = query { - bound.class = keyed[&(old_ranks[bound.class], *selector)]; + bound.class = keyed[&(bound.kind, old_ranks[bound.class], *selector)]; } } l.ordering_ranks = (0..keyed.len()).collect(); @@ -456,6 +457,12 @@ fn ordering_term_length_constant(l: &mut Leakage, _: &mut TestRng) { } } +/// A number's and a string's terms compare in jsonb's order, numbers first, +/// whatever the key: the order the page says the scheme does not keep. +fn mixed_kinds_jsonb_order(l: &mut Leakage, _: &mut TestRng) { + l.kind_order = KindOrder::Fixed(Placement::Above); +} + /// The terms show their order and lengths and no shared prefix beyond what /// those force: what the page would say without "the term also shows how /// long that common prefix is". @@ -892,6 +899,7 @@ mutants!( ordering_terms_all_distinct => "OrderingLeak::class (\"which rows hold equal values\")", ordering_term_length_constant => "OrderingLeak::len (\"Its length is the leaf's length\")", ordering_terms_per_path => "OrderingLeak::class across paths (\"two ordered paths holding equal values carry equal terms\")", + mixed_kinds_jsonb_order => "Leakage::kind_order (\"the order between a number and a string depends on the column key and is not jsonb's\")", ordering_prefixes_hidden => "Leakage::ordering_prefixes (\"the term also shows how long that common prefix is\")", member_ranks_follow_key_order => "EntryLeakage::step (a member's ordinal is its keyed rank)", hit_rule_uniform => "EntryLeakage::rule (the member or element rule a walk's hit matches; added to the page by #1362)", diff --git a/packages/stack-encrypt/tests/json_leakage/profile.rs b/packages/stack-encrypt/tests/json_leakage/profile.rs index f071fa461..2f0e4ffe5 100644 --- a/packages/stack-encrypt/tests/json_leakage/profile.rs +++ b/packages/stack-encrypt/tests/json_leakage/profile.rs @@ -197,13 +197,57 @@ pub enum Step { Element(u64), } +/// The kind of leaf an ordering term is of. Each kind's terms are under a +/// key of its own: the order space's salt names its domain. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum OrderKind { + Number, + String, +} + +/// Where a column's key puts the strings' terms among the numbers', at an +/// ordered path that holds both. A CLLW term begins with a carry byte and +/// then `PRF(key, ε) + 128 × first bit`, and the two kinds' keys differ, +/// so under one key every string compares the same way with every negative +/// number (first bit 0) and the same way with every non-negative one (first +/// bit 1): the strings sort as one block, in one of three places. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Placement { + /// Below every number. + Below, + /// Above the negative numbers and below the non-negative ones. + Between, + /// Above every number: jsonb's order, which sorts numbers first. + Above, +} + +impl Placement { + pub const ALL: [Self; 3] = [Self::Below, Self::Between, Self::Above]; +} + +/// How a number's term and a string's compare. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum KindOrder { + /// The page: "the order between a number and a string depends on the + /// column key and is not jsonb's". The key picks a [`Placement`], once + /// per column, and the simulator does what the key does, drawing one. + ByKey, + /// A placement that does not depend on the key: what only a mutant + /// states. + Fixed(Placement), +} + /// The ordering term of one leaf, or of one query bound. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct OrderingLeak { /// "The order of that leaf's value among every row that holds one, and /// which rows hold equal values." Equal values share a class; the - /// classes' order is [`Leakage::ordering_ranks`]. + /// classes' order within their kind is [`Leakage::ordering_ranks`]. pub class: OrderClass, + /// Whether the leaf is a number or a string: "the terms also show which + /// leaves are numbers and which are strings", since each kind's terms + /// are under a key of its own and so begin with that key's bytes. + pub kind: OrderKind, /// "Its length is the leaf's length: one byte per bit of the value's /// order encoding plus one, so 65 bytes for every number and /// `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` @@ -400,14 +444,19 @@ pub struct Leakage { pub key_tag_len: usize, pub rows: Vec, pub queries: Vec, - /// The rank of each [`OrderClass`] among all of them: "the order of that - /// leaf's value among every row that holds one", and the bound's "place - /// in the order of the stored values at every ordered path" ("leaves at - /// two ordered paths compare with each other too, and two ordered paths - /// holding equal values carry equal terms"). + /// The rank of each [`OrderClass`] among the classes of its kind: "the + /// order of that leaf's value among every row that holds one", and the + /// bound's "place in the order of the stored values at every ordered + /// path" ("leaves at two ordered paths compare with each other too, and + /// two ordered paths holding equal values carry equal terms"; "within + /// each kind the order is the values'"). pub ordering_ranks: Vec, - /// For each two classes next to each other in rank order, `(lower, - /// higher)`, how many leading bits their order encodings share: "two + /// How the two kinds' terms compare ([`KindOrder`]). + pub kind_order: KindOrder, + /// For each two classes of one kind next to each other in rank order, + /// `(lower, higher)`, how many leading bits their order encodings + /// share. A number's term and a string's share none: their keys + /// differ. "Two /// values whose encodings share leading bits […] get terms that share /// leading bytes, […] so the term also shows how long that common /// prefix is". CLLW OPE writes one byte per bit, the PRF of the bits @@ -556,63 +605,70 @@ impl Ids { }; // One byte per bit of the order encoding, plus one. let len = match &key { - OrderKey::Number(_) => 8 * 8 + 1, + OrderKey::Number(..) => 8 * 8 + 1, OrderKey::String(encoded) => 8 * encoded.len() + 1, }; + let kind = match &key { + OrderKey::Number(..) => OrderKind::Number, + OrderKey::String(_) => OrderKind::String, + }; let class = self.ordering.id(key.clone()); if class == self.ordering_keys.len() { self.ordering_keys.push(key); } - OrderingLeak { class, len } + OrderingLeak { class, kind, len } } - /// The rank of every ordering class. The page does not say how a - /// number's term compares with a string's, so a column that orders both - /// is refused rather than guessed at. + /// The rank of every ordering class among the classes of its kind. pub fn ordering_ranks(&self) -> Vec { - let numbers = self - .ordering_keys - .iter() - .any(|k| matches!(k, OrderKey::Number(_))); - let strings = self - .ordering_keys - .iter() - .any(|k| matches!(k, OrderKey::String(_))); - assert!( - !(numbers && strings), - "the page does not say how a number's ordering term compares with a string's; \ - a world orders one kind of leaf" - ); - let mut order: Vec = (0..self.ordering_keys.len()).collect(); + let mut ranks = vec![0; self.ordering_keys.len()]; + for kind in [OrderKind::Number, OrderKind::String] { + for (rank, class) in self.of_kind(kind).into_iter().enumerate() { + ranks[class] = rank; + } + } + ranks + } + + /// The classes of `kind`, in their order. + fn of_kind(&self, kind: OrderKind) -> Vec { + let mut order: Vec = (0..self.ordering_keys.len()) + .filter(|&c| { + matches!( + (&self.ordering_keys[c], kind), + (OrderKey::Number(..), OrderKind::Number) + | (OrderKey::String(_), OrderKind::String) + ) + }) + .collect(); order.sort_by( |&a, &b| match (&self.ordering_keys[a], &self.ordering_keys[b]) { (OrderKey::Number(x), OrderKey::Number(y)) => { f64::from_bits(*x).total_cmp(&f64::from_bits(*y)) } (OrderKey::String(x), OrderKey::String(y)) => x.cmp(y), - _ => unreachable!("refused above"), + _ => unreachable!("one kind"), }, ); - let mut ranks = vec![0; order.len()]; - for (rank, class) in order.into_iter().enumerate() { - ranks[class] = rank; - } - ranks + order } - /// The leading bits each two classes next to each other in `ranks`' - /// order share ([`Leakage::ordering_prefixes`]). - pub fn ordering_prefixes(&self, ranks: &[usize]) -> BTreeMap<(OrderClass, OrderClass), usize> { - let mut by_rank: Vec = (0..ranks.len()).collect(); - by_rank.sort_by_key(|&class| ranks[class]); - by_rank - .windows(2) - .map(|pair| { - let shared = common_bits( - &self.ordering_keys[pair[0]].encoding(), - &self.ordering_keys[pair[1]].encoding(), - ); - ((pair[0], pair[1]), shared) + /// The leading bits each two classes of one kind next to each other in + /// its order share ([`Leakage::ordering_prefixes`]). + pub fn ordering_prefixes(&self) -> BTreeMap<(OrderClass, OrderClass), usize> { + [OrderKind::Number, OrderKind::String] + .into_iter() + .flat_map(|kind| { + self.of_kind(kind) + .windows(2) + .map(|pair| { + let shared = common_bits( + &self.ordering_keys[pair[0]].encoding(), + &self.ordering_keys[pair[1]].encoding(), + ); + ((pair[0], pair[1]), shared) + }) + .collect::>() }) .collect() } @@ -1114,12 +1170,12 @@ pub fn leak(column: &Column, clauses: Clauses, docs: &[Value], queries: &[Query] .iter() .map(|q| l_query(column, &mut ids, docs, q)) .collect(); - let ordering_ranks = ids.ordering_ranks(); Leakage { key_tag_len: KEY_TAG_LEN, rows, queries, - ordering_prefixes: ids.ordering_prefixes(&ordering_ranks), - ordering_ranks, + ordering_ranks: ids.ordering_ranks(), + kind_order: KindOrder::ByKey, + ordering_prefixes: ids.ordering_prefixes(), } } diff --git a/packages/stack-encrypt/tests/json_leakage/simulator.rs b/packages/stack-encrypt/tests/json_leakage/simulator.rs index 863a7cf5a..5a449fd81 100644 --- a/packages/stack-encrypt/tests/json_leakage/simulator.rs +++ b/packages/stack-encrypt/tests/json_leakage/simulator.rs @@ -35,11 +35,13 @@ use proptest::test_runner::TestRng; use serde_json::{json, Value}; use super::chain::{ - cllw_ope, element_token, member_token, parent_tag, self_tag, simulated_rank, Rule, Tag16, Tag8, + cllw_ope, element_token, member_token, parent_tag, self_tag, shared_bits, simulated_rank, Rule, + Tag16, Tag8, }; use super::observe::QueryKind; use super::profile::{ - GroupLeakage, Leakage, OrderingLeak, QueryLeakage, RowLeakage, Step, TokenName, TokenStep, + GroupLeakage, KindOrder, Leakage, OrderKind, OrderingLeak, Placement, QueryLeakage, RowLeakage, + Step, TokenName, TokenStep, }; /// The version byte `vitaminc-encrypt` writes at the front of every @@ -101,10 +103,17 @@ struct Simulator<'a> { } /// Build the rows' wire JSON and the queries' wire JSON from `leakage` and -/// `rng` alone. -pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(QueryKind, Value)>) { +/// `rng` alone. Where the profile leaves the place of the strings' ordering +/// terms among the numbers' to the column's key and the column orders both, +/// they go at `placement`, or at one drawn from `rng` for `None`: the +/// calibration passes each placement a key could pick in turn. +pub fn simulate( + leakage: &Leakage, + rng: &mut TestRng, + placement: Option, +) -> (Vec, Vec<(QueryKind, Value)>) { let rank_key = random(rng); - let ordering = ordering_terms(leakage, &random(rng)); + let ordering = ordering_terms(leakage, rng, placement); let mut sim = Simulator { leakage, rng, @@ -129,16 +138,24 @@ pub fn simulate(leakage: &Leakage, rng: &mut TestRng) -> (Vec, Vec<(Query (rows, queries) } -/// Every ordering class's term: a CLLW OPE term ([`cllw_ope`]) under `key` -/// over a code of the class's bit length, the codes built in rank order from -/// the profile's shared prefixes alone ([`successor`]). So the terms -/// compare as the ranks do, are equal where the classes are, are as long as -/// the profile says, and share leading bytes exactly as far as the profile -/// says the values' encodings share leading bits. -fn ordering_terms(leakage: &Leakage, key: &[u8; 32]) -> HashMap> { - let mut bits: BTreeMap = BTreeMap::new(); +/// Every ordering class's term: a CLLW OPE term ([`cllw_ope`]) under its +/// kind's key over a code of the class's bit length, each kind's codes built +/// in rank order from the profile's shared prefixes alone ([`successor`]). +/// So within a kind the terms compare as the ranks do, are equal where the +/// classes are, are as long as the profile says, and share leading bytes +/// exactly as far as the profile says the values' encodings share leading +/// bits. Where a column orders both kinds, the two keys are drawn until the +/// strings' terms fall at `placement` among the numbers' ([`placed`]): the +/// one the profile fixes, else the one given, else one drawn at random, as +/// a column's key picks one. +fn ordering_terms( + leakage: &Leakage, + rng: &mut TestRng, + placement: Option, +) -> HashMap> { + let mut classes: BTreeMap = BTreeMap::new(); let mut note = |op: &OrderingLeak| { - let _ = bits.insert(op.class, op.len - 1); + let _ = classes.insert(op.class, (op.kind, op.len - 1)); }; for row in &leakage.rows { for op in row.entries.iter().filter_map(|e| e.ordering.as_ref()) { @@ -150,24 +167,83 @@ fn ordering_terms(leakage: &Leakage, key: &[u8; 32]) -> HashMap> note(bound); } } - let mut classes: Vec = bits.keys().copied().collect(); - classes.sort_by_key(|&class| (leakage.ordering_ranks[class], class)); - let mut terms = HashMap::new(); - let mut previous: Option<(usize, Vec)> = None; - for class in classes { - let n = bits[&class]; - let code = match &previous { - None => vec![false; n], - Some((below, code)) => successor( - code, - n, - leakage.ordering_prefixes.get(&(*below, class)).copied(), - ), - }; - let _ = terms.insert(class, cllw_ope(key, &code)); - previous = Some((class, code)); + let mut codes: HashMap)> = HashMap::new(); + for kind in [OrderKind::Number, OrderKind::String] { + let mut mine: Vec = classes + .iter() + .filter(|(_, (k, _))| *k == kind) + .map(|(&class, _)| class) + .collect(); + mine.sort_by_key(|&class| (leakage.ordering_ranks[class], class)); + let mut previous: Option<(usize, Vec)> = None; + for class in mine { + let n = classes[&class].1; + let code = match &previous { + None => vec![false; n], + Some((below, code)) => successor( + code, + n, + leakage.ordering_prefixes.get(&(*below, class)).copied(), + ), + }; + let _ = codes.insert(class, (kind, code.clone())); + previous = Some((class, code)); + } } - terms + let both = [OrderKind::Number, OrderKind::String] + .iter() + .all(|kind| codes.values().any(|(k, _)| k == kind)); + let target = match leakage.kind_order { + KindOrder::Fixed(fixed) => fixed, + KindOrder::ByKey => { + placement.unwrap_or_else(|| Placement::ALL[(rng.next_u32() % 3) as usize]) + } + }; + for _ in 0..100_000 { + let keys: [[u8; 32]; 2] = [random(rng), random(rng)]; + let terms: HashMap> = codes + .iter() + .map(|(&class, (kind, code))| { + let key = &keys[usize::from(*kind == OrderKind::String)]; + (class, cllw_ope(key, code)) + }) + .collect(); + if !both || placed(&terms, &codes, target) { + return terms; + } + } + panic!("no two keys put the strings {target:?} in 100,000 tries") +} + +/// Whether the strings' terms sit at `placement` among the numbers': each +/// string below or above every number whose code begins with 0 (a negative +/// one, or every number where none is negative) as `placement` says, and +/// likewise for those whose code begins with 1, and sharing no leading +/// byte with any of them, as a number's and a string's term under two keys +/// share none. A pair of keys under which some string and some number tie +/// to within a carry at the front (about one pair in forty) is refused: a +/// real key like that makes a few such pairs compare by their later bytes, +/// which no clause states (`mod.rs`, "Out of reach"). +fn placed( + terms: &HashMap>, + codes: &HashMap)>, + placement: Placement, +) -> bool { + let of = |kind: OrderKind| { + codes + .iter() + .filter(move |(_, (k, _))| *k == kind) + .map(|(class, (_, code))| (&terms[class], code[0])) + }; + of(OrderKind::String).all(|(string, _)| { + of(OrderKind::Number).all(|(number, high)| { + let above = match (placement, high) { + (Placement::Below, _) | (Placement::Between, true) => false, + (Placement::Above, _) | (Placement::Between, false) => true, + }; + (string > number) == above && shared_bits(string, number) == 0 + }) + }) } /// The least `n`-bit code above `below` that shares `shared` leading bits From 7c2f4701afe22728195218db293ee7ddbdbb6523 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 20:47:29 +1100 Subject: [PATCH 17/19] test(stack-encrypt): hold the page's "equality is as a 64-bit float" sentence The owner's decision: the page says that a number's equality under its ordering term is as a 64-bit float. So two different integers past 2^53 can round to one float, share a term and tie under ORDER BY, while their value selectors differ. The profile already modelled this (the term is derived from as_f64), and the ordered-numbers column holds 9007199254740992 and 9007199254740993. What holds the sentence is a reading, NumberOrder::Exact, under which every distinct number gets its own term. It is a reading rather than a mutant in mutants.rs because a mutant transforms the leakage and never sees the values it would have to split. given_integers_ordered_exactly shows that ordering_term_order tells it from the real scheme on that column. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql/docs/reference/json-tree-leakage.md | 2 +- .../tests/json_leakage/calibration.rs | 36 +++++++++++++++- .../stack-encrypt/tests/json_leakage/mod.rs | 7 ++-- .../tests/json_leakage/profile.rs | 42 +++++++++++++++---- 4 files changed, 73 insertions(+), 14 deletions(-) diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index 5ace4d7fc..ce1a83f82 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -22,7 +22,7 @@ Someone who holds the column and no key, such as a database administrator, a bac | `p`, the parent tag below the chain's heads | Nothing beyond the entry order on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes, and it does not say which entry is the parent. Which entries carry no `p` does show: with the sealed root on, those are the root's direct members, and since everything beneath a member follows it, **the node count of each top-level field's subtree** can be read off a single row exactly. With the sealed root off, only the root lacks one. | | `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | -| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column for each kind of leaf, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. At an ordered path whose values are numbers in some rows and strings in others, the order between a number and a string depends on the column key and is not jsonb's (jsonb sorts numbers before strings); within each kind the order is the values'. The two kinds' terms are under different keys, so the terms also show which leaves are numbers and which are strings, and the strings sort together, as one block that the key places below the numbers, between the negative and the non-negative ones, or above them. By default no path is ordered; see the options below. | +| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column for each kind of leaf, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. A number's equality is as a 64-bit float: two different integers past 2⁵³ can round to one float, and then they share a term and tie under `ORDER BY` while their value selectors differ. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. At an ordered path whose values are numbers in some rows and strings in others, the order between a number and a string depends on the column key and is not jsonb's (jsonb sorts numbers before strings); within each kind the order is the values'. The two kinds' terms are under different keys, so the terms also show which leaves are numbers and which are strings, and the strings sort together, as one block that the key places below the numbers, between the negative and the non-negative ones, or above them. By default no path is ordered; see the options below. | | `h`, the header | The keyset the data key was minted under, the data key's identifier (unique to the row), and whether the root is sealed. `v` and `i` are the version and the table and column identifier every EQL value carries. | Not revealed at rest: any value's or key name's text, and, except through the ordering terms at the paths the index orders, whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the byte order of sibling key names. diff --git a/packages/stack-encrypt/tests/json_leakage/calibration.rs b/packages/stack-encrypt/tests/json_leakage/calibration.rs index 17548a24b..6def548e1 100644 --- a/packages/stack-encrypt/tests/json_leakage/calibration.rs +++ b/packages/stack-encrypt/tests/json_leakage/calibration.rs @@ -15,8 +15,8 @@ use super::distinguishers::{self, walk_positions}; use super::mutants; use super::observe::{ObservedQuery, ObservedWorld}; use super::profile::{ - flatten, leak, Clauses, Column, KeyOrder, KindOrder, Leakage, OrderKind, Placement, Query, - QueryLeakage, StringOrder, + flatten, leak, Clauses, Column, KeyOrder, KindOrder, Leakage, NumberOrder, OrderKind, + Placement, Query, QueryLeakage, StringOrder, }; use super::{real, simulator}; use crate::common::stack_cipher; @@ -269,6 +269,38 @@ mod given_strings_ordered_by_their_bytes { } } +mod given_integers_ordered_exactly { + use super::*; + + /// The page says a number's equality under its term "is as a 64-bit + /// float": 9007199254740992 and 9007199254740993 round to one float and + /// share a term. Read as every distinct number its own term, the profile + /// is told from the real scheme on the column that holds both. + #[test] + fn the_ordering_terms_tell_it_apart() { + let runtime = runtime(); + let cipher = runtime.block_on(stack_cipher()); + let clauses = Clauses { + number_order: NumberOrder::Exact, + ..Clauses::default() + }; + let world = handcrafted() + .into_iter() + .find(|w| w.name == "numbers and strings at one ordered path") + .expect("the corpus has it"); + let real = real_world(&runtime, &cipher, &world); + let leakage = leak(&world.column, clauses, &world.docs, &world.queries); + let fired: Vec<&str> = separating(&features(&real), &leakage, &mut rng()) + .into_iter() + .map(|(n, _, _)| n) + .collect(); + assert!( + fired.contains(&"ordering_term_order"), + "two integers that round to one float share a term: {fired:?}" + ); + } +} + mod given_the_jsonb_reading_of_the_key_order { use super::*; diff --git a/packages/stack-encrypt/tests/json_leakage/mod.rs b/packages/stack-encrypt/tests/json_leakage/mod.rs index 8a17a2ba3..2ac95900c 100644 --- a/packages/stack-encrypt/tests/json_leakage/mod.rs +++ b/packages/stack-encrypt/tests/json_leakage/mod.rs @@ -76,9 +76,10 @@ //! [`profile::Clauses::as_published_in_1363`] turns it off, and a test //! holds that the result is then told from the real scheme, by the //! distinguisher that names it and the three its absence breaks. -//! - **Two readings.** The page states the key order (UTF-8 byte order) -//! and that strings order folded; a profile read the other way on either -//! is told from the real scheme, so both sentences are load-bearing. +//! - **Three readings.** The page states the key order (UTF-8 byte order), +//! that strings order folded, and that a number's equality under its term +//! is as a 64-bit float; a profile read the other way on any of them is +//! told from the real scheme, so each sentence is load-bearing. //! - **The calibration.** Each mutant profile drops one clause. The test //! expects some distinguisher to tell each mutant's world from the real //! one, and prints a table of which (run with `--no-capture`). A mutant diff --git a/packages/stack-encrypt/tests/json_leakage/profile.rs b/packages/stack-encrypt/tests/json_leakage/profile.rs index 2f0e4ffe5..bffc691b1 100644 --- a/packages/stack-encrypt/tests/json_leakage/profile.rs +++ b/packages/stack-encrypt/tests/json_leakage/profile.rs @@ -130,7 +130,19 @@ pub enum StringOrder { Bytes, } -/// The clauses of the profile that are on or off as a whole, and the two +/// What a number's ordering term tells equal by. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum NumberOrder { + /// The page: equality "is as a 64-bit float", so two integers past + /// 2^53 that round to one float share a term (the term is derived from + /// `as_f64`). + Float, + /// Every distinct number its own term, as a reader could take "equal + /// values carry equal terms" to mean. + Exact, +} + +/// The clauses of the profile that are on or off as a whole, and the /// readings the page could leave open. The default is what the code needs. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct Clauses { @@ -139,6 +151,8 @@ pub struct Clauses { pub key_order: KeyOrder, /// How a string's ordering term orders. pub string_order: StringOrder, + /// What a number's ordering term tells equal by. + pub number_order: NumberOrder, /// **Added to the page by #1362.** The rule a node's token takes under /// its parent's, member or element, which the database learns for every /// node a walk reaches: it tests the member rule against the hit's self @@ -156,6 +170,7 @@ impl Default for Clauses { Self { key_order: KeyOrder::Utf8Bytes, string_order: StringOrder::Folded, + number_order: NumberOrder::Float, hit_rule: true, } } @@ -449,7 +464,8 @@ pub struct Leakage { /// bound's "place in the order of the stored values at every ordered /// path" ("leaves at two ordered paths compare with each other too, and /// two ordered paths holding equal values carry equal terms"; "within - /// each kind the order is the values'"). + /// each kind the order is the values'"). A number's equality "is as a + /// 64-bit float". pub ordering_ranks: Vec, /// How the two kinds' terms compare ([`KindOrder`]). pub kind_order: KindOrder, @@ -499,7 +515,10 @@ impl Interner { /// derived from `as_f64`); strings by their order encoding's bytes. #[derive(Clone, Debug, PartialEq, Eq, Hash)] enum OrderKey { - Number(u64), + /// The `f64`'s bits, and under [`NumberOrder::Exact`] an integer's own + /// value beside them, which keeps integers that round to one float + /// apart. + Number(u64, Option), String(Vec), } @@ -509,7 +528,7 @@ impl OrderKey { /// string's marker and folded text as [`Ids::ordering`] keeps them. fn encoding(&self) -> Vec { match self { - Self::Number(bits) => vitaminc_ore::order( + Self::Number(bits, _) => vitaminc_ore::order( vitaminc_ore::OrderTarget::F64, None, vitaminc_ore::Physical::Float(vitaminc_ore::Float::F64(f64::from_bits(*bits))), @@ -588,7 +607,14 @@ impl Ids { f != 0.0 || f.is_sign_positive(), "the corpus has no -0.0, whose term the page does not describe" ); - OrderKey::Number(f.to_bits()) + let exact = match self.clauses.number_order { + NumberOrder::Float => None, + NumberOrder::Exact => n + .as_i64() + .map(i128::from) + .or_else(|| n.as_u64().map(i128::from)), + }; + OrderKey::Number(f.to_bits(), exact) } Value::String(s) => OrderKey::String(match self.clauses.string_order { // The marker, then the folded text: the order encoding. @@ -643,9 +669,9 @@ impl Ids { .collect(); order.sort_by( |&a, &b| match (&self.ordering_keys[a], &self.ordering_keys[b]) { - (OrderKey::Number(x), OrderKey::Number(y)) => { - f64::from_bits(*x).total_cmp(&f64::from_bits(*y)) - } + (OrderKey::Number(x, i), OrderKey::Number(y, j)) => f64::from_bits(*x) + .total_cmp(&f64::from_bits(*y)) + .then(i.cmp(j)), (OrderKey::String(x), OrderKey::String(y)) => x.cmp(y), _ => unreachable!("one kind"), }, From e50c18bb5df4d36204a47d9dcda9190ccc228038 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 20:47:29 +1100 Subject: [PATCH 18/19] docs(eql): say the ordering terms' shared prefix is CLLW's known profile The owner's decision: the reworded prefix sentence stands, and the page says, after it, that this is CLLW's known profile and that a scheme which hides the prefix is planned for a later version. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/eql/docs/reference/json-tree-leakage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index ce1a83f82..a5130d0e5 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -22,7 +22,7 @@ Someone who holds the column and no key, such as a database administrator, a bac | `p`, the parent tag below the chain's heads | Nothing beyond the entry order on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes, and it does not say which entry is the parent. Which entries carry no `p` does show: with the sealed root on, those are the root's direct members, and since everything beneath a member follows it, **the node count of each top-level field's subtree** can be read off a single row exactly. With the sealed root off, only the root lacks one. | | `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | -| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column for each kind of leaf, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. A number's equality is as a 64-bit float: two different integers past 2⁵³ can round to one float, and then they share a term and tie under `ORDER BY` while their value selectors differ. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. At an ordered path whose values are numbers in some rows and strings in others, the order between a number and a string depends on the column key and is not jsonb's (jsonb sorts numbers before strings); within each kind the order is the values'. The two kinds' terms are under different keys, so the terms also show which leaves are numbers and which are strings, and the strings sort together, as one block that the key places below the numbers, between the negative and the non-negative ones, or above them. By default no path is ordered; see the options below. | +| `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column for each kind of leaf, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. A number's equality is as a 64-bit float: two different integers past 2⁵³ can round to one float, and then they share a term and tie under `ORDER BY` while their value selectors differ. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. This is CLLW's known profile; a scheme that hides the prefix is planned for a later version. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. At an ordered path whose values are numbers in some rows and strings in others, the order between a number and a string depends on the column key and is not jsonb's (jsonb sorts numbers before strings); within each kind the order is the values'. The two kinds' terms are under different keys, so the terms also show which leaves are numbers and which are strings, and the strings sort together, as one block that the key places below the numbers, between the negative and the non-negative ones, or above them. By default no path is ordered; see the options below. | | `h`, the header | The keyset the data key was minted under, the data key's identifier (unique to the row), and whether the root is sealed. `v` and `i` are the version and the table and column identifier every EQL value carries. | Not revealed at rest: any value's or key name's text, and, except through the ordering terms at the paths the index orders, whether two different paths hold the same value. Revealed at rest, by the entry order, although no field stores it: the shape of every row, the parent of every node, every array's length, and the byte order of sibling key names. From fd229008d5c7394abdf4db811e9050470737d136 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 20:47:43 +1100 Subject: [PATCH 19/19] docs(plans): record the owner's decisions in the leakage note The note's review section listed mixed kinds at an ordered path and f64 equality as left to the owner. Both are now decided, stated on the page and held by the profile, so the section says what was decided and how the test holds it, with the new totals. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- docs/plans/2026-10-25-json-tree-leakage-before-and-after.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md index 990706713..d950adc6c 100644 --- a/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md +++ b/docs/plans/2026-10-25-json-tree-leakage-before-and-after.md @@ -219,6 +219,6 @@ A review of the port (2026-10-25) found one disclosure that both profiles missed - **New checks on the method.** `cross_kind_equalities` compares values across kinds (a token equal to a selector, say), which every other feature names apart. `term_fields` checks the field names inside `q` and every query term, which the parser would otherwise drop unread. `token_derivability` tries the element rule up to index 64 or the longest token list, not only below 4. - **Four clauses with no mutant** now have one: `key_tag_shortened`, `order_selector_fresh`, `element_ordinals_reversed`, `parents_lifted`. - **The corpus.** The random columns hold big integers, floats, strings that fold, and non-scalars at ordered paths. They order numbers or strings, and run 256 cases instead of 48. A fifth handcrafted column orders numbers of both signs and many magnitudes, and holds two integers past 2⁵³ that share a term. -- **Left to the owner:** a path ordered over numbers and strings both. Their terms are under different keys, so how they compare is set by the column key, and the profile refuses such a column; `mod.rs` records what was measured. Also, a number's equality under its term is equality as an `f64`, which the page does not say. +- **Decided by the owner, and now on the page and in the profile.** At a path ordered over numbers and strings both, the order between a number and a string depends on the column key and is not jsonb's; within each kind the order is the values'. The two kinds' terms are under different keys (the order space's salt names its domain), so they show which leaves are numbers and which are strings, and the strings sort as one block that the key places below the numbers, between the negative and non-negative ones, or above them. The profile states the kinds and the within-kind order and leaves the placement to the key; the calibration simulates each placement, and the mutant fixing jsonb's numbers-first order (`mixed_kinds_jsonb_order`) is caught. A number's equality under its term is as a 64-bit float, held by the reading `NumberOrder::Exact`. The shared prefix is CLLW's known profile, and a scheme that hides it is planned for a later version. -After the review: 38 distinguishers, 49 mutants, 11 tests. +After the review and the owner's decisions: 38 distinguishers, 50 mutants, 12 tests.