feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards by coderdan · Pull Request #1363 · cipherstash/stack

coderdan · 2026-10-24T14:02:02Z

Summary

EQL v4's json_tree is the encrypted JSON column Stack Encrypt writes: one encrypted entry per JSON node, so PostgreSQL can answer containment, path extraction and "give me the subtree at this path" without the key. A subtree read sends the database a token, a keyed hash that lets it find a node's entries and the entries beneath it in every row.

Until now the client could only derive tokens for top-level fields, so reading $.profile.address.city sent $.profile's token, which exposes the structure of all of $.profile in every row, for as long as the column lives, and walked down through "gates". It also could not return the node itself, so {}, [], a scalar and a missing field all came back as "no rows" (#1356).

This PR implements the agreed design change (the design-doc section "Design change (2026-10-24)" on this PR's base, #1361): every token is now derivable from the path alone, every stored entry carries a self tag x by which the database finds a node, and wildcards are expanded on the client into concrete paths up to a bound. A read now sends exactly the token of the node it reads, returns that node, and answers the four cases the way jsonb does. The owner's decisions D1–D10 are folded into the design doc as "Decisions (2026-10-24)".

Changes

Stack Encrypt (packages/stack-encrypt)

Derivation: member token H16("t" ‖ t(parent) ‖ s) (no ordinal); element token keeps its index; x = H8("x" ‖ iv ‖ t) on every entry, the root and heads included. The parent tag p is unchanged and used for enumeration only. A wire entry is {s, c, x, p?, op?}. JsonHeader::FORMAT_VERSION and v are unchanged (D5).
JsonSubtreeKey is a list of tokens: one per concrete path, plus a probe per wildcard level at the bound N (default 64, JsonSubtreeKey::DEFAULT_BOUND, set per read with KeysetCipher::json_subtree_key_with_bound). tokens(), token_bytes(), probes(), bound(). Gates are gone.
Reads open as Value, Option<Value> (None means no node) and Vec<Value> (one value per concrete path). A probe hit fails with the new Error::JsonWildcardTruncated { bound }, walked or via the sealed-root fallback (D3, D4).
Row check (D8): the depth-0 row's selector and decrypted label against its token's path; deeper rows at consecutive j with the parent's path's selector plus their label; every row's x against the recomputed token (a consistency check only, and documented as such). The p check is dropped.
$ is a read under the sealed root (D6); TermError::JsonSealedRoot is removed (it had no other use); the new TermError::JsonWildcardTooWide refuses a read past 65,536 tokens.
JsonDocument::find / children / subtree(&[tokens]), the in-crate model of the SQL, follow the new rules.
CHANGELOG (unreleased entry rewritten), fuzz seeds regenerated with x, the stale mutants exclusion for the removed (0, true) arm dropped, proptest added as a dev-dependency (already in the lockfile).

EQL SQL (packages/eql/src/v3/json_tree)

eql_v4.json_tree_subtree(e, tokens text[], budget integer DEFAULT 64); json_tree_subtree_at removed (D9). The budget counts rows across all tokens; the recursion's LIMIT is entries × tokens, and the comment "The bound" is rewritten.
json_tree_children(e, token) and json_tree_child_count(e, token) lose gates; the count is NULL for an absent node and for the sealed root (D7).
New internals: json_tree_self_tag, json_tree_member_token, json_tree_element_token, json_tree_hit_token (the member-or-element choice by x, D2), json_tree_by_self; json_tree_child_token removed. The entry CHECK requires x.

eql-bindings: JsonTreeEntry gains x; JsonTreeSubtree opens to Option<Value>; docs and CHANGELOG.

Docs: the design doc section gets "Decisions (2026-10-24)"; the leakage page's read bullets and "What is authenticated" are updated, including the two allowances (another row; the same selector under another element) and that a database can withhold rows.

Changeset: @cipherstash/eql minor.

Verification

From the worktree root:

mise x --env test -- cargo nextest run -p stack-encrypt --all-features → 853 tests run: 853 passed, 1 skipped (includes the trybuild ui binary)
mise x --env test -- cargo test -p stack-encrypt --doc --all-features (what test:doc:stack-encrypt runs) → 50 passed; 0 failed
cargo clippy -p stack-encrypt --all-features --all-targets -- -D warnings → exit 0
cargo fmt --check → exit 0 (root and packages/eql)
RUSTDOCFLAGS="-D warnings" cargo doc -p stack-encrypt --no-deps --all-features (what doc:stack-encrypt runs) → no warnings
pnpm run test:scripts (.cargo/mutants.toml changed) → Test Files 81 passed (81), Tests 1432 passed | 29 skipped. A first run alongside the mutants job timed out two unrelated tests at 5 s; the rerun on an idle machine passed.
mise run fuzz:json-wire -- -runs=0 (seed replay, as CI runs it) → Done 6 runs, no crash, with the regenerated seeds.

From packages/eql:

cargo test --locked -p eql-bindings --all-features → every suite ok
mise run test:encryption → exit 0 (both feature sets, the encryption tests, both clippy runs)
mise run test:encryption:postgres against a disposable PostgreSQL 17 container on port 5455 → exit 0; the json_tree suite 11 passed
mise run types:check → exit 0 (run after committing: before that it diffs the hand-written json_tree.rs)
mise run docs:validate:source, test:docs_v3_grep, test:self_contained_v3, test:installer_complete, test:symbol_order_v3, test:v4_isolation, test:build_ordering_helpers, test:doc-anchors → all exit 0
tasks/docs/validate/documented-sql.sh against the same container → Validated: 290 Errors: 0

Mutants: mise x --env test -- cargo mutants --in-diff <diff vs docs/json-tree-derivable-tokens> -p stack-encrypt -j 4 (settings from .cargo/mutants.toml; a cold build in a fresh worktree) → 132 mutants tested in 40m: 1 missed, 91 caught, 40 unviable. The one miss (next = at + 1 → at * 1 in place, letting two rows match one token) is killed by one_token_answered_twice_is_refused in the last commit; applying the mutant by hand fails that test.

Not run: benchmarks (step 3 of the plan, for a separate agent; check:v4:json_tree in cipherstash/benches calls the gated walk and has to move to the token list). No new fuzz target: the existing json_wire_decode already fuzzes the JsonDocument and JsonNode decoders the client runs on every row the database returns, and is wired into fuzz:json-wire and the CI replay. A target for the row check itself would need a key source inside the fuzz crate.

New tests, by what they hold:

Pinned vectors (tests/json_tree_vectors.rs, fixture tests/fixtures/json_tree_vectors.json): s, x and p per entry, and the tokens for the root, a head, a member, an array, an element and a member under an element, sealed and open, under the deterministic key source. Regenerate only with STACK_ENCRYPT_UPDATE_JSON_TREE_VECTORS=1 (exactly 1).
Property tests (tests/json_properties.rs, proptest, and unit proptests in derive.rs): a member's token is independent of its siblings; element tokens are distinct by index and from every member (keys include "", "0", ., [*], ], $, quotes, a backslash and unicode); every path's read equals select, and the reads reassemble the document; a read then a step equals the longer read, and $ (walked, or sealed and opened whole) plus any path equals that path's read; -> equals the read for every scalar; children equals the child count, and is empty under a scalar; selectors, layout and tokens are independent of values and iv, with every x, p and c distinct; x on every entry and p absent only on the root and heads; a key sends exactly the tokens of the concrete paths (checked against a witness document through the stored self tags), the probes, and no ancestor's token; the serialized key is exactly those tokens.
Row check (tests/json_read_checks.rs): every refusal of D8 (wrong depth-0 selector, wrong label, non-consecutive j, wrong x at depth 0 and deeper, a splice, a gap, an orphan, a wrong depth, truncation, the root among rows, one token answered twice), the two allowed cases, a 127-deep document, and the four answers as Value, Option<Value> and Vec<Value>.
Wildcards: arrays at N − 1, N and N + 1 for N = 1, 3 and 64, walked and fallen back; nested wildcards send N² + N + 1; the expansion limit.
Size regression: x adds +9.8% characters on the nested benchmark shape and +9.4% on the flat one; the bounds are 8–12% and 7–12%.
PostgreSQL oracle (packages/eql/tests/encryption/tests/json_tree.rs): postgres_every_subtree_read_answers_as_jsonb_does (ported from fix(eql): return the node itself from every json_tree subtree read #1359 and extended with nested empties, degenerate roots, the stress keys, numbers, null against absent, a 300-key object and 40-level nesting; sealed and open; budgets 0, 1 and 64; Value, Option and Vec; children and child_count against jsonb_object_keys and jsonb_array_length, NULL included; -> against the read); postgres_wildcard_reads_answer_as_jsonb_path_query; postgres_reads_never_answer_wrongly_under_tampering (120 random rows, nine kinds of tampering; observed 512 refusals, 1,402 right answers, 98 withheld); the hostile tests rewritten for tokens and x (a forged x, another position, a moved node, swapped entries, truncated and reordered rows, the two allowances); and postgres_walks_do_not_multiply_on_forged_member_tags (new) beside the duplicated-entries and deep-chain tests.

Rewritten rather than kept: every test that used gates, json_tree_subtree_at, JsonSubtreeKey::token() or gates(), or expected a read to return only a node's descendants; the_sealed_root_has_no_token_and_its_fields_are_heads (the sealed root has a token now) and a_position_past_postgres_integer_is_refused_not_sent (no position reaches the server as an integer any more; the refusal stays, since no stored element sits there).

Review notes

Merge order. test: hold json_tree containment to jsonb with proptest, and restructure the JSON index tests #1345 conflicts with fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358 and merges first; this PR stacks on docs(plans): design json_tree tokens derivable from the path alone #1361, which stacks on fix(stack-encrypt): check that a json_tree read returns the entry asked for #1358. Nothing here touches test: hold json_tree containment to jsonb with proptest, and restructure the JSON index tests #1345 or fix(eql): return the node itself from every json_tree subtree read #1359; fix(eql): return the node itself from every json_tree subtree read #1359's mechanism is superseded, and its semantics and oracle test are carried over.
Where to look first. place and rebuild_walked in packages/stack-encrypt/src/json/derive.rs (the row check), subtree_key_for (the expansion), and json_tree_subtree in functions.sql.
Departures from the brief, with reasons.
- TermError::JsonWildcardTooWide (at most 65,536 tokens per read) is not in D1–D10; without it, three nested wildcards at the default bound would send 266,305 tokens.
- json_tree_child_count of the sealed root is NULL, not 0: its members chain from nothing, so its token cannot count them, and 0 would be a wrong answer that looks right.
- The tamper oracle accepts a third allowance beyond the two of D8: None, the database withholding rows. Nothing in a row says how many nodes a read should find, so a read answered with no rows opens as absent, and a wildcard read can lose elements the same way. The leakage page now says so. Closing it would need an authenticated count, for example reading the array node's child count beside a wildcard.
- The SQL LIMIT is entries × tokens rather than entries + 1: forged member tags can make one token's subtree be visited more than once. D9 asked to keep the per-row bound and reword it; this is that bound restated for many tokens.
- The wildcard oracle compares with strict jsonb_path_query_first per concrete path rather than one strict jsonb_path_query: strict mode aborts the whole query when one element does not fit a step (.x on a number), where the index answers per element.
Found, not fixed (pre-existing, out of scope). A stored number is parsed back with serde_json, which without float_roundtrip can return an f64 one unit in the last place off (-926809.5051008083 came back as …085). The new property tests found it, and now draw exactly representable floats; it deserves its own issue.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

A member's token depended on its ordinal, a keyed rank among its siblings' keys, so the client could not derive the token of any node below a top-level field. Reads therefore sent the head's token and walked the rest through gates: a read of `$.profile.address.city` handed the database a capability over all of `$.profile`, in every row, for the life of the column, and a read could not return the node it named, so `{}`, `[]`, a scalar and an absent field were one answer (#1356). A member's token is now H16("t" ‖ t(parent) ‖ s), an element's keeps its index, and every entry carries a self tag x = H8("x" ‖ iv ‖ t) by which the server finds the node a token names, the root and the heads included. Ordinals stay, in the parent tag, for enumeration only. The subtree key is the list of tokens of the concrete paths a path names: one for a path with no wildcard, `$` included under the sealed root, and for a wildcard the positions below a bound (64 by default, `json_subtree_key_with_bound`) and a probe at it. A probe that finds an element fails the read with `Error::JsonWildcardTruncated` rather than return part of the list, walked or fallen back to the sealed root, so a row's answer does not depend on the budget. The client checks each row: the depth-0 row's selector and decrypted label against its token's path, each deeper row at the next position with its parent's path's selector extended by its label, and every row's self tag against the token it recomputes. The self tag is unauthenticated, so that last check is for consistency; the selector, position and label are the binding. The rows open as `Value`, `Option<Value>` (None for no node) and `Vec<Value>`. `TermError::JsonSealedRoot` had no use left and is removed; `TermError::JsonWildcardTooWide` refuses a read past 65,536 tokens. The derivations are pinned under the deterministic key source, and property tests cover the read's metamorphic relations, the independence of tokens from values and iv, and exactly which tokens a key sends. JsonHeader::FORMAT_VERSION is unchanged: the family is unreleased. Refs #1360 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

With every token derivable from its node's path, the server no longer walks from a head through gates to reach a node: it is handed the token of exactly the node read, finds it by its self tag, and returns it at d = 0 with everything beneath it. `json_tree_subtree(doc, tokens text[], budget DEFAULT 64)` replaces `json_tree_subtree(doc, token, budget)` and `json_tree_subtree_at`, and answers `{}`, `[]`, a scalar and an absent node as four answers, as `jsonb` does (#1356). Under the member rule a hit's token no longer says whether it is a member or an element, so the walk keeps whichever rule's self tag matches the hit's `x`. The budget counts rows across the tokens; the recursion's LIMIT is the entry count times the token count, since forged member tags can make one subtree be visited twice. `json_tree_child_count` is NULL for a token that names no node, and for the sealed root, whose members chain from nothing. The entry validator requires `x`; `JsonTreeEntry` gains it and `JsonTreeSubtree` opens to `Option<Value>`. The PostgreSQL oracle holds every read with no wildcard to strict `jsonb_path_query_first`, wildcard reads to `jsonb_path_query` with truncation past the bound, the child functions to `jsonb_object_keys` and `jsonb_array_length`, and reads of randomly tampered rows to the right answer, a refusal or a documented allowance. Closes #1356 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

The design change section left eight questions open; the owner settled them before the implementation, and the section now records them as "Decisions (2026-10-24)": keep the consecutive-position check beside the self-tag check and drop the parent-tag one, choose the member or element rule by the self tag with no stored kind bit, send probes as ordinary tokens, apply the bound in the sealed-root fallback, edit the derivation in place without a version bump, allow `$` under the sealed root, return NULL for an absent node's child count, and drop the gates. The leakage page's read and authentication sections now say what the check binds and what it leaves to the database by design: the row an entry comes from, the element a node at a selector sits under, and whether rows were withheld. Refs #1360 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

cargo-mutants found that nothing failed when a read's rows could match the same token more than once: `next = at + 1` survived as `at * 1`. A server that rewrites a second node's self tag to the first token's would then have two genuine nodes at one selector answer one token. The test forges exactly that and expects the refusal. Refs #1360 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

changeset-bot · 2026-10-24T14:02:08Z

🦋 Changeset detected

Latest commit: 683802b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Minor
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-10-24T14:02:12Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-24T14:22:18Z

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
92	0	40	0

Every mutant in the changed lines was caught by a test.

cipherstash/stack#1363 changes the json_tree stored form (a self tag on every entry, ordinal-free member tokens, reads by token list, wildcards expanded on the client). Deciding whether to ship it needs the new form measured against its base on the same documents and the same machine, with the noise the comparison carries stated. The crate now builds either format from one source, by feature: `derivable` (default, stack 683802bcc) or `gates` (its base, edabee625), each a git dependency pinned by rev and renamed so both resolve in one Cargo.lock. What differs between them is confined to `ReadKey` in lib.rs. The plan builder's rename (`encrypt_with_layout`) is picked up. - dt_bench: `plan` derives each format's scenarios against its own database; `time` runs both plans from one process, alternating which format goes first, per call and per document in-server. - check_json_tree_v4 runs in both formats and opens every read through the key that derived it; it now also covers deep member paths, positions, wildcards at a bound, `$`, and the open-root and wide tables. - Two corpora for what the others cannot show: the nested corpus with the root open, and a wide one (300 members, 300 elements). - The ingest reports seal and INSERT time separately. - mise tasks bench:dt:{build,ingest,check,plan,time} and report:dt. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

The measurement cipherstash/stack#1363 asked for, so the owner can decide whether to ship the format change: stack 683802bcc against its base edabee625, the same documents in two databases on one cluster, timed interleaved, 10,000 rows in full and 100,000 rows for sizes and the headline reads. Wildcard-free deep paths, single-level wildcards and wide enumeration got faster. Whole-subtree reads, the budget fallback, a full walk of a wide array, a two-level wildcard at N = 64 and json_tree_child_count on a small node got slower: the per-hit self-tag test, and a second whole-document map built on every read. Stored documents are 11% larger on disk (nested) and 7% (flat). Both formats pass every correctness check at both tiers. report/JSON_TREE_DERIVABLE_TOKENS.md holds the measurement. The JSON_TREE_COMPARISON.md Results tables are regenerated on the new format (the SteVec columns are the second measurement's, not re-run), and a fifth-measurement section there summarises it. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

coderdan · 2026-10-24T16:25:49Z

Benchmarks: this PR (`683802b`) against its base (`edabee6`)

I measured the derivable-token format against the gated walk on the same documents, on the same machine. Each format had its own database on one PostgreSQL 17.10 cluster, and the two were timed interleaved. Full report, with methodology, the noise floor and the EXPLAIN plans: report/JSON_TREE_DERIVABLE_TOKENS.md (benches commit 3c837be). Both formats pass every correctness check at 10,000 and 100,000 rows: every timed read was decrypted and compared with the plaintext.

Summary

Better

Deep member paths: −22% for 4 steps. They send one token where the base sent a head token plus 3 gates.
Single-level wildcards: −34% to −58% at N = 8, 64 and 256.
Two-level wildcard at N = 8: −80%.
json_tree_children and json_tree_child_count on a 300-wide node: −39% to −49%.
$ is now readable under the sealed root.
Every read now returns the node itself (json_tree: a subtree read of {}, [], a scalar or a missing field all return no rows, so the client cannot tell them apart #1356).

Worse

Whole-subtree reads: +38% to +46% per call, +45% to +59% in-server.
Full walk of a 300-element array: +78%.
The budget fallback: +51%.
Two-level wildcard at N = 64 (4,161 tokens): +48%.
json_tree_child_count on a small node: +59%.
Stored nested documents: +11% on disk.
Ingest: +3% to +9%.
->, ordering and recheck-heavy containment on the nested corpus: +2% to +10%, in step with the larger document.

Unchanged within noise

Member paths 1 step below a field.
Flat and selective containment.
-> on the flat corpus.
json_tree_children on a small node.
GIN size and build time.
Every outer query plan.

Where the cost comes from

The self-tag test on every hit. In isolation, json_tree_hit_token costs 15.5 µs when the member test hits and 38.2 µs when it falls back to the element rule. The base's json_tree_child_token cost 7.8 µs. Inside a walk the wide corpus puts the extra at about 36 µs per member and 80 µs per element.
A second whole-document map on every read. json_tree_by_self costs 0.41 ms per nested document.
json_tree_child_count builds that map twice: once to test that the node exists, then again inside json_tree_children. Checking existence against the map json_tree_children already builds would remove about 1 ms per call.

Stored size

corpus	entries / doc	chars / doc	`pg_column_size` / doc	heap incl. TOAST	containment GIN
nested, 10k	217	49,042 → 53,603 (+9%)	31,714 → 35,256 B (+11%)	320 → 355 MiB (+11%)	39.9 MiB, unchanged
nested, 100k	217	49,003 → 53,561 (+9%)	31,691 → 35,228 B (+11%)	3,193 → 3,543 MiB (+11%)	358.1 MiB, unchanged
flat, 10k	5	1,279 → 1,384 (+8%)	1,398 → 1,502 B (+7%)	16 → 16 MiB (+0%)	1.0 MiB, unchanged
wide (300 + 300), 1k	603	125,967 → 138,630 (+10%)	91,195 → 99,998 B (+10%)	91 → 100 MiB (+10%)	—

Each entry gains 21 characters of ::text and about 16 bytes on disk. The flat heap does not move, because a page still holds 5 rows.

Ingest (seconds; 10k rows is the median of 3 interleaved runs, 100k one run)

corpus	seal: gates → derivable	INSERT: gates → derivable
nested 10k	8.37 → 8.75 (+5%)	23.68 → 25.62 (+8%)
nested 100k	80.00 → 82.73 (+3%)	224.96 → 245.13 (+9%)
flat 10k	0.29 → 0.31 (+5%)	1.29 → 1.35 (+5%)
flat 100k	2.83 → 2.95 (+4%)	12.28 → 12.95 (+5%)

Reads with no wildcard (nested, 10,000 rows)

These are per-call client times by id over 600 calls (200 rows × 3 passes), which include about 0.1 ms of round trip. The in-server column is one query over 500 documents.

read	sent (gates → derivable)	gates, median (p95) ms	derivable, median (p95) ms	per call	in-server per doc
`$.user.contact.emergency_contact.name` (4 steps)	1 tok + 3 gates → 1 tok	2.48 (2.76)	1.93 (2.37)	-22%	-27%
`$.company.name`	1 tok + 1 gate → 1 tok	1.97 (2.27)	1.98 (2.42)	+0%	+3%
`$.orders[2].status`	1 tok + 1 gate → 1 tok	1.88 (2.13)	1.95 (2.41)	+4%	+7%
`$.orders[2]` (object, 24 entries)	1 tok → 1 tok	3.57 (4.92)	5.08 (7.13)	+42%	+52%
`$.company` (whole object)	1 tok → 1 tok	2.00 (2.29)	2.81 (3.34)	+41%	+59%
`$.user` (whole object)	1 tok → 1 tok	2.54 (2.84)	3.52 (4.01)	+38%	+49%
`$.orders`, budget 0 (176 entries)	1 tok → 1 tok	18.1 (26.2)	25.9 (37.8)	+43%	+46%
`$.orders`, default budget (falls back to the sealed root)	1 tok → 1 tok	6.36 (6.70)	9.62 (10.1)	+51%	+56%
`$`, root sealed	refused → 1 tok	—	1.97 (2.42)	new	new
`$`, root open (212 entries)	1 tok → 1 tok	21.1 (29.2)	30.7 (42.2)	+46%	+45%
`$.first_name`, flat (gates returns no row, #1356)	1 tok → 1 tok	0.62 (0.75)	0.65 (0.78)	+5%	+80%

Wildcard reads (nested, 10,000 rows)

The base sends the same head token and gates at every N. 33% of rows hold more than 8 orders, so at N = 8 the derivable read refuses on those rows (JsonWildcardTruncated), and that is the p95.

read	sent (gates → derivable)	gates, median (p95) ms	derivable, median (p95) ms	per call	in-server per doc
`$.orders[*].total`, N = 8	1 tok + 2 gates → 9 tok	5.39 (6.96)	2.27 (7.14)	-58%	-46%
`$.orders[*].total`, N = 64	1 tok + 2 gates → 65 tok	5.51 (7.08)	2.54 (3.06)	-54%	-59%
`$.orders[*].total`, N = 256	1 tok + 2 gates → 257 tok	5.47 (7.06)	3.62 (4.16)	-34%	-38%
`$.orders[].items[].price`, N = 8	1 tok + 4 gates → 73 tok	17.8 (25.8)	3.50 (8.52)	-80%	-78%
`$.orders[].items[].price`, N = 64	1 tok + 4 gates → 4161 tok	17.4 (25.2)	25.8 (27.0)	+48%	+52%

Enumeration and the member/element test

The first five rows are the wide corpus (1,000 rows, 300 members and 300 elements per document). The last two are nested $.orders, from the focused run.

read	sent (gates → derivable)	gates, median (p95) ms	derivable, median (p95) ms	per call	in-server per doc
`json_tree_children`, 300-element array	1 tok → 1 tok	25.1 (26.1)	12.8 (13.4)	-49%	-49%
`json_tree_children`, 300-member object	1 tok → 1 tok	25.1 (26.2)	12.9 (13.6)	-49%	-49%
`json_tree_child_count`, 300-element array	1 tok → 1 tok	24.8 (25.9)	15.1 (15.9)	-39%	-40%
full read, 300-member object	1 tok → 1 tok	31.1 (32.3)	42.0 (43.8)	+35%	+37%
full read, 300-element array	1 tok → 1 tok	31.1 (32.4)	55.4 (57.6)	+78%	+81%
`json_tree_children`, `$.orders` (5–10 elements)	1 tok → 1 tok	1.74 (2.16)	1.81 (2.30)	+4%	+12%
`json_tree_child_count`, `$.orders`	1 tok → 1 tok	1.75 (2.17)	2.79 (3.59)	+59%	+71%

Containment, `->` and ordering (10,000 rows, focused run: 10 passes, 100 reps)

read	sent (gates → derivable)	gates, median (p95) ms	derivable, median (p95) ms	per call	in-server per doc
`@>` one array element (nested, 1 row)	– → –	0.64 (0.79)	0.64 (0.75)	+1%	—
`@>` deep object (nested, LIMIT 10)	– → –	1.84 (2.03)	1.92 (2.13)	+4%	—
`@>` flat	– → –	0.44 (0.62)	0.43 (0.58)	-1%	—
`->` nested	– → –	0.46 (0.57)	0.49 (0.68)	+7%	+10%
`->` flat	– → –	0.18 (0.31)	0.18 (0.32)	+0%	+5%
`ORDER BY ord_term(… -> sel) LIMIT 10`, nested	– → –	2.17 (2.45)	2.33 (2.59)	+7%	—
the self-tag map alone (built by every derivable read)	–	n/a (no such map)	0.55 (0.72)	n/a	n/a
detoast and parse the document (floor)	– → –	0.14 (0.19)	0.15 (0.20)	+4%	+9%

100,000 rows, headline reads

read	sent (gates → derivable)	gates, median (p95) ms	derivable, median (p95) ms	per call	in-server per doc
`$.user.contact.emergency_contact.name`	1 tok + 3 gates → 1 tok	2.37 (2.65)	1.84 (2.32)	-22%	-27%
`$.company`	1 tok → 1 tok	1.90 (2.15)	2.68 (3.14)	+41%	+61%
`$.orders`, budget 0	1 tok → 1 tok	16.6 (24.8)	24.1 (36.1)	+45%	+47%
`$.orders`, default budget	1 tok → 1 tok	6.03 (6.35)	9.19 (9.75)	+53%	+63%
`$.orders[*].total`, N = 64	1 tok + 2 gates → 65 tok	4.95 (6.63)	2.47 (3.12)	-50%	-60%
`$.orders[].items[].price`, N = 64	1 tok + 4 gates → 4161 tok	16.4 (24.3)	24.8 (26.1)	+51%	+50%
`json_tree_child_count`, `$.orders`	1 tok → 1 tok	1.74 (2.13)	2.75 (3.58)	+58%	+69%
`@>` deep object, LIMIT 10	– → –	6.12 (6.29)	6.20 (6.25)	+1%	—
`->` nested	– → –	0.46 (0.54)	0.48 (0.54)	+4%	+9%

The noise floor is run 1 against run 2 of the whole 10k matrix in the same format. The median change was under 1%. Above 1 ms, every gates-to-derivable change reproduced within 3.5 points. Below 1 ms, round-trip noise reaches ±20% per call, so read the in-server column there.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

coderdan added 4 commits October 25, 2026 00:17

This was referenced Oct 24, 2026

json_tree: a stored float can come back one ulp off because numbers are re-parsed without serde_json float_roundtrip #1364

Open

fix(eql): return the node itself from every json_tree subtree read #1359

Closed

Navigation Menu

Repository navigation

feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards - #1363

feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards#1363
coderdan wants to merge 4 commits into
docs/json-tree-derivable-tokensfrom
feat/json-tree-derivable-tokens

coderdan commented Oct 24, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Oct 24, 2026

Uh oh!

coderabbitai Bot commented Oct 24, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 24, 2026

Uh oh!

coderdan commented Oct 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

coderdan commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

Related

Review notes

Uh oh!

changeset-bot Bot commented Oct 24, 2026

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Oct 24, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Oct 24, 2026

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

Uh oh!

coderdan commented Oct 24, 2026

Benchmarks: this PR (683802b) against its base (edabee6)

Summary

Stored size

Ingest (seconds; 10k rows is the median of 3 interleaved runs, 100k one run)

Reads with no wildcard (nested, 10,000 rows)

Wildcard reads (nested, 10,000 rows)

Enumeration and the member/element test

Containment, -> and ordering (10,000 rows, focused run: 10 passes, 100 reps)

100,000 rows, headline reads

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderdan commented Oct 24, 2026 •

edited

Loading

Mutation testing (cargo-mutants, `--in-diff`, stack-auth + stack-encrypt)

Benchmarks: this PR (`683802b`) against its base (`edabee6`)

Containment, `->` and ordering (10,000 rows, focused run: 10 passes, 100 reps)