Skip to content

feat(eql)!: EQL v4 text search queries carry only the term their operator reads; produce TextSearch - #1274

Merged
coderdan merged 4 commits into
nextfrom
feat/eql-v4-text-search-optional-terms
Oct 19, 2026
Merged

coderdan merged 4 commits into
nextfrom
feat/eql-v4-text-search-optional-terms

Conversation

@coderdan

@coderdan coderdan commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

EQL is the SQL that lets PostgreSQL search encrypted columns. A column stores an encrypted value plus "index terms": an equality term (hm) for =, an order term (op) for < and ORDER BY, and a bloom filter (bf) for fuzzy text matching (@@). A query sends a matching "query value" carrying terms but no ciphertext.

TextSearch is the EQL type that does all three in one column. Until now EQL v4 (the version Stack Encrypt and the Go SDK write) required every query value to carry all three terms. The bloom filter is built from 3-letter pieces of the text, so a name like Al has none, and WHERE name = 'Al' could not be written. That is why the Go generator (stashgen) refused TextSearch fields.

This PR lets an EQL v4 text search query carry only the term its operator needs, and makes TextSearch producible from Rust and Go. EQL v3 (cipherstash-client's format) is untouched: its SQL bundle is byte-for-byte the same.

Changes

EQL SQL (v4 only)

  • eql_v4.query_text_search and eql_v4.query_text_search_ore now require at least one of their terms instead of all of them (plus the existing v, i, no c, version checks; a present ob is still a non-empty array).
  • Every extractor over those query values (eq_term, ord_term, ord_term_ore, match_term, so every operator and function form) reads its term through a new eql_v4_internal.require_query_term, which raises invalid_parameter_value naming the domain, the missing term and the operators that compare it. It never echoes the query value. A term that is JSON null or an empty array counts as missing. It raises wherever the extractor reads the operand (a literal folds at plan time; a generic plan whose predicate is never reached returns no row and no error), instead of reading as NULL (silently no rows) or as an empty filter. The extractors stay LANGUAGE sql, so indexes still engage (checked with EXPLAIN).
  • How v4 diverges from the shared source: the renderer that writes the one SQL source out as eql_v3 and eql_v4 gains version blocks, lines between -- {{#v3}}/-- {{/v3}} or -- {{#v4}}/-- {{/v4}}. A block is kept for its own version and dropped from the other, and the directive lines are dropped from both. There was no mechanism for this before; this is the smallest one that keeps a single source. Malformed blocks (nested, unmatched, unclosed) fail the build. The generator emits blocks for the two search query domains, and the helper sits in a v4 block of src/v3/scalars/functions.sql.
  • ADR-0002 gets a dated amendment ("Amended 2026-10-19"); packages/eql/AGENTS.md documents version blocks.

Stack Encrypt (stack-encrypt, breaking)

  • TargetResolver::query and dynamic::record::query now receive the index that answers the query's operator (&IndexSpec), which query_term already knew. A type whose query carries every term ignores it.

eql-bindings

  • v4 TextSearchQuery and TextSearchOreQuery hold each term as an Option, skipped in JSON when None.
  • Generated projections TextSearchEqualityQuery, TextSearchOrderQuery, TextSearchMatchQuery each derive only their one term and convert into TextSearchQuery. The by-name dispatch (encryption::targets::query, which now takes the index and refuses one the type lacks with TargetError::NoIndex) runs the projection for that index.
  • TextSearch is producible. A stored value carries all three terms. Decision: short text is stored. Text too short for a 3-gram ("", "Al") gets the empty filter "bf": []: = and < find it, no @@ query does. Its bf field is a new v4::terms::SearchBloomFilter (same JSON and JSON Schema as BloomFilter), which takes the empty filter; TextMatch and every query keep BloomFilter and keep refusing such text. The stored domain's CHECK accepts bf: [], and matches cannot find an empty stored filter because a query's filter is never empty (the projection refuses short text and the SQL refuses "bf": []).
  • TextSearchQuery/TextSearchOreQuery's Option terms are listed under Breaking in the crate CHANGELOG; no released eql-bindings has the v4 module (3.0.6 does not), and the changeset stays minor.

Go

  • Regenerated encrypt/eql table: TextSearch producible; the guest forwards the index. A TextSearch field gets Eq, Lt, Lte, Gt, Gte and Matches. An empty or two-letter value is stored (empty filter); Matches on it is ErrTerm.
  • READMEs and CHANGELOG updated.

Verification

Re-run after the review fixes, rebased on feat/eql-v4-ordered-ope-types at f3ddc0fe1:

  • v3 bundle and uninstaller cmp-identical to the base's. mise run test:encryption:postgres passes on a fresh v4 database and on one with v3 and v4 installed (PostgreSQL from mise): short text stored and found by =/<, never by @@; "hm": null and "bf": [] operands refused with 22023.
  • heap_hygiene now watches every producible text type through every index. It found a pre-existing unwiped free of the plaintext in stack-encrypt's OPE derivation (sem::OpeVisitor::visit_block) on every op term; excluded by backtrace like the aws-lc one, counted and printed, not fixed here.
  • cargo metadata --locked clean at every commit (root, packages/eql, both Go guests, stack-encrypt fuzz). eql-bindings/encryption/codegen tests, types:check, codegen:parity, fmt, clippy -D warnings (EQL workspace minus the credentialed sqlx crate), rustdoc -D warnings, mise run doc, guests, wasm:guest:test (68 + 71), go:test, go:lint, go generate clean. Not re-run: the SQLx suite (needs credentialed fixtures), root-workspace nextest and mutants (no root-workspace code changed by the fixes).

Original verification:

Run locally on macOS (PostgreSQL 17.5 from mise, no Docker):

  • release/cipherstash-encrypt.sql and its uninstaller are byte-identical (cmp) to the build before this PR. Only the v4 bundle changes.
  • New PostgreSQL tests in packages/eql/tests/encryption/tests/v4_text_search.rs (real Stack Encrypt values, fake KMS), run with mise run test:encryption:postgres against a fresh v4 database, and again with v3 and v4 both installed: queries with only hm, only op or only bf are accepted and one with none is refused (23514). =, <>, <, >=, @@ and eql_v4.lt(...) find the rows the plaintexts say, two-letter text included on =/</>. Every operator and extractor refuses an operand missing its term with 22023, naming the term and not quoting the operand. The ORE twin follows the same CHECK rule. All four suites of that task passed.
  • New SQLx test a_v4_search_query_needs_one_term_and_a_v3_one_needs_all (v3 still needs all three terms; v4 needs one; each operator refuses a missing term). Run locally only against hand-made fixture rows: the real fixtures need CipherStash credentials I don't have, so CI is its first run against real cipherstash-client terms.
  • eql_v4.lints() is clean on the new bundle.
  • Rust: cargo test --locked -p eql-bindings --all-features, eql-encryption-tests (incl. heap hygiene), eql-codegen; mise run types:check and mise run codegen:parity clean; clippy -D warnings and fmt on both workspaces; mise run doc and mise run test:doc clean.
  • mise x --env test -- cargo nextest run --workspace --all-features: 1220 passed, 1 failed: stack_encrypt::stored_rows::every_frozen_row_opens_and_seals_as_it_did. It fails the same way on this PR's base, feat/eql-v4-ordered-ope-types at 82ab168e6, without these commits: the base's order-term commits are older than the ones on its own base branch, where this test passes in CI. Not regenerated (it's a frozen fixture).
  • Mutants on changed stack-encrypt lines (cargo mutants --in-diff): 2 mutants, both unviable, none survived. Run with the frozen-rows test filtered out of the baseline because of the base failure above.
  • Guests rebuilt (wasm:guest:build, :eql, both :deterministic), mise run wasm:guest:test (68 + 71 pass), mise run go:test (all packages ok), mise run go:lint (0 issues), go generate ./... produces no drift. New Go test TestV4TextSearchStoresEveryTermAndQueriesCarryOne and a stashgen golden case directory.

Related

Refs #1131

Review notes

  • Start with packages/eql/crates/eql-codegen/src/render.rs (version blocks), then the template changes in templates/query_types.sql.j2 and templates/functions/extractor.sql.j2, then src/v3/scalars/functions.sql.
  • TextSearchOre stays unproducible for its own reason (block ORE). Only its SQL CHECK and struct shape change.
  • CI is likely to show the pre-existing frozen-rows failure above (Rust crates and mutants jobs) until the base PR is rebased onto its own base.

@changeset-bot

changeset-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 213d998

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@cipherstash/eql Minor
stash Patch
@cipherstash/stack-prisma Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/prisma-example Patch
@cipherstash/stack-drizzle Patch
@cipherstash/stack-supabase Patch
@cipherstash/stack Patch
@cipherstash/wizard Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 35d8b585-b8dd-49d1-a753-21c45f1bfc89

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
0 0 2 0

Every mutant in the changed lines was caught by a test.

@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 82ab168 to 31563a5 Compare October 19, 2026 00:44
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from a77b70b to ae2df35 Compare October 19, 2026 00:45

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of ae2df35 against feat/eql-v4-ordered-ope-types. No blocking defect found; one design question and a few small points, ranked below. Inline comments carry the detail.

Findings (most severe first)

  1. Medium, design. A TextSearch column can never hold text shorter than one 3-gram ("", "Al"), because the stored side reuses the match visitor's query-side refusal. So the headline case, WHERE name = 'Al', is now expressible but can never match a row, and a Go Encrypt batch fails on one empty name. The engine's own doctrine stores such text as the empty filter, and the SQL matches guard already keeps an empty filter from matching everything (verified below). Now that stored and query filters are built by different structs, the stored side could allow the empty filter. See the inline comment on v4_text_search.rs.
  2. Low. "A missing term never reads as NULL" has two gaps: the helper only tests key presence, so "hm": null passes the CHECK and compares as NULL (0 rows, no error); and the refusal is raised when the operand is evaluated, so under a generic plan a bound parameter whose qual is never reached (WHERE flag AND name = $1 with no flag row, or an empty table with no index) returns 0 rows with no error. Neither returns a wrong row; both are missed diagnostics. Inline on functions.sql and the ADR.
  3. Low. eql-bindings is 3.0.6 and TextSearchQuery / TextSearchOreQuery change public field types to Option, which is semver-breaking for a Rust consumer, while the CHANGELOG files it under Changed and the changeset is minor. Inline on the CHANGELOG.
  4. Nit. The PR body says stored_rows::every_frozen_row_opens_and_seals_as_it_did fails on this base; it passes here (682/682 under nextest) and in CI's crates job, so that paragraph is stale.
  5. Nit. Heap hygiene runs the TextSearch query through the first index only (eq), so the order and match projections are not watched. Inline on heap_hygiene.rs.

Verified sound

  • Built both bundles, from the base branch tip (origin/feat/eql-v4-ordered-ope-types) and from this head: release/cipherstash-encrypt.sql and its uninstaller are byte-identical; the v4 diff is exactly the helper, the two relaxed CHECKs and the six extractor bodies. The REQUIRE edge added to the two query function files did not reorder the installer.
  • Installed the head v4 bundle on a scratch PostgreSQL 15.12 and exercised it: with hash/btree/gin indexes on eq_term/ord_term/match_term, =, <, >= and @@ all take an Index Scan (or Bitmap Index Scan) with a one-term literal operand and with a $1 parameter under force_generic_plan; EXPLAIN VERBOSE shows the extractor inlined ((name)::jsonb ->> 'hm' = require_query_term(...) ->> 'hm'), so the plpgsql helper does not block inlining. =, <>, <, >=, @@, both commutator forms, eql_v4.lt/eq/gte/matches and the bare extractors all raise 22023 with the documented message when their term is absent, and the message never contains the operand's other term values. The stored domain still rejects a value missing any of hm/op/bf; the v4 query domain rejects no-term, c-present, and v=3; the ORE twin rejects an empty ob and accepts hm alone. An empty bf query matches 0 rows, not all. eql_v4.lints() is clean. Helper is IMMUTABLE STRICT PARALLEL SAFE; extractors stay LANGUAGE sql STRICT.
  • Renderer version blocks: directive parsing, nesting/unmatched/unclosed errors, literal-name refusal inside a skipped block, and the "other version unchanged" property are all tested; the template emits directives alone on a line so they are never read as placeholders.
  • Codegen divergence points (optional_term_keys, required_term, V3_SCALARS_BLOCKER require, projection rendering, dispatch arms, no_index fall-through) match the hand-written SQL and the generated Rust; cargo test -p eql-codegen, -p eql-bindings --all-features, -p eql-encryption-tests and mise run types:check pass locally.
  • Producer side: each projection derives exactly one term and converts into the twin; query refuses an index the type lacks before deriving; the =/< query on two-letter text carries hm/op only and @@ on it is refused. run_projected zeroizes the plaintext as run_target does.
  • stack-encrypt: mise x --env test -- cargo nextest run -p stack-encrypt --all-features 682 passed; clippy -D warnings clean. TargetResolver::query change is under Breaking in its CHANGELOG. Go: go test ./stashgen/... passes, the directory golden case is registered and matches the generator; go generate drift is checked in CI, which is fully green at this head.
  • ADR-0002 amendment matches the code. Skills do not mention the Go SDK or stashgen, and the v3 types.TextSearch guidance in stash-encryption is unaffected, so no skill change is owed. Only v3 JSON schemas and TS bindings are committed, so nothing is stale there.

Not run: the SQLx coexistence test (no Docker on this machine); CI's PG17 shards ran it and passed.

Comment thread packages/eql/tests/encryption/tests/v4_text_search.rs Outdated
Comment thread packages/eql/src/v3/scalars/functions.sql Outdated
Comment thread docs/adr/0002-one-value-model-three-encodings-and-eql-v4-by-producer.md Outdated
Comment thread packages/eql/crates/eql-bindings/CHANGELOG.md
Comment thread packages/eql/tests/encryption/tests/heap_hygiene.rs Outdated
@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 31563a5 to f3ddc0f Compare October 19, 2026 01:58
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from ae2df35 to e09295b Compare October 19, 2026 02:12
@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from f3ddc0f to 238aee3 Compare October 19, 2026 02:30
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from e09295b to 4e44e5b Compare October 19, 2026 02:31
@coderdan

Copy link
Copy Markdown
Contributor Author

Force-pushed: restacked on #1235's fix for an unwiped plaintext in the order visitors (see that PR). The OPE exclusion in heap_hygiene.rs is gone, folded into b431cfa. Every producible text type now runs through every index with no exclusion except vitaminc's aws-lc seal, and the test passes.

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of e09295b against feat/eql-v4-ordered-ope-types at f3ddc0f. Range-diff against ae2df35 (old base c1bdf66): the base rewrite dropped the two ordered-OPE commits that used to ride under this PR; every change left in the range-diff is the author's own fix-up, and all four commits still apply cleanly on the new base. No blocking defect. Two low findings and one nit below; the five earlier findings are accounted for first.

Prior findings (review 5566064727)

  1. Medium, short text could not be stored in TextSearch — fixed. The stored struct's bf is the new v4::terms::SearchBloomFilter, whose SearchMatchVisitor returns the empty filter where MatchVisitor refuses; TextMatch and every query keep BloomFilter. stored_term_newtype in the generator emits it only for a v4 search domain's stored struct, with a test pinning each struct's field type and that v3 never names it. Verified end to end: the encryption-tests suite, the Postgres-backed v4_text_search test against a scratch PG 15.12 running the head v4 bundle (rows "Al" and "" stored with "bf": [], found by =/</lt(), found by no @@ needle), and the Go test.
  2. Low, "hm": null read as NULL and the raise is at evaluation — fixed. require_query_term now treats JSON null and an empty array as missing; the helper doc, ADR amendment and changeset say "wherever the extractor reads the operand" and spell out the literal-vs-generic-plan difference. Verified on the head bundle: hm: null on =, op: null on <, bf: [] and bf: null on @@, on eql_v4.matches() and on the commutator form all raise 22023 naming the term; the ORE twin raises the same for hm: null and bf: [], and its CHECK still refuses ob: null and ob: [] (23514).
  3. Low, Option fields filed under Changed — fixed, and the decision to keep minor is sound. The entry is under a new ### Breaking heading stating the field changes, and the @cipherstash/eql@3.0.6 tag's crates/eql-bindings/src/ holds v3/ and from_v2/ only, so no published crate has the v4 module; the changeset says so.
  4. Nit, stale stored_rows claim in the PR body — partly fixed. The old paragraph is now labelled "Original verification", but "Review notes" still says CI is likely to show the frozen-rows failure in the crates and mutants jobs; both are green at this head.
  5. Nit, heap hygiene watched eq only — fixed. The test reads the producible text types from the target table and runs each query through every index, which is how it surfaced the pre-existing OPE leak (see finding 2 below).

New findings (most severe first)

  1. Low. "No @@ query finds a stored empty filter" holds for query operands, but not for the stored-vs-stored forms, which this PR does not touch: matches(a, b) over two eql_v4_text_search values keeps the shared guard a @> b AND (cardinality(b) > 0 OR cardinality(a) = 0), so two short-text rows match each other. Inline on v4_text_search.rs.
  2. Low. The new heap exclusion is a substring match on stack_encrypt::sem::OpeVisitor over the whole backtrace, so every unwiped free beneath visit_block is excluded, not only the dropped String clone, and the per-run count is printed but never asserted. Inline on heap_hygiene.rs. No issue is filed for the stack-encrypt leak itself yet; the exclusion comment would be the natural place to point at one.
  3. Nit. PR body "Review notes", as in prior finding 4.

Verified sound at this head

  • Built both bundles from the base tip and from this head with mise run build: release/cipherstash-encrypt.sql, its uninstaller and the v4 uninstaller are cmp-identical; the v4 installer's diff is exactly the helper, the two relaxed CHECKs and the six extractor bodies. The committed bundles and release manifests are unchanged against the base.
  • On the head v4 bundle (PG 15.12): hash/btree/gin indexes on eq_term/ord_term/match_term are used for =, < and @@ with a one-term literal and with $1 under force_generic_plan; EXPLAIN VERBOSE shows require_query_term inline inside the Index Cond, so the plpgsql helper still does not block inlining the LANGUAGE sql extractors. A stored "bf": [] is accepted by the domain CHECK; a non-empty @@ operand never returns such a row; eql_v4.lints() is clean.
  • SearchMatchVisitor falls back to MatchVisitor for non-empty positions, so the i16 bound check is kept; SearchBloomFilter delegates schema_name/schema_id/json_schema to BloomFilter, so the schema is one definition; no v4 TS export exists, and only v3 schemas are committed, so types:check passes with nothing stale.
  • cargo test -p eql-codegen, -p eql-bindings --all-features, -p eql-encryption-tests (including the Postgres-backed v4_text_search, v4_types and v4_order tests against the scratch database, and heap_hygiene with --nocapture: exactly one excluded free per OPE derivation, none elsewhere), mise run types:check. No stack-encrypt source changed since ae2df35, and CI's crates, mutants, Go and PG17 shard jobs are all green at e09295b.
  • The OpeVisitor leak claim checks out against the code: Orderable::encrypt_in takes &self, and OpeVisitor(T, OrderTarget) owns the plain String clone that Owned-mode zip hands it, so it drops unwiped at the end of visit_block.

Not run: the SQLx coexistence suite (credentialed fixtures); CI's PG17 shards ran it.

Comment thread packages/eql/tests/encryption/tests/v4_text_search.rs
Comment thread packages/eql/tests/encryption/tests/heap_hygiene.rs

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of the update to 4e44e5b.

Compared with e09295b, only commit 2 changed. It removes the stack_encrypt::sem::OpeVisitor exclusion from packages/eql/tests/encryption/tests/heap_hygiene.rs, leaving only vitaminc's aws-lc seal, and updates the module doc to match. The other three commits are identical apart from the rebase.

This resolves the second LOW in review 5566371952. The exclusion matched on a whole-backtrace substring and its count was never asserted, so it could have absorbed other frees under visit_block. The underlying leak is now fixed at its source in #1235 (ba29e03): order visitors carry the plaintext as Carried<T> and wipe it on drop. The widened heap-hygiene test now holds every order term of TextOrd, TextOrdOpe and TextSearch to the no-residue rule. It still needs #1235's fix in this branch's base to pass, and CI is green at this head.

Still open from 5566371952:

  • LOW: comparing two stored values with @@ / eql_v4.matches matches when both filters are empty.
  • Nit: the stale "Review notes" sentence in the PR body.

@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 238aee3 to 68cec8c Compare October 19, 2026 03:19
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 4e44e5b to 4430c0c Compare October 19, 2026 03:19
@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 68cec8c to 3e96092 Compare October 19, 2026 05:38
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 4430c0c to 9048302 Compare October 19, 2026 05:38
@coderdan

Copy link
Copy Markdown
Contributor Author

Restacked onto the new #1266 and force-pushed. The conflicts were the smallint additions; generated files were regenerated. The review fixes are doc-only (the @@ scope in the ADR amendment, README and CHANGELOG).

@coderdan coderdan left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental re-review of the restack to 30d3a35 (onto #1266's 548c546). Range-diff against 4430c0c: commits 2 and 4 are identical; commits 1 (the SQL) and 3 (the bindings) changed.

Substantive change, response to review 5566371952.

  • LOW, stored-vs-stored @@ matches two empty filters: addressed by documentation at this head, as agreed in the thread. The ADR-0002 amendment, the eql-bindings README and the CHANGELOG now scope "no @@ finds a stored empty filter" to query operands, and say a @@ between two stored values (a self-join, or matches(col, <stored value as jsonb>)) keeps the shared guard, as in v3, and that no producer builds that predicate. The three SQL functions are unchanged. Closed.
  • The OPE heap-exclusion removal was already in at 4430c0c; heap_hygiene.rs is unchanged here.

Rebase adaptations. The stashgen README table, the Go README, doc.go and eql_test.go lists carry the base's smallint entries beside TextSearch; the eql-bindings README sentence on Date/Timestamp/Numeric follows the base's wording; the bindings.rs and targets.rs doc comments take the base's "no Rust plaintext type" phrasing; tests/targets.rs drops the smallint reason match. None of the smallint types belongs in this PR's text-search surface and none is misplaced. No fixture changed, and the v3 bundle is not in the diff. mise run types:check at this head: no drift.

Still open: nit, the PR body's "Review notes" still says CI is likely to show the frozen-rows failure in the crates and mutants jobs until the base is rebased; both have been green for several heads, this one included.

Checks (no credentials): CI green at this head. At the top of the stack (#1280's 2189d5a, which contains these four commits): cargo test --locked -p eql-codegen -p eql-bindings --all-features and -p eql-encryption-tests pass (v4_text_search 5 passed, 1 ignored without a database); mise run types:check no drift; cargo nextest run -p stack-encrypt --all-features 695 passed; go test ./stashgen/... ./cmd/stashgen/... ok; go vet clean.

@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 548c546 to 17171df Compare October 19, 2026 06:32
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 30d3a35 to 61eb180 Compare October 19, 2026 06:32
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto the new #1266 (the chain is now on next after #1262, #1263 and #1264). One conflict, in a dynamic/record.rs test: this PR's query call with the answering index is kept, with its input written as Tree::UInt32(1), since next removed FfiValue.

@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 17171df to bea3dc2 Compare October 19, 2026 08:06
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 61eb180 to 422f4a1 Compare October 19, 2026 08:06
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto #1266's new head (61eb180 → 422f4a1), no conflicts. At the tip: nextest (1223), clippy, guest tests with and without eql, EQL crate tests and types:check, the guests and the Go suite, no go generate drift.

@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 422f4a1 to bd9091a Compare October 19, 2026 08:39
@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from bea3dc2 to 9ce4789 Compare October 19, 2026 08:39
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from bd9091a to 92b0e1c Compare October 19, 2026 09:27
@coderdan
coderdan force-pushed the feat/eql-v4-ordered-ope-types branch from 9ce4789 to 596d160 Compare October 19, 2026 09:27
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto next at 0800375 (after #1283 merged main back in). Only lockfiles changed, by exactly the crate moves that merge brought; the commits' content is otherwise identical to the previous head. Every Cargo.lock passes --locked at every commit, and the full Rust, guest, EQL and Go suites pass at the chain's tips against the merged lock.

Base automatically changed from feat/eql-v4-ordered-ope-types to next October 19, 2026 10:04
…perator reads

eql_v4.query_text_search required hm, op and bf on every query value. bf
is the bloom filter built from 3-grams, so text shorter than one 3-gram has
none, and `col = 'Al'` or `col < 'Al'` could not be written against an
eql_v4_text_search column. That is why Stack Encrypt could not produce the
type.

In v4 the search query domains (query_text_search and its _ore twin) now
require at least one of their terms. Each extractor over them reads its
term through eql_v4_internal.require_query_term, which raises
invalid_parameter_value naming the domain, the term and the operators that
compare it when the term is absent, so `=` on a value with no hm, `<` with
no op and `@@` with no bf are refused rather than reading NULL and
matching nothing. The extractors stay LANGUAGE sql, so the wrappers still
inline and the eq_term/ord_term/match_term indexes still engage.

v3 is cipherstash-client's format and does not change. The SQL source is
rendered once per version and had no way to say "only in v4", so the
renderer gains version blocks: lines between `-- {{#v3}}`/`-- {{/v3}}`
or `-- {{#v4}}`/`-- {{/v4}}` render for that version only, and the
directive lines for neither. The generator emits them for the two search
query domains, and the helper sits in a v4 block of
src/v3/scalars/functions.sql. release/cipherstash-encrypt.sql and its
uninstaller are byte-identical to the build before this change.

ADR-0002 gains a dated amendment recording the decision.
…s operator

TargetResolver::query took a type name, a label and a value, so a
resolver had to build the type's whole query twin, every term of it, for
every operator. EQL v4's TextSearch needs to emit only the term its
operator reads: a match term cannot be made from text shorter than one
3-gram, and `=` or `<` on such text must not be refused for want of one.

query_term already knows the answering index (OpSpec::answered_by over
the type's indexes). It now passes it through record::query to
TargetResolver::query as an &IndexSpec after the type's name. A resolver
whose twins carry every term ignores it.

eql-bindings' encryption::targets::query takes it the same way and
refuses an index the type does not carry (TargetError::NoIndex) before
deriving anything. The Go guest forwards it; an index the type lacks can
only come from an engine bug, so the guest reports it as internal.

BREAKING CHANGE: TargetResolver::query and dynamic::record::query take an
&IndexSpec; eql_bindings::encryption::targets::query takes one after the
name.
…perator's term

TextSearch was unproducible because its query twin carried hm, op and bf
on every query, and text shorter than one 3-gram has no bf: `=` or `<` on
a two-letter name would have been refused. EQL v4's search query domain
now takes any one term, so the twin can carry just the one its operator
reads.

In v4, TextSearchQuery (and the unproducible TextSearchOreQuery) hold
each term as an Option, skipped in the JSON when None. The generator
emits one projection per term, TextSearchEqualityQuery,
TextSearchOrderQuery and TextSearchMatchQuery, each deriving
EncryptFrom<String> over the envelope and its one term and converting
into TextSearchQuery. The by-name query dispatch, which now receives the
answering index, runs the projection for that index, so a match term is
derived only for `matches`. The twin itself carries no derive.

The stored TextSearch carries all three terms, the same bytes TextEq,
TextOrd and TextMatch store, so text too short for a 3-gram is refused on
write with EmptyTermText, exactly as TextMatch refuses it: a stored row
with an empty filter would be one no match query finds.

TextSearch joins ENCRYPTION_DOMAINS, the regenerated target table marks
it producible, and the Go encrypt/eql table follows. The engine answers
eq, lt, lte, gt, gte and matches on it.

Tests: each index's query carries its term alone and equal to the
stored one; "Al" is queried by eq and ope and refused for match; the
typed projections serialize to what the dispatch writes; and, against an
EQL v4 database, `=`, `<>`, `<`, `>=`, `@@` and the function forms find
the plaintexts' rows, a one-term operand is a query_text_search and a
termless one is not, and each operator and extractor refuses an operand
missing its term with invalid_parameter_value naming the term and never
the operand.
…ng one term

A directory Entry model holds one TextSearch field. Generated over the
deterministic EQL guest, its stored value carries bf, c, hm, i, op and v
and opens back; Eq, Lt and Gte return a query with only hm or op, the
stored term, for "Alice" and for the two-letter "Al"; Matches returns
one with only bf, which Verify checks; and Matches on "Al", like storing
"Qz" or "", is refused with ErrTerm without quoting the value.

stashgen gets a golden case for the same field over its static engine,
which now lists TextSearch with its three indexes, so the generated
methods (Eq, Lt, Lte, Gt, Gte, Matches) are pinned and compile against
the stub SDK.
@coderdan
coderdan force-pushed the feat/eql-v4-text-search-optional-terms branch from 92b0e1c to 213d998 Compare October 19, 2026 12:33
@coderdan
coderdan marked this pull request as ready for review October 19, 2026 12:33
@coderdan
coderdan requested a review from a team as a code owner October 19, 2026 12:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T12:41:26.226176Z 213d998 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-19T12:39:29.613283Z 213d998 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 213d998b3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +127 to +129
-- {{#v4}}
AND (VALUE ? 'hm' OR VALUE ? 'ob' OR VALUE ? 'bf')
-- {{/v4}}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add an upgrade note for the payload change

This changes which terms query_text_search_ore requires and introduces new runtime errors when an operator reads an omitted term, but the commit adds neither a numbered upgrade note under docs/upgrading/ nor a changeset cross-link. Callers that construct query payloads manually need the compatibility, verification, and rollback guidance before adopting this release.

AGENTS.md reference: packages/eql/AGENTS.md:L299-L306

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried in #1280, since that PR changes an eql_v4 payload too (the block ORE term layout) and the two notes belong in one file: packages/eql/docs/upgrading/v3.1.md at 45a79a0, U-001 for this change (which terms a text search query must carry, the invalid_parameter_value raise and where it fires, what a hand-built payload must do, verification, rollback) and U-002 for the block ORE scheme. Both changesets cross-link their note.

@coderdan
coderdan merged commit 358c0fd into next Oct 19, 2026
57 checks passed
@coderdan
coderdan deleted the feat/eql-v4-text-search-optional-terms branch October 19, 2026 12:55
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased onto next at d827132 (92b0e1c → 213d998), after #1272, #1243, #1265 and #1244 merged. Conflicts, each resolved in the commit where it arose:

Verified at the tip: every Cargo.lock passes --locked at every commit; fmt (root and EQL workspaces), nextest (1239 passed), clippy -D warnings, trybuild, guest tests with and without eql, the EQL crates, types:check and codegen:parity, all guests built, the Go suite, no go generate drift. Mutants over the packages/stack-encrypt diff: 2 mutants, both unviable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant