Repository navigation
fix(orders): keep FTS5 trigram search candidates a superset of Unicode case-folded matches #1256
Description
Summary
GET /api/orders?search=... combines an FTS5 trigram candidate query (orders_search_fts and customers_search_fts) with the authoritative JavaScript substring matcher (orders_search_contains) using AND. Because SQLite's built-in trigram tokenizer uses a legacy Unicode 6.0 1-to-1 case-folding table while orders_search_contains uses V8 String.prototype.toLowerCase(), non-ASCII Unicode queries (such as Georgian აბგ for stored ᲐᲑᲒ or Turkish decomposed i\u0307st for stored İstanbul) are excluded by candidateClause before udfClause can evaluate them.
Verified Reproduction
On 08187174bfa47f08154e2b9b64cdb17b0ef2c459:
- Seed a customer with Georgian Mtavruli name
ᲐᲑᲒ(ORD-GEORGIAN) and a customer with Turkish nameİstanbul(ORD-TURKISH). - Query
GET /api/orders?search=%E1%83%90%E1%83%91%E1%83%92(აბგ) andGET /api/orders?search=i%CC%87st(i\u0307st). - With
orders_search_ftsandcustomers_search_ftsactive, both queries return0orders (200 OK), whereas bypassing FTS returns the matching orders viaorders_search_contains.
Root Cause
In main/routes/orders.ts:662-664:
const hasControlChars = /[\x00-\x1F\x7F]/.test(search);
const isEligible = !hasControlChars && [...search].length >= 3;
const hasFts = isEligible && isSearchIndexAvailable(db);isEligible admits non-ASCII search queries into the FTS5 trigram candidateClause even though SQLite FTS5 trigram does not fold post-Unicode 6.0 scripts (e.g. Georgian Mtavruli U+1C90..U+1CBA) or multi-codepoint case mappings (e.g. İ U+0130 -> i\u0307).
Expected Behavior & Chosen Design
Restrict FTS5 trigram candidate filtering in main/routes/orders.ts to printable ASCII queries (/^[\x20-\x7E]+$/.test(search) && search.length >= 3), where SQLite FTS5 trigram and JavaScript .toLowerCase() case folding are equivalent, and route non-ASCII queries through the existing udfClause fallback.
Acceptance Criteria
-
GET /api/orders?search=...returns matching orders for Georgian (ᲐᲑᲒqueried withაბგ), Turkish (İstanbulqueried withi\u0307st), and non-ASCII order numbers/customer names with FTS tables active. - Printable ASCII queries of length >= 3 retain the FTS5 trigram candidate index fast path and pass existing 100k/300k performance and query-plan assertions.
- Control-character, NUL-byte, short-query (
< 3chars), phone-number, and pagination behaviors remain intact.
Required Regression Tests
- Georgian Mtavruli/Mkhedruli case-insensitive customer and order search parity with FTS active.
- Turkish dotted capital
İstanbul/i\u0307stsearch parity with FTS active. - Existing
tests/orders-search-api.test.tsandtests/orders-search-performance.test.tssuites pass.
Related Historical Context & Considerations
- Related PRs / Issue: fix(main): optimize order search with indexed candidate retrieval #1204, fix(orders): preserve order-number punctuation in FTS search and handle embedded NUL #1209, Order search scans large histories through JavaScript UDFs #1185
- Performance & Security Considerations: Preserves parameterized SQL bindings, requires no schema migration, and keeps the FTS5 trigram index active for printable ASCII searches.
Activity
khaira777 commentedon Oct 21, 2026
Post-Merge Verification & Closure (N4)
31f89d45aa73292028c7b92022c5d814393d35c0(updated ontomainas2f35beb860fe8651789898699ba11cbe04962703)mainCommit SHA: (see GitHub merge event above)GET /api/orders(main/routes/orders.ts), SQLite FTS5trigramcandidate pre-filtering folded only ASCIIA-Z, whereas the authoritative JavaScriptorders_search_containsUDF applied Unicode NFKC +toLowerCase(). Non-ASCII queries (e.g.,MÜNCHEN,İSTANBUL,ᲗᲑᲘᲚᲘᲡᲘ) or ASCII queries ending ini/Imatching Turkishİwere dropped by the FTS candidate query before reachingorders_search_contains.hasFtscandidate pre-filtering inmain/routes/orders.tsto printable ASCII queries of length >= 3 that do not end iniorI(/^[\x20-\x7E]+$/.test(search) && !/[iI]$/.test(search) && search.length >= 3), routing non-ASCII and trailing-i/Iqueries through the authoritativeudfClauseso FTS candidates remain a strict superset oforders_search_containsmatches.tests/orders-search-api.test.ts(87/87assertions passed, covering German umlauts, Turkish dotted/dotless I, Georgian Mtavruli/Mkhedruli case folding, and punctuated ASCII FTS parity).fix(orders): keep FTS search candidates a superset of Unicode queries (…