Skip to content

fix(orders): keep FTS5 trigram search candidates a superset of Unicode case-folded matches #1256

Closed

Description

@khaira777

Summary

GET /api/orders?search=... combines an FTS5 trigram candidate query (orders_search_fts and customers_search_fts) with the authoritative JavaScript substring matcher (orders_search_contains) using AND. Because SQLite's built-in trigram tokenizer uses a legacy Unicode 6.0 1-to-1 case-folding table while orders_search_contains uses V8 String.prototype.toLowerCase(), non-ASCII Unicode queries (such as Georgian აბგ for stored ᲐᲑᲒ or Turkish decomposed i\u0307st for stored İstanbul) are excluded by candidateClause before udfClause can evaluate them.

Verified Reproduction

On 08187174bfa47f08154e2b9b64cdb17b0ef2c459:

  1. Seed a customer with Georgian Mtavruli name ᲐᲑᲒ (ORD-GEORGIAN) and a customer with Turkish name İstanbul (ORD-TURKISH).
  2. Query GET /api/orders?search=%E1%83%90%E1%83%91%E1%83%92 (აბგ) and GET /api/orders?search=i%CC%87st (i\u0307st).
  3. With orders_search_fts and customers_search_fts active, both queries return 0 orders (200 OK), whereas bypassing FTS returns the matching orders via orders_search_contains.

Root Cause

In main/routes/orders.ts:662-664:

const hasControlChars = /[\x00-\x1F\x7F]/.test(search);
const isEligible = !hasControlChars && [...search].length >= 3;
const hasFts = isEligible && isSearchIndexAvailable(db);

isEligible admits non-ASCII search queries into the FTS5 trigram candidateClause even though SQLite FTS5 trigram does not fold post-Unicode 6.0 scripts (e.g. Georgian Mtavruli U+1C90..U+1CBA) or multi-codepoint case mappings (e.g. İ U+0130 -> i\u0307).

Expected Behavior & Chosen Design

Restrict FTS5 trigram candidate filtering in main/routes/orders.ts to printable ASCII queries (/^[\x20-\x7E]+$/.test(search) && search.length >= 3), where SQLite FTS5 trigram and JavaScript .toLowerCase() case folding are equivalent, and route non-ASCII queries through the existing udfClause fallback.

Acceptance Criteria

  • GET /api/orders?search=... returns matching orders for Georgian (ᲐᲑᲒ queried with აბგ), Turkish (İstanbul queried with i\u0307st), and non-ASCII order numbers/customer names with FTS tables active.
  • Printable ASCII queries of length >= 3 retain the FTS5 trigram candidate index fast path and pass existing 100k/300k performance and query-plan assertions.
  • Control-character, NUL-byte, short-query (< 3 chars), phone-number, and pagination behaviors remain intact.

Required Regression Tests

  • Georgian Mtavruli/Mkhedruli case-insensitive customer and order search parity with FTS active.
  • Turkish dotted capital İstanbul / i\u0307st search parity with FTS active.
  • Existing tests/orders-search-api.test.ts and tests/orders-search-performance.test.ts suites pass.

Related Historical Context & Considerations

Activity

  1. khaira777 commented on Oct 21, 2026

    @khaira777
    ContributorAuthor

    Post-Merge Verification & Closure (N4)

    • Resolved via PR: fix(orders): keep FTS search candidates a superset of Unicode queries #1262
    • Reviewed Branch Head SHA: 31f89d45aa73292028c7b92022c5d814393d35c0 (updated onto main as 2f35beb860fe8651789898699ba11cbe04962703)
    • Merged main Commit SHA: (see GitHub merge event above)
    • Root Cause: In GET /api/orders (main/routes/orders.ts), SQLite FTS5 trigram candidate pre-filtering folded only ASCII A-Z, whereas the authoritative JavaScript orders_search_contains UDF applied Unicode NFKC + toLowerCase(). Non-ASCII queries (e.g., MÜNCHEN, İSTANBUL, ᲗᲑᲘᲚᲘᲡᲘ) or ASCII queries ending in i/I matching Turkish İ were dropped by the FTS candidate query before reaching orders_search_contains.
    • Fix Summary: Restricted hasFts candidate pre-filtering in main/routes/orders.ts to printable ASCII queries of length >= 3 that do not end in i or I (/^[\x20-\x7E]+$/.test(search) && !/[iI]$/.test(search) && search.length >= 3), routing non-ASCII and trailing-i/I queries through the authoritative udfClause so FTS candidates remain a strict superset of orders_search_contains matches.
    • Regression Tests Verified: tests/orders-search-api.test.ts (87/87 assertions passed, covering German umlauts, Turkish dotted/dotless I, Georgian Mtavruli/Mkhedruli case folding, and punctuated ASCII FTS parity).
  2. added a commit that references this issue on Oct 21, 2026
    a24500d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions