Skip to content

Match a request’s language tag against declared locales as a language range, not by exact string #794

Description

@ddeboer

Every place that lines a language tag up against a declared locale compares the
two strings with ===. A tag that names a region – nl-NL, en-GB, pt-BR
therefore matches no declared locale at all, even when the declaration names its
primary subtag. Browsers send exactly such tags, so this is the common case, not
the edge one.

Three instances, all the same root:

  • compileSort (packages/search-typesense/src/query-compiler.ts) resolves the
    request locale with field.locales.indexOf(locale) and, on -1, silently
    sorts on locales[0]. A request carrying Accept-Language: nl-NL against a
    field declaring ['en', 'nl'] sorts Dutch readers by the English title.
    The fallback is quiet, so the result looks ordered and is ordered by the wrong
    key.
  • defaultLanguageOrder (packages/search-api-graphql/src/language.ts) filters
    the accepted languages by available.includes(language). A nl-NL request
    finds no Dutch label available, so the Dutch value drops out of the requested
    block into “the rest”, and [0] – the value a surface displays, and the one
    the per-field Content-Language reports – can come back in another language
    the document happens to carry. Negotiator does emit the bare nl after
    nl-NL for a typical browser header, which softens this one to a wrong
    ranking rather than a wrong language, but the ordering it produces is still
    not the one the header asked for.
  • The projection (applyText in packages/search/src/project.ts) buckets a
    literal into a locale with value.lang === locale. A publisher tagging its
    titles nl-NL gets no search or sort key under a declared nl: the
    values are displayed and then never matched or sorted on. The existing
    normalisation only repairs the separator (pt_BRpt-BR); it does not
    bring a subtagged literal to its declared locale.

Language tags should be matched as BCP-47 language ranges – a declared nl
matches nl, nl-NL and nl-BE, most-specific first – in all three places,
with the exact match still winning where both are present (en and en-GB
declared side by side). und stays the untagged bucket and matches nothing but
untagged. One shared helper is the natural home, since the three today disagree
only by accident.

Found while fixing #792, which addresses a different sort-key defect on the same
path (a document titled in one language had no sort key for the others). That
fix leaves this one intact: it makes every declared locale's key present, but
the request locale is still resolved by exact string.

Reported downstream at https://codeberg.org/limburg/lol/issues/158 – LOL is not
bitten by the first instance only because its locales happen to start with
nl.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions