fix: support Hunspell flag aliases and lazy affix expansion

[fmagnucz]

Problem

Vale previously expanded every Hunspell dictionary entry eagerly while loading
the dictionary. It generated all reachable prefixed, suffixed, cross-product,
and continuation forms, then retained every generated surface form in memory.

This approach becomes impractical for highly inflected languages. With the
Hungarian hu_HU dictionary:

• a controlled sample of 1,000 entries generated 3,832,856 forms;
• the estimated full expansion was approximately 370 million generated forms;
• an earlier full load did not finish within 45 seconds and reached about
  1.7 GiB of memory.

Correctly parsing Hunspell flag aliases made this problem more visible because
the aliases enabled many affix rules that Vale had previously ignored.

The affected dictionaries also exposed several compatibility issues:

• AF alias indices from .dic entries were interpreted as literal flags;
• AF alias indices in affix-rule continuation fields were also interpreted as
  literal flags, so later suffixes in a chain were never reached;
• the default 8-bit flag format was handled inconsistently between .aff and
  .dic files;
• UTF-8, long, and numeric flag encodings needed explicit handling;
• a hyphen inside a Hunspell character class is literal, but Go regular
  expressions interpreted it as a range;
• single-byte special flags could be represented differently from dictionary
  entry flags, causing valid Italian words such as di to be rejected;
• the hand-written default spelling filters treated UTF-8 bytes as non-letters,
  so Unicode words could be skipped before dictionary lookup;
• a zero affix carrying continuation flags, such as 0/L, retained a literal
  0 because zero normalization happened before the continuation was split
  from the affix text;
• prefix rules ignored their Strip field in the forward path, while the lazy
  reverse path removed the added prefix without restoring the stripped text.

The final two issues predated this change and were also reproducible with
v3.19.0, but the new lazy implementation needed to preserve the corrected
semantics in both its forward and reverse paths.

Solution

• parse Hunspell AF flag alias tables and resolve aliases in both .dic
  entries and affix continuation fields;
• support default 8-bit, UTF-8, long, and numeric flag encodings;
• normalize single-flag directives consistently between .aff and .dic
  files;
• split continuation flags before normalizing Hunspell's 0 representation of
  an empty affix;
• apply Strip when generating prefixed forms and restore it when recovering
  candidate roots in the lazy reverse path;
• translate literal hyphens in Hunspell affix conditions correctly;
• retain dictionary roots and their flag vectors instead of materializing every
  derived surface form;
• resolve affixed words lazily using indexed reverse candidates followed by
  exact forward validation;
• restrict recursive continuation and cross-product traversal to
  reverse-reachable candidates, avoiding the combinatorial expansion exposed by
  real Hungarian continuation aliases;
• preserve continuation flags, prefix-suffix cross-products, compounds,
  homographs, and suggestions;
• use a bounded, thread-safe cache for repeated spell checks;
• implement the built-in spelling-filter semantics with Unicode-aware letter
  and uppercase checks while retaining the ASCII fast path;
• add regression coverage for aliases, encodings, recursive affixes,
  cross-products, compounds, suggestions, the Italian di case, zero affixes
  with continuation flags, and prefix stripping.

Follow-up regressions

A Hungarian AsciiDoc reproducer exposed that affix continuation values also use
the AF alias table. Native Hunspell accepts the inflected forms
szoftvertervezőt, képeslapot, lekérdezéseket, kigondolása, készítése,
and dokumentálása. Vale now accepts the same forms, while the intentional typo
szoftvvvvertervezőt is still reported.

A minimal AF 2 fixture verifies that a rule such as SFX A 0 ed/2 . resolves
continuation alias 2 instead of treating it as a literal flag.

The default spelling filters now pass Unicode words to the dictionary checker
and still skip words ending in Unicode uppercase letters. The spelling
end-to-end expectation was updated to confirm that the Japanese token
オプション is no longer silently skipped.

A follow-up review against dictionary-fr@3.0.0 exposed two additional,
pre-existing Hunspell compatibility gaps.

First, a rule such as:

SFX F 0 0/L .

must produce an unchanged form carrying continuation flag L. The parser
previously checked for an exact 0 before splitting the continuation, so 0/L
became affix text "0" with continuation "L". As a result, Vale rejected
l'ordinateur and accepted the invalid forms ordinateur0 and
l'ordinateur0. Continuation flags are now split first, after which the
remaining 0 is normalized to an empty affix.

Second, a prefix rule such as:

PFX A a l'A a

must strip the initial a before adding l'A, producing l'Ami from ami.
The forward path previously produced l'Aami, and the lazy reverse path did
not restore the stripped a when recovering the dictionary root. Prefix
stripping is now applied in the forward path and reversed when constructing
lazy lookup candidates.

The reviewer's minimal fixtures are covered by
TestZeroAffixWithContinuationFlags and TestPrefixStrip. They verify that
Vale accepts l'ordinateur and l'Ami while rejecting ordinateur0,
l'ordinateur0, and l'Aami.

Performance

Measurements used the system Hungarian hu_HU dictionary.

Scenario	Result
Eager expansion of a representative 100-entry subset	766,715 unique surface forms
Time for the eager 100-entry subset	4.04 s
Maximum RSS for the eager 100-entry subset	492,640 KiB
Lazy loading of the complete dictionary	approximately 0.19 s
Maximum RSS for the complete lazy dictionary	78,720 KiB
Roots loaded from the complete dictionary	91,242
Warm spell lookup	1.21–1.36 µs/op
Warm spell lookup allocations	37 B/op, 4 allocs/op

The eager full-dictionary figure is not presented as a completed benchmark:
the approximately 370-million-form value is an extrapolation from the
controlled 1,000-entry sample.

On the supplied Hungarian AsciiDoc reproducer, the pruned continuation lookup
completed in approximately 0.46–0.60 s. The earlier lookup took about 7.17 s,
while resolving continuation aliases without reverse-reachable pruning did not
complete within 90 s.

Automated testing

The following checks passed on the final implementation:

• go test ./internal/spell
• go test -race ./internal/spell
• go vet ./internal/spell
• go test ./internal/check
• go test ./internal/e2e -run 'TestScenarios/checks/spelling'
• go test ./internal/spell -run 'TestZeroAffixWithContinuationFlags|TestPrefixStrip' -count=1
• gofmt on all changed Go files
• git diff --check

A full go test ./... run was attempted after the French compatibility
follow-up. The affected packages passed, but the complete run could not finish
successfully in the restricted environment because package synchronization
required unavailable network access and some core tests attempted to write to
the read-only global Vale styles directory. The generated Tree-sitter Lua
NUL-character compiler warning was non-fatal. golangci-lint was not available
locally.

Cross-language smoke tests

Five languages with different flag formats and morphological characteristics
were tested:

• English: en_US
• German: de_DE from hunspell-de-de 20161207-12
• Italian: it_IT from hunspell-it 1:24.2.1-1
• French: fr from hunspell-fr-classical 1:7.0-1
• French follow-up: dictionary-fr@3.0.0
• Russian: ru_RU from hunspell-ru 1:24.2.1-1

Each sentence was checked first with native Hunspell and then with the final
Vale CLI. In every language, the correct sentence produced no alerts and the
misspelled sentence produced exactly the same three alerts in native Hunspell
and Vale.

English — en_US

Correct:

The careful engineer reviews the updated documentation before deploying the
reliable service to production.

Misspelled:

The carefull engineer revievs the updated documentaton before deploying the
reliable service to production.

Detected: carefull, revievs, documentaton.

German — de_DE

Correct:

Der freundliche Entwickler prüft heute die neue Dokumentation und installiert
anschließend das zuverlässige Programm auf dem Rechner.

Misspelled:

Der freundliche Entwikler prüft heute die neue Dokumentazion und installiert
anschließend das zuverlassige Programm auf dem Rechner.

Detected: Entwikler, Dokumentazion, zuverlassige.

Italian — it_IT

Correct:

La giovane sviluppatrice controlla attentamente la nuova documentazione prima
di installare il programma aggiornato sul computer moderno.

Misspelled:

La giovane svilupatrice controlla attentamente la nuova documentazzione prima
di installare il programma aggiornatto sul computer moderno.

Detected: svilupatrice, documentazzione, aggiornatto.

The correct sentence deliberately includes di, covering the Italian
single-byte flag regression.

French — dictionary-fr@3.0.0 follow-up

Correct:

L'Ami de cette jeune développeuse relit attentivement la nouvelle
documentation avant de vérifier le programme installé sur l'ordinateur du
bureau.

Misspelled:

L'Ami de cette jeune dévelopeuse relit attentivement la nouvelle
documentattion avant de vérifier le programne installé sur l'ordinateur du
bureau.

Detected: dévelopeuse, documentattion, programne.

The correct sentence deliberately includes both L'Ami and l'ordinateur.
The former covers prefix stripping, while the latter covers a zero affix
carrying continuation flags. With v3.19.0, Vale reported both expressions as
misspellings. The final implementation produces no alerts, matching native
Hunspell.

Russian — ru_RU

Correct:

разработчик внимательно проверяет новую документацию перед установкой
обновлённой программы на рабочий компьютер в офисе.

Misspelled:

разработтчик внимателно проверяет новую докуминтацию перед установкой
обновлённой программы на рабочий компьютер в офисе.

Detected: разработтчик, внимателно, докуминтацию.

[clemlesne]

Thanks for the work here. Two Hunspell compatibility gaps remain at 4c0077c9a65e8ac307ed2ad4be5d6795048e3428. I reproduced both against native Hunspell 1.7.3, and both affect dictionary-fr@3.0.0.

1. A zero affix with continuation flags keeps a literal 0

Minimal test.aff:

SET UTF-8
PFX L Y 1
PFX L 0 l' .
SFX F Y 1
SFX F 0 0/L .

test.dic:

1
ordinateur/F

• Hunspell accepts l'ordinateur and rejects ordinateur0 and l'ordinateur0.
• Vale rejects l'ordinateur and accepts both invalid forms.

The parser checks for an exact 0 before strings.Cut. It therefore parses 0/L as AffixText: "0" and Cont: "L" instead of an empty affix with continuation L.

The real French case is SFX Fc 0 0/L'D'Q' [td]eur with ordinateur/Fc.

2. Prefix rules ignore Strip

Minimal test.aff:

SET UTF-8
PFX A N 1
PFX A a l'A a

test.dic:

1
ami/A

• Hunspell accepts l'Ami and rejects l'Aami.
• Vale rejects l'Ami and accepts l'Aami.

The forward path prepends AffixText without removing Strip. The reverse path removes AffixText without restoring Strip before it checks the condition.

The real French rule is PFX L' a l'A a.

These expectations follow Hunspell's affix syntax and its prefix continuation example. With the complete French package, Hunspell accepts and Vale rejects l'ordinateur, L'Ordinateur, l'ami, and L'Ami.

In a temporary copy, splitting continuation flags before zero normalization and applying Strip in both prefix paths fixed both cases. The invalid forms were then rejected, and go test ./internal/spell -count=1 passed. The native CLI fixtures used WORDCHARS only to preserve apostrophes and trailing digits as single input tokens.

[fmagnucz]

Hi @clemlesne,

First of all, thank you for your comment.

I have fixed your issue, and I have updated the PR description as well. But these problems already exists in the v3.19.0. I hope I didn't mess anything else up with this. :-)
Please, test it again.

[fmagnucz]

Hi @jdkato,

What are your plans for this PR? Are you going to reject it, or merge it into the active branch? If you're going to merge it, please send me a message in case I need to resolve the merge conflicts.

Thank you for your answer.

Best regards,
Feri