[ML] Fail gracefully when restoring a categorizer with an out-of-range token ID

[edsavage]

Summary

Fixes a residual native crash (SIGSEGV) when restoring a categorizer from an inconsistent or truncated state document, seen on a 9.6.0-SNAPSHOT build (see #2875).

CTokenListDataCategorizerBase::acceptRestoreTraverser restores a list of tokens (m_TokenIdLookup) and a list of categories. Each restored category references token IDs (its base and common-unique token IDs). Those IDs are later used to index m_TokenIdLookup without a bounds check — e.g. in cacheReverseSearch:

const CTokenInfoItem& info{m_TokenIdLookup[tokenId]};      // out of range -> garbage
... m_ReverseSearchCreator->addInOrderCommonToken(m_TokenIdLookup[tokenId].str(), ...);

m_TokenIdLookup is a Boost multi-index container whose random-access operator[] performs no bounds checking. If a corrupt/truncated state leaves a category referencing a token ID at or beyond the end of the restored lookup, this is an out-of-bounds access that returns a garbage CTokenInfoItem; calling .str() on it then dereferences a bogus pointer, crashing the autodetect process with a SIGSEGV inside libc (strlen/memcpy) rather than failing the restore.

This matches the crash signature reported in #2875 on 9.6: si_signo 11, si_code 1 (SEGV_MAPERR), crash PC in libc.so.6. The graceful invalid-state handling added in #2895 / #2898 does not cover this particular path.

Fix

Validate, at the end of acceptRestoreTraverser, that every token ID referenced by a restored category exists in the restored token ID lookup. If any is out of range, log an error and fail the restore gracefully (returning false), consistent with the intent of #2895 / #2898, instead of proceeding with an inconsistent state that crashes later.

Testing

Added two unit tests in CTokenListDataCategorizerTest:

• testRestoreWithInconsistentTokenIdFailsGracefully — a state with a single token but a category referencing token ID 5; restore now returns false (previously undefined behaviour / crash).
• testRestoreWithConsistentTokenIdSucceeds — a valid state referencing the only valid token ID (0) still restores successfully, confirming the new check does not reject good state.

Both pass, along with the existing testPersist round-trip test (so valid states are unaffected).

Relates to #2875

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @edsavage, I've created a changelog YAML for you.

[Copilot]

Pull request overview

This PR hardens categorizer state restoration in ml-cpp to avoid native crashes when encountering inconsistent/truncated persisted state. It adds post-restore validation to detect out-of-range token references (and related invalid bounds) early, returning false so the caller can fail the restore gracefully instead of later triggering out-of-bounds container access.

Changes:

• Add restore-time validation that every restored category’s referenced token IDs are within the restored token lookup bounds, failing restore if not.
• Add restore-time validation that each category’s ordered common token bounds form a valid subrange of its base token list.
• Add unit tests covering inconsistent vs consistent token ID restoration behavior (plus invalid ordered-bounds restoration).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
lib/model/unittest/CTokenListDataCategorizerTest.cc	Adds unit tests ensuring invalid restored state fails gracefully and valid state still restores.
lib/model/CTokenListDataCategorizerBase.cc	Adds post-restore validation to prevent later unchecked indexing from crashing the process.
docs/changelog/3143.yaml	Adds a changelog entry documenting the bug fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[elastic-vault-github-plugin-prod]

💚 All backports created successfully

Status	Branch	Result
✅	8.19
✅	9.4
✅	9.5

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details