[SKEEP-006]: Input front ends and model identity #1331

Proposal document: docs/modules/skeep/pages/006-input-frontends-and-model-identity.adoc — prepared in a personal fork: https://github.com/michalharakal/SKaiNET-core-upstream/blob/feature/skeep-006-input-frontends/docs/modules/skeep/pages/006-input-frontends-and-model-identity.adoc
Status: Implemented
Branch: feature/skeep-006-input-frontends (fork; the upstream pull request follows once the #1321 good-first-issue lanes have landed, so the contributors working them are not disturbed)

Trigger

Public Kotlin API (new modules skainet-model-identity, skainet-frontend-core, skainet-frontend-text; the sk.ainet.io.tokenizer package becomes deprecated aliases) · compatibility/migration policy (three-release deprecation of the superseded identity switches and of the SKaiNET-transformers legacy tokenizer stack) · runtime integration (the resolver is what model loaders, Java callers and cartridge facades use).

Summary

A checkpoint is weights, an identity and the recipe that turns raw input into the tensors the weights expect. Today the recipe is re-derived downstream: SKaiNET-transformers carries a second tokenizer stack that disagrees with the engine's, model identity is split three ways in the engine and a fourth way downstream, and audio/image preprocessing has no home at all. The proposal introduces FrontEnd<I, O> with Tokenizer as its first implementor, a string-based ModelIdentity with an immutable ModelFamilyRegistry that weight mapping and front ends both key on, and FrontEndResolver as the single entry point. Algorithms live in the engine, golden-tested against the Hugging Face tokenizers library; families (downstream) contribute policy only. Audio and image features follow on the same contract in later lanes.

Related DARC features

• [Feature]: HF tokenizer.json parity for BPE encoders — array-form merges, normalizer, non-special added tokens, Metaspace BPE #1321 — tokenizer.json parity for BPE encoders (the lane fixes [Lane 2 · kotlin-core] tokenizer.json parity — accept the array form of model.merges #1323–[Lane 2 · kotlin-core] tokenizer.json parity — Metaspace BPE with byte fallback (encode side of #912) #1326 are folded into the moved implementations; the lanes stay open for their contributors and are closed by the upstream PR of this proposal or by their own PRs, whichever lands first)
• [Feature]: tokenizer.json parity gate — fix the silent-wrong-ids paths, fail fast on unsupported files, HF goldens for every shipped family SKaiNET-transformers#471 — legacy tokenizer defects and parity gate (its lanes become the migration gate)
• Two independent GGUF tokenizer implementations silently disagree (GGUFTokenizer vs TokenizerFactory) SKaiNET-transformers#330 — two GGUF tokenizer paths disagree (answered by ModelFamilyDescriptor + FrontEndResolver)

Sub-issues

Filed once the proposal is Accepted: engine F2 (contract, identity, resolver — implemented in the fork branch), F4 log-mel front end, F5 image front ends; SKaiNET-transformers F3 (consume engine tokenizers — implemented in the fork branch feature/skeep-006-frontend-consumer of https://github.com/michalharakal/SKaiNET-transformers), F6 audio families.

Status upkeep

• Proposal registered in docs/modules/skeep/nav.adoc and the "Current Proposals" table (fork branch)
• Maintainer moved status to Accepted (implemented directly; SKEEP-006: input front ends and model identity (text lane) #1332)
• Implementation PR(s) linked here and flipped the proposal's Status: to Implemented (SKEEP-006: input front ends and model identity (text lane) #1332, docs(skeep): SKEEP-006 status Implemented, link the input-frontends page from fromTokenizerJson #1351)