ops(model): automate promoted-model production rollout and verification

[Teakowa]

Depends on: #8
Related: #9, OWBastion/owbastion.com#80

Background

Production OCRKit already resolves its model through the stable release channel, but a newly promoted model is not loaded until the production container is recreated. The current operational path therefore has a manual last mile: publish/promote a verified model, access the host, recreate OCRKit, then manually confirm which model actually loaded.

This makes it easy for production to remain on an older model even when newer verified artifacts already exist.

Goal

Provide one controlled production rollout operation for an already-promoted stable model that recreates/restarts OCRKit as required, verifies that the intended immutable manifest/model is loaded, runs a representative recognition smoke check, and leaves a clear rollback path.

Do not implement in-process hot model swapping unless repository evidence shows it is necessary. A controlled container recreate/restart is acceptable and preferred when it preserves the current simple runtime model.

Target flow

candidate published
→ explicit promotion to stable (#8)
→ explicit production rollout
→ recreate/restart OCRKit
→ model manifest download + checksum/load verification
→ health verification
→ assert loaded model_version/manifest == promoted target
→ released-HUD recognition smoke
→ success

Candidate publication alone must never trigger production rollout.

Rollout boundary

The implementation should reuse the existing deployment topology and secrets rather than introducing a new deployment control plane. Determine the safest mechanism from the current repository/deployment reality (for example existing CI/CD, an approved remote deployment action, or host-side deployment tooling).

Requirements:

• no manual per-model edit of OCRKIT_MODEL_MANIFEST_KEY for normal releases;
• stable release-channel selection remains the production source;
• model artifacts remain immutable and verified;
• application image and model artifact lifecycles remain independent: a model-only rollout must not require rebuilding the OCRKit application image unless application code also changed;
• production secrets, R2 credentials, service tokens, screenshots, and private OCR payloads must not be exposed in Actions output or committed evidence.

Verification

A deployment/recreate or HTTP 200 is not sufficient evidence.

The rollout must verify at minimum:

• OCRKit process/service is healthy after restart;
• the loaded model_version matches the intended promoted target;
• where available, the loaded manifest identity/hash corresponds to the promoted immutable manifest;
• the configured OCR engine can initialize and serve recognition;
• one safe representative smoke case for the currently released/supported settlement HUD succeeds, including a critical strict-format field such as run code once test(recognition): gate releases against supported Bastion screenshot revisions #9 provides the reusable compatibility gate/fixture path.

Prefer reusing #9 compatibility/evaluation tooling rather than implementing a separate recognition test stack.

Failure behavior

If restart, model download, checksum verification, load verification, health verification, or recognition smoke fails:

• the rollout must report failure explicitly;
• do not mark the model as production-verified;
• preserve enough information to identify the attempted stable target;
• provide/test the supported rollback operation that repoints stable to a previously verified manifest and reruns the normal rollout path.

Do not silently fall back to an arbitrary model while reporting success.

Rollback

Rollback should be operationally equivalent to:

select previous verified immutable manifest as stable
→ run the same production rollout
→ verify loaded version + recognition smoke

It must not require retraining or republishing the old artifact.

Evidence and platform coordination

This issue owns OCRKit model rollout mechanics and OCR service verification only. Full submission/mastery/title production verification remains in OWBastion/owbastion.com#80 and must use sanitized traceable evidence after the relevant Bastion/OCR compatibility is released.

Do not treat this issue's successful model smoke test as proof that the whole platform business path works.

Acceptance criteria

• An explicitly promoted stable model can be rolled out to production through one documented/automated operation without manually editing the model manifest environment per release.
• Candidate publication without stable promotion cannot trigger production rollout.
• The rollout recreates/restarts the service as needed and verifies the exact intended model_version after startup.
• Manifest/artifact checksum and load verification remain fail-closed.
• The rollout runs a safe representative released-HUD recognition smoke, reusing test(recognition): gate releases against supported Bastion screenshot revisions #9 compatibility tooling where practical.
• A health check or deployment completion alone is not reported as production model verification.
• Failure at download/load/health/smoke stages produces an explicit failed rollout rather than silent success/fallback.
• Rollback to a previous verified immutable manifest is documented and tested through the same rollout path.
• Model-only rollout does not require rebuilding the OCRKit application image when application code is unchanged.
• Deployment logs/evidence do not expose credentials, private screenshots, signed URLs, raw production OCR payloads, or player identity.
• Documentation reflects the new standard model release -> promotion -> rollout -> verification sequence.

Non-goals

• Runtime hot-reload with zero restart.
• Automatically promoting candidates to stable.
• Automatically training from new platform annotations.
• Replacing the existing production deployment topology with a new orchestration platform.
• Declaring the full owbastion.com submission/mastery flow production-verified.

[Teakowa]

Implementation is in stacked Draft PR #14: #14

The controlled rollout reads the stable channel/manifest, recreates only the OCRKit service with --no-build --force-recreate, verifies exact manifest identity and loaded model_version, waits for health, and runs an authenticated local smoke request. Rollback reuses #8's previously verified stable manifest selector and reruns the same rollout. No production rollout was executed: the repository has no safe live deployment credentials/host authorization in scope, so production evidence remains an explicit follow-up rather than a claim.