Skip to content

feat(model): add candidate-to-stable model promotion workflow #8

Description

@Teakowa

Background

OCRKit already has immutable model artifacts, manifests, verification, a stable.json release channel, evaluation gates, and Model Studio support for reviewed platform snapshots. The current release script, however, publishes a trained/evaluated model directly to the selected release channel, which makes model artifact publication and production selection too tightly coupled.

This issue should turn the existing model lifecycle into an explicit candidate -> compare -> promote -> stable workflow without moving training or release ownership to the platform.

Goal

Separate model artifact publication from production-stable selection so a newly trained model can be published and evaluated as a candidate without changing the production stable target, then promoted explicitly after comparison and release gates pass.

Target flow

reviewed platform dataset snapshot
→ train / continue fine-tune
→ evaluate candidate checkpoint
→ publish immutable candidate artifact + manifest
→ compare candidate against current stable
→ explicit maintainer promotion
→ stable channel points to candidate manifest

Publishing a candidate must not by itself alter models/pp-ocrv6-small/channels/stable.json.

Candidate and stable semantics

Preserve immutable model artifacts and manifest hashes. The implementation may use a candidate channel or another repository-consistent representation, but there must be a clear distinction between:

  • a verified model artifact that is available for comparison/testing;
  • the model currently selected by the stable production channel.

Do not introduce mutable model prefixes or overwrite released artifacts.

Comparison requirements

Before promotion, Model Studio / release tooling should make the decision operationally clear using existing evaluation data where possible. At minimum expose or record:

  • candidate model version/manifest;
  • current stable model version/manifest;
  • reviewed dataset snapshot provenance when applicable;
  • training base model/checkpoint provenance;
  • holdout evaluation result;
  • supported screenshot/layout fixture result;
  • per-critical-field deltas where evaluation data exists;
  • false-confident/error regressions relevant to the existing release gates.

Do not build a dashboard for its own sake. Prefer a compact release-decision view/report that reuses the existing Model Studio and evaluation scripts.

Promotion gate

Candidate -> stable must be an explicit maintainer action and must fail closed when required release evidence is missing or failing.

At minimum preserve the existing release requirements around:

  • evaluation;
  • full test suite;
  • manifest generation and checksum verification;
  • immutable artifact upload;
  • model load/verification checks.

Once OWBastion/Bastion#161 and the corresponding OCRKit compatibility gate land, promotion must also require the supported released screenshot-contract compatibility gate.

Do not force a newly trained model to be promoted simply because training completed. A no-promotion outcome is valid when the candidate does not improve or safely match the stable baseline.

Rollback

Rollback must remain selection of a previously verified immutable manifest. It must not require retraining or rebuilding an old model artifact.

Record enough stable-channel history/provenance through the existing release mechanism or a minimal extension so the operator can identify the previous target safely.

Model Studio

Update the existing local Model Studio rather than creating a new service. It should support the operational sequence:

import snapshot
→ train/evaluate
→ publish candidate
→ inspect candidate vs stable
→ promote explicitly

The Studio remains local/trusted tooling and does not become a platform business-control surface.

Acceptance criteria

  • A trained/evaluated model can be published as an immutable candidate without changing the stable production channel.
  • Candidate and current stable versions/manifests are clearly distinguishable in tooling and provenance.
  • Candidate-vs-stable comparison reuses existing evaluation outputs and surfaces critical-field/fixture regressions needed for a release decision.
  • Promotion to stable is an explicit maintainer action and fails closed if required evaluation/verification gates are missing or failing.
  • Stable promotion updates only the release-channel selection; released model artifacts remain immutable.
  • A candidate may be rejected/left unpromoted without corrupting or replacing the current stable model.
  • Rollback can repoint stable to a previously verified immutable manifest without retraining.
  • Existing reviewed-snapshot provenance, source-level train/holdout separation, artifact verification, and production/runtime boundaries remain intact.
  • Model Studio and CLI/release tests cover candidate publication, promotion refusal, successful promotion, and rollback selection.

Non-goals

  • Automatically training whenever platform annotations change.
  • Automatically promoting every successful training run.
  • Automatically deploying the production OCRKit container; tracked separately.
  • Moving model artifacts or training execution into owbastion.com.
  • Replacing immutable manifests with mutable model directories.
  • Adding a hosted/public Model Studio.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions