Skip to content

Ground truth: explore a single-source DSL generating both Kotlin and Python fixtures #988

Open

Description

@michalharakal

Sub-issue of #984 — stretch/exploratory, not scoped work yet.

The idea

Define each ground-truth test case (op name, input shapes/dtypes, params) once in a
single declarative spec, then generate both the Python @Executable fixture (for
skainet-ground-truth) and the Kotlin op-dispatch entry (OperationExecutor's
executeOperation when branch) from it — instead of maintaining the mapping between
PyTorch's op API and SKaiNET's TensorOps API by hand in two places that can silently
drift (see #985: today only 4 of ~25 dispatched ops have any Python fixture at all,
and #986: the mapping isn't validated to actually agree on parameter names).

SKaiNET already has real precedent for registry-introspection-driven codegen elsewhere
in the repo (generateKernelMatrix, the ops-status-matrix pipeline) — this would be the
same idea applied across the Kotlin/Python boundary instead of within Kotlin.

What's actually hard here

Not the codegen mechanism (KSP, or a plain script reading a shared spec — either can
emit Python text files, KSP doesn't care what language it's writing). The hard part is
formalizing the semantic mapping between PyTorch's op signatures and SKaiNET's
TensorOps signatures (different parameter names/orderings/defaults per op) into a
schema — that mapping currently only exists as hand-written Kotlin in
OperationExecutor. The payoff of doing this is real (one definition, impossible for
the two sides to drift, closes #985's coverage gap structurally rather than
suite-by-suite) but it's a genuine design project, not a quick win.

Task

Scope a small proof-of-concept for 2-3 ops (e.g. add, relu, conv2d — one trivial,
one unary, one with real params) before committing to anything broader. If it doesn't
clearly pay for itself at that scale, close this as "not worth the complexity" rather
than half-build it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dsl/apiDSL and API designresearchResearch and evaluation tasks (DARC: R)sub-issueSub-issue of a tracking issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions