kernel_package authoring guide (English)

English | 简体中文（从零编写）

This guide is for an agent that already has a GPU kernel implementation — with its own code and its own tests — and wants to rewrite it into a kernel_package that the kernel_zoo platform can benchmark. If instead you are creating a package from a blank slate, read the step-by-step authoring guide at kernel_package_guide.md (简体中文) first; this document assumes you have code to port and explains the same contracts through that lens.

By the end of this guide you will know:

the exact on-disk layout of a kernel_package and why it is shaped that way;
what each file is responsible for, why the design forces that shape, and how the server/runner consume each file;
a concrete before/after recipe for moving your implementation and your test suite into the kernel_zoo protocol.

Normative references (read these when you need the exact field lists):

Document	What it pins down
`kernel_package_format.md`	on-disk format + every schema table
`schemas/`	the JSON Schemas (source of truth)
`api_reference.md`	the HTTP API
`examples/`	a complete, validating example package

1. The system you are plugging into

kernel_zoo is a git-repository-as-database leaderboard. There is no external database: the kernel repo is the database, and every kernel_package directory in it is one operator implementation. Understanding the lifecycle is the fastest way to understand why every file contract exists.

┌──────────────┐  POST /api/submissions   ┌────────────────────────────────┐
│  You (agent) │ ───────────────────────▶ │              server             │
└──────────────┘   multipart tar.gz       │                                │
                                          │  validate → enqueue            │
                                          │  pending/<uuid>/               │
                                          │                                │
┌──────────────┐  GET /api/runner/claim   │                                │
│    runner    │ ◀─────────────────────── │                                │
│ (container)  │                          │                                │
│              │  GET /api/runner/package/│  downloads your tarball        │
│              │      <uuid> ────────────▶│  un-tars into a fresh workdir  │
│              │                          │                                │
│              │  python build_test_env.py│                                │
│              │      <workdir>           │                                │
│              │  python benchmark.py     │                                │
│              │      <workdir>           │                                │
│              │  ── stdout = JSON ──────▶│                                │
│              │  POST .../{uuid}/result  │  accept? → two-phase git commit│
└──────────────┘                          │  → index.json rebuild          │
                                          └────────────────────────────────┘

The runner drives your package through exactly two entry points, in order:

python build_test_env.py <workdir> — construct the experiment.
python benchmark.py <workdir> — run it, judge it, report a single JSON on stdout.

If either exits non-zero, or benchmark.py's JSON violates its schema, the result is recorded as rejected. There is no other way your code runs on the platform.

2. The target layout

A kernel_package is a directory inside the kernel repo. Its relative path is its kernel_id — that string appears in the URL, in desc.yaml's implicit location, in bench_result.yaml, and in the benchmark output, and every copy must match exactly.

<arch>/<op_class>/<sub_class>/<quant>/<your_name>/      ← this path = kernel_id
├── .kernel_package/
│   ├── desc.yaml             ← operator metadata (you write)
│   ├── bench_result.yaml     ← scoreboard (seed only; server owns it after first accept)
│   ├── build_test_env.py     ← generates inputs + expected outputs + build artifacts
│   └── benchmark.py          ← runs your kernel, checks correctness, measures, decides accept
└── src/                      ← your authored implementation; layout is flexible, content is not
    ├── my_kernel.py          ← (example) your ported kernel code
    └── reference.py          ← (example) a naive reference your tests compare against

Hard rules

Every path segment matches ^[A-Za-z0-9._-]+$ — no spaces, no slashes, no .., no Unicode.
.kernel_package/ contains all four files. A package missing any one is marked invalid (visible in listings but not claimable).
desc.yaml.reference_sources lists relative file paths that must actually exist inside the package — the server checks this before accepting any submission.
Only the server writes bench_result.yaml. The submission tarball must still include it — it is one of the four required .kernel_package/ files, and validation rejects a tarball missing it (MISSING_FILE). But the submitted copy is a seed used for validation only: on accept the server strips it from the tarball and preserves its own, so the scoreboard can never be overwritten by a submitter.

The split between src/ and .kernel_package/ is the core design decision: src/ is your code (freely shaped, fully overwritten on every accept), while .kernel_package/ is the contract area — metadata that belongs to the platform. Your implementation never touches the scoreboard, and the platform never merges your code; it replaces src/ wholesale.

Source-authored, generated-evidence-free package policy

A kernel repository and every submission tarball may contain authored source, metadata, and deterministic generation recipes only. They must not contain model weights, generated tensors, golden test data, compiled assets, runtime traces, profiler exports, hardware/software census or environment dumps, benchmark or compiler logs, captured stdout/stderr, generated reports, or summaries, tables, and statistics derived from those artifacts. This rule is independent of encoding: JSON, CSV, YAML, Markdown, and plain-text evidence are forbidden just like .npy, .so, and other binary outputs. Renaming, truncating, converting, or manually summarizing runtime evidence does not make it package content.

build_test_env.py is the required test-environment constructor: it deterministically generates temporary test inputs, synthetic model weights, expected outputs, optional build products, manifests, and diagnostics under the runner's temporary workdir. benchmark.py may create runtime logs and reports only there. The workdir is discarded after the run; never copy generated files back into the package, add them to Git, or put them in a submission tarball.

This policy applies to src/, sibling source directories, metadata fields, and all tarball members. “Flexible” and “free-form” describe directory organization or the schema shape of compact authored metadata; they do not permit arbitrary generated files, trace excerpts, evidence blobs, or derived summaries.

3. Inventory your existing project

Most GPU kernel projects contain the pieces below. Do this mapping first — it tells you exactly which files to create.

Your existing artifact	Maps to	How
the kernel source (`kernel.cu`, `kernel.py`, `Makefile`, …)	`src/…`	copy/adapt it; reference it from `desc.yaml`
a reference / naive implementation used for validation	`src/…`	keep it; `build_test_env.py` imports it to produce expected outputs
test data generation (random tensors, fixtures, golden files)	`build_test_env.py`	writes `workdir/inputs/` + `workdir/expected/`
correctness assertions (`assert_allclose`, `pytest.approx`, …)	`benchmark.py`	re-expressed as a correctness check against `workdir/expected/`
a latency / throughput micro-benchmark	`benchmark.py`	re-expressed as `metrics`
per-case test configurations (shapes, dtypes, batch sizes)	`KERNEL_ZOO_TEST_CASE_METAS` + `desc.yaml.io_signature`	the platform's notion of "test cases"
the build/install step	`build_test_env.py`	compile into `workdir/build/` so the benchmark can load it
the "does the suite pass?" summary	`overall.accept` + `accept_reason`	your pass/fail policy becomes the accept policy

The key mental shift: you are not shipping your test runner to kernel_zoo. You are translating its intent into two files that the platform's own runner executes. Your assertions do not run under pytest — they run inside benchmark.py as the correctness gate. Your timings do not print a report — they become JSON metrics that drive the leaderboard.

4. Step 1 — choose `kernel_id` and your test cases

The first four path segments follow the classification convention and should mirror desc.yaml.category:

<arch>/<op_class>/<sub_class>/<quant>/<your_name>/
gfx928/Attention/MHA/fp16/my_flash_attn/

Decide this up front because it is replicated in four places that must agree: the directory, desc.yaml.category, bench_result.yaml.kernel_id, and every benchmark output's kernel_id.

Then decide your test cases. A test case is one (parameterized) run of the operator — a particular shape, batch, dtype configuration. Pick a small set that captures the operator's behaviour and its performance profile (e.g. a small shape for correctness sanity and a large shape for throughput). These become KERNEL_ZOO_TEST_CASE_METAS and the params are free-form but should be self-describing.

5. Step 2 — place your implementation in `src/`

Copy source text for your kernel into src/ (or any sibling source directory you prefer — include/, Makefile, CMakeLists.txt, .cu, .hip, and .cl are all fine). The layout is flexible, but the binary-free package policy applies to every sibling directory: do not place weights, generated tensors, golden files, compiled objects, or other binary assets there. Every path listed in desc.yaml.reference_sources must exist.

Keep your reference/naive implementation in src/ too — build_test_env.py will import it to generate expected outputs. The example package does exactly this: build_test_env.py does from ref_attention import reference_attention.

To make your code importable from the package root, both build_test_env.py and benchmark.py prepend src/ to sys.path:

PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))

If your kernel needs a compiler step (e.g. hipcc/nvcc), do that in build_test_env.py and emit artifacts into workdir/build/.

6. Step 3 — write `desc.yaml`

desc.yaml is your operator's "datasheet". It is read by the WebUI, by index.json, and above all by future optimizing agents that have never seen your code — so the description field is the most load-bearing piece of the whole package.

schema_version: "1.0.0"
name: MHA FP16 Reference
category:                        # must mirror the first 4 path segments
  arch: gfx928
  op_class: Attention
  sub_class: MHA
  quant: fp16
summary: Pure-Python reference multi-head attention used to exercise the contract layer.
description: |
  ## 数学定义
  S[b,h,i,:] = (Q[b,h,i,:] @ K[b,h,j,:].T) / sqrt(d_k)
  P = softmax(S);  O = P @ V

  ## 输入布局
  Q, K, V: [batch, heads, seq_len, head_dim], fp16.

  ## 性能提示
  - Use MFMA instructions for the batched GEMMs.
  - Use online safe-softmax.

  ## 已知陷阱
  - Layout of K/V (BHD vs BSH) changes KV-cache performance.
io_signature:
  - name: Q
    kind: in                     # in | out | inout
    dtype: fp16                  # generic dtype name, not vendor-specific
    shape: [batch, heads, seq_len, head_dim]
    shape_desc: "[B, H, S, D]"
    semantics: Query tensor.
  - name: O
    kind: out
    dtype: fp16
    shape: [batch, heads, seq_len, head_dim]
    shape_desc: "[B, H, S, D]"
    semantics: softmax(Q K^T / sqrt(d)) V.
quantization:
  scheme: none                   # or fp8_e4m3, int8, …; plus optional granularity/block_size/group_size
reference_sources:               # relative file paths; must exist at accept time
  - src/ref_attention.py
status: normal                   # normal | frozen | deprecated (frozen blocks claiming)
tags: [attention, fp16]          # optional, for WebUI filters
owner: alice                     # optional

Field-by-field (required unless marked optional):

Field	Purpose & why
`schema_version`	semver for future schema migration gating.
`name`	human-readable display name for the WebUI.
`category`	the four classification axes. Why: it duplicates the directory prefix so a single file can be classified without walking paths — but `index.json` reads this field, not the path, so the two must agree or the WebUI mis-classifies.
`summary`	≤300 chars; shown in `/api/kernels` listings.
`description`	long markdown, the contract for agents. Why: an optimizer agent must be able to write a competitive implementation without reading your `src/`. Structure it as math definition / input layout / perf hints / known traps.
`io_signature`	ordered I/O contract. `kind` is `in`/`out`/`inout`; `dtype` uses generic names (`fp16`, `bf16`, `fp8_e4m3`, `int32`) not vendor naming. `shape` entries may be integers or symbolic names.
`quantization`	`scheme` required; `granularity` ∈ `per_tensor`/`per_block`/`per_channel`; sizes are optional. Use `scheme: none` for unquantized.
`reference_sources`	array of file paths relative to the package. Why: the server verifies these exist before accepting a result — this is what stops a submission from silently referencing files that were never shipped. Note: it is not URLs/DOIs; put paper references in `description`.
`status`	`normal`/`frozen`/`deprecated`. `frozen` prevents claiming (used to stop old packages being re-benchmarked).
`min_arch_feature` / `tags` / `owner`	optional: runner arch-feature selection, WebUI filters, attribution.

Anti-patterns (all rejected by the schema, which is closed — additionalProperties: false):

acceptance.* fields — the accept policy lives in benchmark.py, not here.
reference_sources containing URLs or DOIs — only relative file paths.
io_signature items missing name/kind/dtype/shape, or using a vendor dtype like float16 instead of fp16.

7. Step 4 — port test-data generation into `build_test_env.py`

This file constructs the experiment in the runner's temporary workdir: it deterministically generates every input tensor, synthetic/model weight, and expected output tensor (and optionally compiles the kernel) so that benchmark.py only has to run and measure. All generated files and runtime evidence — including manifests, profiler summaries, logs, census data, and textual analyses — must remain in the temporary workdir or an external debug channel. Never check any raw or summarized runtime artifact into Git or include it in a submission tarball; only the authored recipe belongs in the package.

Contract

Must define build_env(workdir: Path) -> None.
Must be runnable as a script: python build_test_env.py <workdir>.
Writes (the runner reads exactly these locations):
- workdir/inputs/<test_case_id>_*.npy — input tensors
- workdir/expected/<test_case_id>_*.npy — expected outputs (for correctness comparison)
- workdir/build/ — optional compile artifacts (.so, .cubin, …)
- workdir/build_manifest.json — generation parameters + file sha256

The hard requirement: determinism. Under the same desc.yaml and the same workdir, build_env must produce bit-identical outputs every time. Use a fixed constant seed (np.random.default_rng(SEED)) — never time(), os.urandom, or random.random(). Why: the whole leaderboard is comparable only if two runs of the same code face identical inputs and expected outputs. A non-deterministic generator creates false regressions and destroys the ranking's meaning.

Binary output rule: .npy, .npz, weight files, and compiled artifacts produced in workdir/inputs/, workdir/expected/, or workdir/build/ are temporary runner outputs. Do not copy them back into the package, commit them to Git, or place them in a tarball.

Why separate construction from evaluation? Building inputs/expected offline lets you reuse them across repeated benchmark runs without regenerating (random inputs would break correctness comparison), while keeping all generated binary data and platform-specific build artifacts out of the repository.

Minimal shape (this mirrors the example package):

from __future__ import annotations
import json, sys
from pathlib import Path
import numpy as np

PKG_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PKG_ROOT / "src"))
from my_kernel import reference_fn  # your reference implementation

TEST_CASE_PARAMS = {           # must mirror KERNEL_ZOO_TEST_CASE_METAS in benchmark.py
    "small_b1_h1_s16_d16": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16},
    "large_b1_h2_s64_d32": {"batch": 1, "heads": 2, "seq_len": 64, "head_dim": 32},
}
SEED = 0xBEEF                 # fixed; do not change

def build_env(workdir: Path) -> None:
    (workdir / "inputs").mkdir(parents=True, exist_ok=True)
    (workdir / "expected").mkdir(parents=True, exist_ok=True)
    rng = np.random.default_rng(SEED)
    for tc_id, params in TEST_CASE_PARAMS.items():
        q = rng.standard_normal((params["batch"], params["heads"], params["seq_len"], params["head_dim"])).astype(np.float32)
        o = reference_fn(q, q, q)              # your reference produces the expected output
        np.save(workdir / "inputs" / f"{tc_id}_q.npy", q)
        np.save(workdir / "expected" / f"{tc_id}_o.npy", o)
    (workdir / "build_manifest.json").write_text(json.dumps({"seed": SEED}))

def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: build_test_env.py <workdir>", file=sys.stderr); return 2
    build_env(Path(argv[1])); return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

System interaction: the runner calls this before the benchmark, on a fresh workdir. If it exits non-zero, the runner reports build_ok=false and the server rejects the result outright.

8. Step 5 — port correctness + performance into `benchmark.py`

This is the heart of the package and where your existing tests do most of their work. It runs the kernel, checks correctness against the expected outputs, measures metrics, and — critically — decides accept or reject. The server trusts this decision.

Contract

Defines module-level KERNEL_ZOO_TEST_CASE_METAS: list[dict].
Runnable as python benchmark.py <workdir>.
stdout is exactly one JSON document (schema benchmark_output.json); every log/error line goes to stderr. The runner parses stdout as JSON — one stray line breaks the result.
Makes the accept/reject decision; the server does not re-judge.

KERNEL_ZOO_TEST_CASE_METAS — one entry per test case:

KERNEL_ZOO_TEST_CASE_METAS = [
    {
        "test_case_id": "small_b1_h1_s16_d16",       # ^[A-Za-z0-9._-]+$, unique
        "params": {"batch": 1, "heads": 1, "seq_len": 16, "head_dim": 16},  # free-form
        "primary_metric": "primary_latency_ms",       # must be one of metrics[].name
        "metrics": [
            {"name": "primary_latency_ms", "unit": "ms", "direction": "lower_better"},
            {"name": "tflops", "unit": "TFlops", "direction": "higher_better"},
        ],
        "tolerance": {"correctness_atol": 1e-4, "correctness_rtol": 1e-4},
    },
    # ... one per test case ...
]

Why direction lives here and not in bench_result.yaml: the best-value comparison rule is self-declared by the benchmark (per case), so the server can apply it without re-deriving intent; the scoreboard only stores numbers.

Output JSON (core fields; see ../schemas/benchmark_output.json):

{
  "schema_version": "1.0.0",
  "kernel_id": "gfx928/Attention/MHA/fp16/my_flash_attn",
  "ran_at": "2026-08-03T11:00:00Z",
  "runner_id": "runner-sh01",
  "arch": "gfx928",
  "overall": {"accept": true, "accept_reason": "all cases passed; within 5% of best", "build_ok": true},
  "cases": [
    {
      "test_case_id": "small_b1_h1_s16_d16",
      "correctness": "passed",
      "correctness_detail": {"max_abs_err": 1.2e-4, "max_rel_err": 8e-5},
      "accept": true,
      "accept_reason": "primary_latency 1.20ms vs current 1.23ms (-2.4%)",
      "metrics": [{"name": "primary_latency_ms", "value": 1.20}, {"name": "tflops", "value": 318.7}],
      "error": null
    }
  ]
}

Server-side hard constraints (violating any of them rejects the result):

The JSON passes the benchmark_output.json schema.
cases[].test_case_id is exactly the set in KERNEL_ZOO_TEST_CASE_METAS — no fewer, no more.
If overall.accept is true, every case must have correctness == "passed".
overall.build_ok must be true.

How your existing tests map in. Your correctness assertions (assert_allclose(out, expected, atol=..., rtol=...)) become an explicit correctness check against workdir/expected/:

exp = np.load(expected / f"{tc_id}_o.npy")
abs_err = float(np.max(np.abs(out - exp)))
rel_err = float(np.max(np.abs(out - exp) / (np.abs(exp) + 1e-12)))
passed = abs_err <= tol["correctness_atol"] and rel_err <= tol["correctness_rtol"]

Your micro-benchmark's timing becomes a metric, and your suite's pass/fail summary becomes the accept policy. A sensible default policy:

each case accepts if correctness passes and its primary metric does not regress more than a fixed tolerance (e.g. 5%) vs the current best in bench_result.yaml;
overall.accept = all(case.accept).

Anti-patterns (all cause rejections or runtime failures):

printing logs to stdout (the JSON parser breaks);
correctness == "failed" on a case that still has accept: true (422);
metrics[].value being NaN/Inf;
omitting a test case from cases because it is "not important";
smuggling the result through a side channel (env vars, files) — the runner only reads stdout.

System interaction: the runner runs python benchmark.py <workdir>, validates stdout against the schema, and posts the JSON (plus base64 runner logs) to POST /api/submissions/{uuid}/result. On accept, the server rewrites bench_result.yaml and commits via a two-phase commit — the SHA it stamps is real and present in git log (it cannot be predicted ahead of time, hence two commits, not an amend).

9. Step 6 — seed `bench_result.yaml`

Even though only the server owns the real scoreboard, .kernel_package/ must contain all four files to be valid — so you ship a seed version. The server overwrites it on the first accepted submission.

schema_version: "1.0.0"
kernel_id: gfx928/Attention/MHA/fp16/my_flash_attn   # must equal the directory path
cases:
  - test_case_id: small_b1_h1_s16_d16
    metrics_best: {}
    metrics_current: {}
    correctness_current: unknown
  - test_case_id: large_b1_h2_s64_d32
    metrics_best: {}
    metrics_current: {}
    correctness_current: unknown

Rules that matter:

Leave metrics_best and metrics_current empty ({}) in the seed — never fill them with placeholder values. A kernel with no accepted submission shows - for both its current value and historical best in the webui; the server writes the real records on the first accept.
correctness_current flags whether the current HEAD passes correctness; set it to unknown in the seed. The server updates it on accept. (Legacy seeds used a commit_sha: "0000000" sentinel; the server still overwrites those unconditionally, but new seeds no longer need it.)
Only metrics_best and metrics_current are stored — history is recovered via git log. This keeps the file from growing.
Include this required seed in the submission tarball (see §11). The server validates it, strips the submitted copy on accept, and preserves its server-owned scoreboard.

10. Step 7 — local validation loop (run before you submit)

# 1. Validate the four files exist and every schema passes.
kernel_zoo-validate-package <kernel_package directory>
#    (or: python -m kernel_zoo.cli.validate_package <dir>)
#    exit 0 = valid, 2 = validation issues, 3 = IO/usage error

# 2. Simulate the runner locally.
WORKDIR=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR"
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/benchmark.py" "$WORKDIR" > /tmp/result.json 2>/tmp/bench.log

# 3. Validate the emitted JSON against the schema.
python -c "import json, jsonschema; \
  jsonschema.validate(json.load(open('/tmp/result.json')), \
  json.load(open('schemas/benchmark_output.json')))"

# 4. Idempotency check (the hard requirement).
WORKDIR2=$(mktemp -d)
python "$PWD/gfx928/Attention/MHA/fp16/my_flash_attn/.kernel_package/build_test_env.py" "$WORKDIR2"
diff -r "$WORKDIR/inputs" "$WORKDIR2/inputs" && echo "inputs identical"
diff -r "$WORKDIR/expected" "$WORKDIR2/expected" && echo "expected identical"

11. Step 8 — package and submit

The tarball layout is the source-only on-disk layout: the top-level directory is the kernel_id, and .kernel_package/ carries all four files, including a seed bench_result.yaml (submission validation requires it). Before packaging, verify that the package tree contains no binary files: generated tensors, model weights, golden data, and compiled artifacts belong only in the temporary runner workdir.

gfx928/Attention/MHA/fp16/my_flash_attn/     ← top-level dir name = kernel_id
├── .kernel_package/
│   ├── desc.yaml
│   ├── bench_result.yaml                    ← seed (see §9)
│   ├── build_test_env.py
│   └── benchmark.py
└── src/...                                  ← your implementation

The submitted bench_result.yaml is used for validation only. On accept the server strips it from the tarball before writing the source area and keeps its own scoreboard, so a submitter can never fabricate or overwrite results.

import io, tarfile
from pathlib import Path

KERNEL_ID = "gfx928/Attention/MHA/fp16/my_flash_attn"
pkg_root = Path(KERNEL_ID)                     # local package root (has .kernel_package/ + src/)

buf = io.BytesIO()
with tarfile.open(fileobj=buf, mode="w:gz") as tf:
    tf.add(pkg_root, arcname=KERNEL_ID)   # the whole package root, seed included
tarball_bytes = buf.getvalue()

The server strips bench_result.yaml on accept, so do not exclude that required seed. The binary-free rule is different: binary files must be absent from the package tree before tf.add runs; never rely on the server to remove weights or generated data.

Submit (the long-poll returns the queued/running/done status; the default timeout is 30 s):

curl -X POST http://127.0.0.1:8000/api/submissions \
  -H "X-Submitter-Id: alice" \
  -F "kernel_id=$KERNEL_ID" \
  -F "submitter_id=alice" \
  -F "optimization_summary=vectorized loads; reduced register spills" \
  -F "package=@package.tar.gz;type=application/gzip"

curl -H "X-Submitter-Id: alice" \
  http://127.0.0.1:8000/api/submissions/<uuid>/status

optimization_summary is optional free text for a concise, human-authored description of code or algorithm changes. It is not a channel for traces, profile dumps, census output, logs, large result tables, or generated evidence summaries. On accept it is written into the source commit message as an optimization_summary: block, so every attempt in git log records the optimizer's own notes.

On accept, the server overwrites src/, updates bench_result.yaml, commits twice (source commit → bench-result commit), and rebuilds index.json — your operator is on the leaderboard.

12. Design rationale, in one place

Decision	Why
git repo = database	no DBMS, no schema migration; `git log` is the entire history
`src/` separate from `.kernel_package/`	your implementation vs. the platform's records never mix; server can wholesale-overwrite `src/` without touching the scoreboard
`bench_result.yaml` server-owned	the scoreboard is never written from a submission — on accept the server strips the submitted copy and writes its own, so submitters cannot fabricate scores
`benchmark.py` decides accept	the server stays a thin shell and never re-implements your operator's correctness/regression semantics
`build_test_env.py` deterministic	bit-identical inputs/expected ⇒ comparable, trustworthy leaderboard
construction separated from evaluation	reuse generated data across runs; keep build artifacts out of git
stdout = one JSON, logs to stderr	the runner parses stdout; any interleaving corrupts the result
two-phase git commit	the accept-commit's SHA cannot be known before committing; two commits keep the stamped SHA real
`reference_sources` are file paths	the server can verify the shipped sources actually exist
closed schemas (`additionalProperties: false`)	catches stale/foreign fields (e.g. old `acceptance.*`) at validation time, not in production

13. Pre-submission checklist

Directory & naming

every path segment matches ^[A-Za-z0-9._-]+$
.kernel_package/ has all four files
desc.yaml.category == first 4 path segments
bench_result.yaml.kernel_id == directory path
the kernel_id in benchmark output == directory path

desc.yaml

all reference_sources paths exist inside the package
description is self-contained (an agent could write a competitive impl from it alone)
status is not frozen
no acceptance.* / URL reference_sources (schema rejects them)

build_test_env.py

defines build_env(workdir) and runs as python build_test_env.py <workdir>
fixed random seed (determinism is mandatory)
generates inputs, synthetic weights, expected outputs, and build products only under the temporary workdir
writes to workdir/inputs/, workdir/expected/, workdir/build_manifest.json
no generated tensor, weight, golden-data, compiled binary, trace, profile, census, log, report, captured output, or raw/derived evidence in any encoding is stored in the package or submission tarball

benchmark.py

KERNEL_ZOO_TEST_CASE_METAS matches bench_result.yaml.cases
each primary_metric appears in that case's metrics[].name
runs as python benchmark.py <workdir>
stdout is exactly one valid JSON; all logs go to stderr
a case with correctness: failed also has accept: false
no NaN/Inf metric values
the accept policy is actually implemented (it decides, it doesn't just emit metrics)

Tarball

top-level dir name == kernel_id
includes .kernel_package/bench_result.yaml (seed; the server strips it on accept)
contains authored source, metadata, and generation recipes only; no generated binaries or runtime evidence such as traces, profiles, census data, logs, reports, captured output, or raw/derived summaries in any encoding
filename ends in .tar.gz

14. References

kernel_package_format.md — normative on-disk format and every schema (English).
kernel_package_guide.md — 简体中文：从零创建 kernel_package 的完整中文编写指南.
api_reference.md — HTTP API, roles, error envelope.
examples/gfx928/Attention/MHA/fp16/reference_impl/ — a complete, schema-validating example package you can copy as a skeleton.
../api/openapi.yaml — the normative OpenAPI contract.
../README.md — project overview (English) / ../README.zh-CN.md (简体中文).

Navigation Menu

Uh oh!

kernel_package authoring guide (English)

1. The system you are plugging into

2. The target layout

Source-authored, generated-evidence-free package policy

3. Inventory your existing project

4. Step 1 — choose `kernel_id` and your test cases

5. Step 2 — place your implementation in `src/`

6. Step 3 — write `desc.yaml`

7. Step 4 — port test-data generation into `build_test_env.py`

8. Step 5 — port correctness + performance into `benchmark.py`

9. Step 6 — seed `bench_result.yaml`

10. Step 7 — local validation loop (run before you submit)

11. Step 8 — package and submit

12. Design rationale, in one place

13. Pre-submission checklist

14. References

Uh oh!

FilesExpand file tree

kernel_package_guide_en.md

Latest commit

History

kernel_package_guide_en.md

File metadata and controls

kernel_package authoring guide (English)

1. The system you are plugging into

2. The target layout

Source-authored, generated-evidence-free package policy

3. Inventory your existing project

4. Step 1 — choose kernel_id and your test cases

5. Step 2 — place your implementation in src/

6. Step 3 — write desc.yaml

7. Step 4 — port test-data generation into build_test_env.py

8. Step 5 — port correctness + performance into benchmark.py

9. Step 6 — seed bench_result.yaml

10. Step 7 — local validation loop (run before you submit)

11. Step 8 — package and submit

12. Design rationale, in one place

13. Pre-submission checklist

14. References

4. Step 1 — choose `kernel_id` and your test cases

5. Step 2 — place your implementation in `src/`

6. Step 3 — write `desc.yaml`

7. Step 4 — port test-data generation into `build_test_env.py`

8. Step 5 — port correctness + performance into `benchmark.py`

9. Step 6 — seed `bench_result.yaml`