Document CPU preflight limits and separate preparation/evaluation passes

[msaroufim]

CPU compilation during submission import does not guarantee that every benchmark shape is prepared. This documentation PR explains the limitation and the compile-only harness mode needed to traverse supported cases while keeping user kernel source unchanged.

The current implementation selects import-time load_inline builds; it does not enumerate custom_kernel calls or precompile Triton/CuTe specializations. Running the regular evaluator under fake tensors would also be insufficient: CUDA-dependent input generation can stop discovery before the first launch, and a numerical assertion after each case can stop it after that case.

The proposed design uses shared case/input specifications for two passes:

• CPU preparation constructs fake inputs and discovers/compiles reached specializations, without correctness checks or timing.
• GPU evaluation runs all real checks, warmup, timing, and seeded rechecks through the existing evaluator. Missing specializations retain normal JIT fallback.

The note explains why ten known shapes do not automatically mean ten prepared shapes, why benchmark-only cases must be included, and why task-owned input adapters may be necessary. It also covers artifact compatibility, private-workload boundaries, and the evidence needed before making expanded preflight a default.

Documentation only: this PR does not implement the proposed mode, change production dispatch, or change existing CPU extension compilation or scoring. It adds a README link to the design note.

Validation: checked the described flow against ModalLauncher, python_precompile.py, build_task_config, the stock evaluator, and the vector-add task/input generator; verified relative links and git diff --check. No runtime tests or GPU jobs were run for this documentation-only change.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
utils.py
Project Total

This report was generated by python-coverage-comment-action