Compile Python load_inline submissions on CPU before Modal GPU evaluation

[msaroufim]

Python submissions that compile with load_inline currently hold a GPU while the compiler runs. The Modal launcher now first imports those submissions in a CPU worker, transfers the compiled extensions, and reuses matching libraries during GPU evaluation. Users keep their existing submission.py and load_inline calls.

CPU import errors, timeouts, unsupported builds, and mismatched sources, flags, ABI, image, or header dependencies fall back to normal GPU compilation. Other Python submissions use the existing GPU path. The evaluator and correctness decisions are unchanged. Workers share the runner image and read-only PCH Volume; CPU containers are single-use with restricted Modal access. Artifacts are scoped to one request and limited to 1 MiB compressed. KERNELBOT_CPU_COMPILE=0 disables the CPU attempt.

Validation:

• Full API end-to-end run: unchanged submission uploaded with CLI authentication, queued in real local Postgres, processed by the background manager, and polled to success. All six public/secret test, benchmark, and leaderboard runs passed with 12 artifact reuses and zero fallbacks.
• CI passed: 305 unit tests, seven Modal integration tests, lint, and the GitHub integration job. Four existing multi-GPU Modal tests remain skipped. All 60 focused local tests and diff checks also passed.
• Real Modal T4 run: unchanged inline submission passed test, benchmark, and leaderboard phases with six artifact loads and zero fallbacks. The changed-header check also passed after simplifying the implementation: it rejected an incorrect CPU binary and rebuilt on GPU.
• Compatibility run: GPU work during import, lazy compilation, Triton, plain PyTorch, and generated source changes passed through their expected paths.
• Modal CI confirmed artifact reuse on both T4 and H100; these tests now fail if the runner silently recompiles on GPU.

The guide includes the local debug command, supported scope, and rollout instructions. Production has not been deployed.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
inline_artifacts.py					59, 109
python_precompile.py					30, 32, 52, 55, 69, 82, 87-104, 120-122, 163, 168
utils.py
Project Total

This report was generated by python-coverage-comment-action