Skip to content

[Lang] Cache spec keys for None kernel arguments - #877

Open
hughperkins wants to merge 1 commit into
mainfrom
hp/w1-nonetype-primitive-types
Open

[Lang] Cache spec keys for None kernel arguments#877
hughperkins wants to merge 1 commit into
mainfrom
hp/w1-nonetype-primitive-types

Conversation

@hughperkins

@hughperkins hughperkins commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

A kernel argument of None (an absent optional qd.Tensor / qd.template() slot) is ~21% slower per launch than it should be, because its template-mapper spec-key cache entry is never stored.

TemplateMapper.lookup weakref-tracks every argument whose type is not in _primitive_types, so it can evict the spec-key cache entry when the argument is garbage-collected. weakref.ref(None) raises TypeError, which was caught and downgraded to a warn_once, leaving the entry unstored - so every launch passing None re-ran full spec-key extraction instead of hitting the cache.

This adds NoneType to _primitive_types. None is an immortal singleton that never needs lifetime tracking, so excluding it from the weakref loop is both correct and removes the overhead.

Measurements

CPU, 20000 iterations, microseconds per launch, optional qd.Tensor slot:

variant before after
b=None 148.6 122.8
b=ndarray (present) 123.9 123.9
no optional slot (control) 122.2 122.2

The absent branch does strictly less work than the present branch yet was slower than the control; the gap closes from +20.8% to ~0.3%. Specialization counts and results are unchanged. The spec-key cache now populates for a None argument (0 -> 1 entries).

Test plan

  • New test_none_argument_populates_spec_key_cache passes on x64 and cuda
  • Full CPU suite via the CI harness (tests/run_tests.py -r 3, x64, split by needs_torch exactly as linux/4_test.sh): this branch's unique failing set is identical to unmodified main and the needs_torch phase passes clean, so this PR introduces no new failures.

The only shared failures are three tests that cannot pass in the offline cluster container regardless of branch (they fail the same way on main), all unrelated to this change:

  • test_committed_fatbin_layout - the container's cuobjdump predates CUDA 13 and misreads the committed cubin's build toolkit as 0.2 instead of 13.0+;
  • test_pyi_stubs - python -m pyright cannot fetch its Node runtime on an offline compute node;
  • test_ipython.ipynb - io.UnsupportedOperation from quadrants' stdout wrapping under nbmake's ipykernel capture.

(An earlier revision of this checklist reported raw pytest counts; those were inflated because that run omitted CI's -r 3 flaky-rerun, so load-induced flakes and xdist worker-kill collateral surfaced as hard failures. Under the CI harness they do not.)

Context: first of two prerequisite fixes for optional (None) kernel arguments, motivated by #856.

Made with Cursor

TemplateMapper.lookup weakref-tracks every argument whose type is not in
_primitive_types so it can evict the spec-key cache entry when the argument is
collected. weakref.ref(None) raises TypeError, which was caught and downgraded
to a warn_once, leaving the entry unstored - so every launch passing None
re-ran full spec-key extraction instead of hitting the cache.

Add NoneType to _primitive_types. None is an immortal singleton that never needs
lifetime tracking, so excluding it from the weakref loop is both correct and
removes the ~21% per-launch overhead measured for an absent optional qd.Tensor
argument. Specialization counts and results are unchanged.
@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant