feat: implement the case matrix — records, reachability, methods, and the mark's exit - #13
Merged
Merged
Conversation
… the mark's exit
`test/spec/` was written before the implementation: 174 behaviours of which 111 were
`@test_broken`. This fills them in. 176 behaviours now, all live assertions, 0 broken.
The five questions the package answers, and what was added for each:
* **what this run entered** — unchanged, and now the floor rather than the whole thing.
* **how often, by which paths, how much of the run** — `record`. It emits nothing new: opening
a block clears every probe's flag, so the short-circuit fails and the *write* side, which is
a call rather than an inlined store, does the counting. Counts are exact, survive inlining,
and are correct under threads (per-thread padded counters sized by `maxthreadid()`). Time
comes from Julia's sampler through a new `Profile` extension, so `inclusive`/`exclusive` are
`missing` — never `0.0` — when nobody measured. `assert_clean`, `write_record`, `stamp`.
* **what a caller depends on without naming it** — `reach`, over inferred un-optimised IR.
Three-valued: `:depends`, `:clean`, and `:unknown` for a call site that cannot be pinned to a
method. A site with several candidates is walked through all of them, so "cannot tell which"
only becomes `:unknown` when one of them is actually marked. Module and script entry points,
`ignore=`, `dependents`.
* **what is unfinished** — marks now record the signature they attached to, so `reach` reports
the marked dispatch path and not its siblings, while `audit` still reads the name. Qualified
definitions (`Base.show(io, ::T) = …`) are accepted as method-level marks and stored in the
module that wrote them; `mark_method!` is the imperative route. `audit` gained the method
half — `contributed_methods`, `unaccounted_methods` — which is the only half a package whose
surface is `fetch(model, quantity)` has. `verification`/`coverage` join marks against
`--code-coverage`, flushed in-process.
* **may this mark go** — `until=` states the exit condition next to the reason;
`ready_to_promote` calls it, `marks_without_exit` reports the marks that never said, `age`
and `stale_since` read `since`.
Defects the spec caught, all of them a mark silently recording the wrong thing:
* `(c::C)(x) = …` marked `:c`, the argument name — and the audit then reported `:c` dangling
AND `:C` unaccounted, telling the author to declare what that line declares. Both fixed.
* `since = "0.4.0"` and a non-string reason were refused by the field's own conversion, with a
`MethodError` naming neither the keyword nor `@experimental`.
* a name-keyed mark made every sibling method experimental; `stable` now keeps a name in the
covenant until every method behind it is marked.
* the spec's own `invoke` case named an entry signature no method matched, and its
`@eval`-in-a-loop fixture interpolated at the wrong level. Both invisible while Broken.
One requirement was **withdrawn**: a mark inside a function body cannot be refused with a message
naming `@experimental`. `const` in local scope fails during lowering, before any emitted code
runs, and Julia's message does not name the variable either — measured byte-identical for a
binding whose name is the whole explanatory sentence. `test/spec/README.md` records the three
routes and why each fails; what is kept and asserted is that the blame lands on the author's line.
Also: `@generated`, `Base.@kwdef`, `@inline` and the other annotating macros compose with the
mark; a Documenter extension renders `@experimental` blocks; the API reference is split by source
file so a new file with no page is a build failure.
Verified: every test file individually on 1.12.2 and 1.11.9, and one full `runtests.jl` — 977
assertions, green, 2m49s with coverage and four threads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
📚 Docs preview: https://codes.sota-shimozono.com/ExperimentalAPI.jl/previews/PR13/ (updates on each push to this PR) |
`marks_markdown` was edited after the formatting pass — the heading-free rewrite that the Documenter HTML writer's `!(node.element isa MarkdownAST.Heading)` assertion forced — so `format / format-check` was red on one function. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
`julia nightly — ubuntu-latest` is `continue-on-error`, which makes it allowed to fail and not
allowed to be invisible. It failed 6 of 977. Both causes are real, and one of them was a false
"fully covered" that would have reached a release.
**`Type{X}` is no longer a `DataType`** (measured on 1.14.0-DEV.3115), and `Core.Typeof(Float64)`
now returns the new `Core.TypeEgal{Float64}` rather than `Type{Float64}`. Two `ft isa DataType`
guards — one in `_is_callable_type`, one in `_ftype_identity` — therefore stopped recognising
every constructor call in the graph, and four otherwise-clean fixtures came back `:unknown`. Both
now ask about `t` rather than about how `t` is represented: `t <: Type` with one non-`TypeVar`
parameter, read through `_type_parameter`, and normalised to `Type{X}` before a signature is
built.
**`--code-coverage` now emits a counter for the definition line of a method nothing ever called.**
Up to 1.12 that line had no counter at all, which is what `unverified` was reading: no counters
anywhere in the span meant nothing had ever generated code for it. On 1.14-DEV a one-line
definition comes back at `1.0` — fully covered on the strength of having been *defined*. The two
versions disagree about the same file and only one of them can be read as "the suite ran this".
So `unverified` no longer rests on that. The exact answer was already in the package: the probe.
A marked definition whose flag never fired is `0.0` whatever the line counters say, `unverified`
needs no `--code-coverage` at all now, and coverage supplies only the partial fraction — which is
the question it can still answer. `test/spec/test_spec_verify.jl` lost a version-conditional
branch as a result.
Verified: full suite on 1.14.0-DEV.3115 (978 green), and the affected files on 1.12.2 and 1.11.9.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s on nightly `recording does not disturb Profile` failed on macOS and errored on nightly. Both are defects in what this repository wrote, not in the platforms. **The buffer was not reliably filled.** The test profiled `Sim.driver(M, 200_000)` — about one millisecond, which is *one sampling interval* at Profile's default rate — and then asserted the buffer was non-empty. On ubuntu it caught a sample; on macOS it caught zero, and `before > 0` failed with `0 > 0`. An assertion whose subject is a sample count has to be given a run that is long compared with the interval, so it now profiles `Hot.grind(2_000_000)` at `delay = 1e-5`. That fixture already existed for the attribution test and moved up the file. **`fetch(; include_meta = false)` strips metadata behind an `@assert`.** On 1.14.0-DEV.3115 that assertion fires — `metadata stripping failed` — on a buffer this test did not fill. Two changes: the test reads `Profile.len_data()`, which answers "how much is in the buffer" without fetching or stripping anything; and the `Profile` extension does the strip itself through `has_meta` / `strip_meta` with a fallback to the raw data, because an exception there would have turned `record`'s timing silently off rather than loudly wrong. Verified: `test/spec/test_spec_profile.jl` on 1.12.2, 1.11.9 and 1.14.0-DEV.3115. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`main` moved under this branch: #11 cut the README from 1372 words to 419 for the registry's LLM policy, and #12 replaced the four required matrix contexts with one aggregate. Three files conflicted, and the resolution is not "keep both halves" in any of them: * **README.md** — `main`'s trimmed version wins wholesale. Re-adding the long sections written here would undo the point of #11. What is added back is two things that are now *false* without them: `record` and `reach` did not exist when that README was written, and the closing line called `test/spec/` "the specification for the propagation and profiling work that is not built yet". 559 words, against the 1372 that #11 cut. * **src/audit.jl** and **docs/src/checking.md** — the extension blind spot has a workaround #11 documented (declare the function in the parent, let the extension add only the method) and a report this branch added (`Audit.extensions`). Both are true, so both are said. The "methods on other packages' functions" bullet keeps this branch's version, because it stopped being a blind spot. `docs/make.jl` and `.github/workflows/CI.yml` merged cleanly and correctly — #11's canonical URL with this branch's page list, and #12's `all-tests` job with this branch's corrected runtime comment. Verified after the merge: `test_readme.jl`, `test_dogfood.jl`, `test_audit.jl`, the formatter, and a full docs build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sotashimozono
added a commit
that referenced
this pull request
Sep 8, 2026
…et (#28) Asked whether `@experimental` itself has settled, the churn says yes — `src/mark.jl` has not been touched since #13, across the fourteen commits since. So the question was put to a sweep of 25 definition forms instead, and the sweep found one hole, in the class this package exists to close. @experimental "kernel unverified" @inline f(x) = x # mark outside @inline @experimental "kernel unverified" g(x) = x # mark inside entered(M) → [:g] Both are marked. Both **register a probe**, so both count as observable marked definitions. Only one can ever be entered. With the mark outside, `entered` said "not entered" no matter what ran, `@entered` and `record` reported a clean run through unvalidated code, and `unverified` reported the definition forever. `_subject` was right: `@inline` and its five neighbours are in `_ANNOTATING_MACROS`, the wrapper is rebuilt around the definition, and `instrumentable` is passed through — with a comment saying "an `@inline` marked kernel is exactly the kind that has to be observable". `_instrument`, in the other file, began `(def.head === :function || def.head === :(=)) || return nothing` and handed back `nothing` for the `:macrocall` it was given. Two halves of one feature written to different assumptions. It now recurses through the wrapper and rebuilds it around the instrumented body. **The docstring documented the defect.** Its table read "`Base.@kwdef`, `@inline`, `@noinline` and the other pass-through macros | no", which is what the code did, not what the code was for. The row is now split: the six annotating macros are observed; `@generated` (its body returns an expression) and `Base.@kwdef` (it wraps a struct) are not, and each says why. **The test was named for the claim and asserted something weaker.** `@testset "@inline and the mark compose in both orders"` asserted `Set([:f, :g]) == names marked` — satisfied by a mark that can never fire. It now asserts the observation, and a second testset pins the split that a careless fix would break: instrumenting every macrocall would put a probe inside `@generated`'s returned expression. Three refusals had no test at all — measured by grepping `test/` for their message text and finding zero hits: a wrapping macro this cannot read, a `begin` block with two definitions in it, and a bare qualified name. Each is now pinned on the part that makes it actionable — the name of the macro it could not read, the word `block`, `Sub.g` and the form that would work — rather than on the fact that something was thrown. One incidental find while pinning them: a macro that throws while expanding a `module` body run through `include_string` comes back wrapped in `LoadError` **twice**, so the usual single `.error` leaves a `LoadError` that reads exactly like the failure it hides. 186 behaviours, 1210 assertions, green. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 9, 2026
sotashimozono
added a commit
that referenced
this pull request
Sep 9, 2026
* docs: every block that shows output now produces it
Measured: 33 fenced ```julia blocks across `docs/src`, of which **12 displayed output that had
been typed by hand**. Those 12 are the ones that can be wrong, and three of them were.
All 12 now execute — `@setup` fixtures plus `@repl`/`@example` — and the build fails if their
output stops matching. Typed-output blocks remaining: 0.
What executing them found:
* `@experimental "why" Base.sum(x::Int) = x` is documented as REFUSED. It is accepted: method
marks on another module's generic landed in #13, and the page was never revisited. The
refusal example is now one that is still refused (a bare `Sub.g`), rendered from the real
message, and the accepted-forms list gains the foreign-method case with the note that it is
reported by `contributed_methods` rather than by the name audit.
* `mark(Archeion, :ingest)` was shown without its `signature:` line, which the real display has.
* The `entered()` transcript carried a reason for not being executable — "Documenter evaluates
doctests in a sandbox whose module does not print as `Main`". Measured: an `@repl` block shows
`Main.MyModel.energy`, so the reason no longer holds. It is now `entered(MyModel)`, asked about
a module because the process-wide answer includes every fixture the documentation built.
Two blocks show what a documentation build genuinely gets rather than what a user would, and say
so: `record`'s timings are `missing` because a docs build does not load `Profile`, and
`coverage(...)` is `missing` because it has no `--code-coverage` counters. Both are the documented
contract, now demonstrated instead of described.
`record(; paths = false)` on the observing page: a captured path is the whole call stack, and in a
docs build that starts at `_start` and runs through `makedocs`.
1253 assertions, green; docs build clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: the @entered report was rendered away by Documenter, not lost by the macro
The walkthrough showed
value = ExperimentalAPI.@entered Ising.report(βs)
-1.8886
— the value, and nothing else. The report is the whole of what `@entered` is for.
Not the macro. `IOCapture`, which is what Documenter captures with, sees it:
captured output = "┌ @entered S.f(1.0) at cap.jl:8\n│ Main.S.f ×1 — r\n└ 0 of 1 …\n"
value = 2.0
Measured on a minimal Documenter site, with the control that settles it:
block value rendered
@entered S.f(1.0) 2.0 2.0
println("plain println"); 99 99 99 ← the println is lost too
println("plain println only") nothing plain println only
An `@example` renders the block's value when it has one and its captured output only when the
value is `nothing`. A plain `println` followed by a non-`nothing` value loses its line exactly the
same way, which is what says this is Documenter and not us.
So the walkthrough uses `@entered <io> expr` and prints the buffer — the form `observing.md`
already used for the same reason — and the page now says why, with the control, and adds that a
reader at a terminal writes the plain form. All three `@entered` blocks in the built docs now
carry their report.
1250 assertions, green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/spec/was written before the implementation: 174 behaviours of which 111 were@test_broken. This fills them in. 176 behaviours, all live assertions, 0 broken.What each layer is, and what it costs
recordreachaudit, the coverage joinuntil=,ready_to_promote,age,stale_sincerecordemits nothing newThe statement
@experimentalputs in a body did not change. Opening a block clears everyprobe's flag, so the short-circuit fails and the write side — a function call, not an inlined
store — does the counting. Counts are therefore exact, survive inlining (which is what ruled out
the sampling route: a definition worth marking is small enough to be inlined, and a sampler has no
frame left to attribute to), and are correct under threads via per-thread padded counters sized by
maxthreadid().One change to the emission is load bearing and was found the hard way: the probe statement now
carries the declaration's own
LineNumberNode. Without it the cold write branch inherits whateverstatement the optimiser leaves next to it, and
record's call paths came back reportingiterateand
+whereenergyandinnerbelonged.Time comes from Julia's sampler through a new
Profileextension, soinclusive/exclusivearemissing— never0.0— when nobody measured.reachhas three answers because Julia's call graph is not closed:depends,:clean, and:unknownfor a call site that cannot be pinned to a method. Walksinferred, un-optimised IR:
optimize = trueseesmul_floatand finds nothing. A site withseveral matching candidates is walked through all of them, so "cannot tell which method" only
becomes
:unknownwhen one of the candidates is actually marked — otherwiseconvert(::Type, x)would make every caller that formats a string unknown.
There is deliberately no
verdictfield: a stored one makes:cleanwith a non-emptyunresolvedrepresentable, and that is the single state this must never report.Defects the spec caught
All of them a mark silently recording the wrong thing rather than refusing:
(c::C)(x) = …marked:c, the argument name — and the audit then reported:cdanglingand
:Cunaccounted, telling the author to declare the very thing that line declares.since = "0.4.0"and a non-string reason were refused by the field's own conversion, with aMethodErrornaming neither the keyword nor@experimental.attached to;
stablekeeps a name in the covenant until every method behind it is marked.invokecase named an entry signature no method of its fixture matched, and its@eval-in-a-loop fixture interpolated at the wrong level. Both invisible while Broken.One requirement withdrawn, with the measurement
A mark inside a function body cannot be refused with a message naming
@experimental.constinlocal scope fails during lowering, before any emitted code runs, and Julia's message does not
name the variable either — measured byte-identical for
__EXPERIMENTAL_API_MARKS__and for abinding whose name is the whole explanatory sentence.
globalfails silently;Core.evalremovesthe error altogether.
test/spec/README.mdrecords all three routes. What is kept and asserted isthe part that was in this package's hands: the blame lands on the author's line, never inside this
package.
Verification
Every test file individually on 1.12.2 and 1.11.9, plus one full
runtests.jl: 977assertions, green, 2m49s with coverage and four threads. Docs build clean; the API reference is
split by source file so a new file with no page is a build failure.
Overlaps with the open PRs
CI.yml; this edits only the header comment (the "half a minute"runtime claim is now false). No conflict.
README.md,docs/make.jl,docs/src/checking.mdandsrc/audit.jl, all ofwhich this also changes. Merge docs: trim the README, disclose the LLM assistance, fix the canonical URL #11 first and I will rebase, or the other way round — but not
in parallel.
🤖 Generated with Claude Code