diff --git a/docs/src/index.md b/docs/src/index.md index 3da377b..42dc7f8 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -126,6 +126,23 @@ It is loaded by the package being marked, so it is a normal dependency — but i beyond `TOML`, and `Test` only through a package extension, so a consumer of your package never loads `Test` because of this. +## What this package declares about itself + +Five of its own layers are marked [`@experimental`](@ref), and the reasons are the ones this +package asks of everybody else — a fact you could not write without having gone and looked: + +| layer | why it is not settled | +|---|---| +| [`record`](@ref) and the `Record` it returns | the struct gained a field after `write_record` already had a file format, and `paths` and `timing` are refused together because the pair segfaulted 2 runs in 4 | +| [`@entered`](@ref) | its report is text with no schema, and it changed twice in its first week | +| [`reach`](@ref) | where `:clean` stops and `:unknown` starts is drawn by Julia's internal IR accessors, which differ by minor version | +| [`verification`](@ref) | the counts come from `jl_write_coverage_data` and the `.cov` line format, neither of which Julia documents | +| [`snapshot`](@ref) and [`compare`](@ref) | the file format was a guess, and making it signature-aware changed it | + +`@experimental`, [`entered`](@ref) and [`audit`](@ref) — the three questions on this page — are +not marked, and a test asserts they are not. A package that declared everything would be telling +you nothing. + ## Where to go next - [Declaring](@ref) — the forms `@experimental` accepts, and what it refuses diff --git a/docs/src/observing.md b/docs/src/observing.md index 4e90cc1..f7f0b27 100644 --- a/docs/src/observing.md +++ b/docs/src/observing.md @@ -156,8 +156,11 @@ julia> ExperimentalAPI.@entered sweep(model; βs = 0.05:0.05:2.0) 0.42713… ``` -It returns the value of the expression, so it drops into existing code the way `@time` does. The -last line is what makes a clean answer mean anything: +It returns the value of the expression, so it drops into existing code the way `@time` does — with +one divergence `@time` does not have, since `record` takes a function and the expression therefore +runs inside a closure: a `return` inside it returns from the closure, and `@entered y = f(x)` binds +`y` inside the closure. Write `y = @entered f(x)`. The last line is what makes a clean answer mean +anything: ```julia julia> ExperimentalAPI.@entered publish(result) @@ -169,11 +172,15 @@ julia> ExperimentalAPI.@entered publish(result) not adopted this yet is in the second one. A report that could not tell them apart would read as reassurance on a package where nothing had ever been declared. -It returns the record's `value`, which is what `record` now carries out of the block, so measuring -a call does not cost its result. It is `record(() -> expr; paths = false, timing = false)` plus -the report — the cheap question, -`which` and `how often`, needing neither a backtrace nor a sampler. For call paths, time (never -both — see [`record`](@ref)), or the [`Record`](@ref) as data, call [`record`](@ref). +What it returns is the record's `value` — `record` carries the block's result out, so measuring a +call does not cost you that result. It is `record(() -> expr; paths = false, timing = false)` plus +the report: the cheap question, `which` and `how often`, needing neither a backtrace nor a sampler. +For call paths, time (never both — see [`record`](@ref)), or the [`Record`](@ref) as data, call +[`record`](@ref). + +Name an `io` first to send the report somewhere else — `@entered log sweep(model)` into a file, +`@entered devnull f(x)` to keep only the value. The default is looked up when the block runs, not +when the macro expands, so `redirect_stdout` still catches it. The route is deliberately not printed: a captured path is a list of frame names, and Base's higher-order functions are in it. Measured for `driver(x, n) = sum(inner(x) for _ in 1:n)`, the diff --git a/src/macros.jl b/src/macros.jl index be30275..413d618 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -5,8 +5,15 @@ # the line it was written on. A report that says which call went through unvalidated code, and # where that call is, is a different thing from a list of names. +@experimental """ +the report is a text format with no schema, and it has already changed twice in its first week — \ +the hit lines gained a sort order and column alignment, the footer a singular verb. Anything that \ +parses this output is parsing a guess +""" @entered + """ @entered expr + @entered io expr Evaluate `expr`, print which marked definitions it went through, and return its value. @@ -76,14 +83,28 @@ measurement in its docstring), and for the [`Record`](@ref) as data. This return so at global scope no `y` appears afterwards. Write `y = @entered f(x)` instead — which is what the value coming back is for. +The report goes to `stdout` unless an `io` is named first — `@entered log sweep(model)` puts it in +a file, `@entered devnull f(x)` throws it away and leaves only the value. The default is looked up +when the block runs rather than when the macro expands, so `redirect_stdout` still catches it. + See also [`entered`](@ref) for the whole-process question, [`record`](@ref) for the full instrument, and [`reach`](@ref) for the same question asked without running anything. """ macro entered(ex) - src = __source__ + return _entered_expr(:stdout, ex, __source__) +end + +macro entered(io, ex) + return _entered_expr(esc(io), ex, __source__) +end + +# One body for both arities. `io` arrives already escaped when the caller named one, and as the +# bare symbol `stdout` when they did not — resolved in this module at run time, so the destination +# is not frozen at macroexpansion and `redirect_stdout` still reaches it. +function _entered_expr(io, ex, src::LineNumberNode) return quote local rec = $(record)(() -> $(esc(ex)); paths=false, timing=false) - $(_report_entered)(stdout, rec, $(QuoteNode(ex)), $(QuoteNode(src))) + $(_report_entered)($io, rec, $(QuoteNode(ex)), $(QuoteNode(src))) rec.value end end diff --git a/src/reach.jl b/src/reach.jl index cb9b6ac..8280e3e 100644 --- a/src/reach.jl +++ b/src/reach.jl @@ -17,6 +17,13 @@ # call site with no unique method, and a function passed as a value is specialised on `typeof(f)` # and resolves. +@experimental """ +the line between `:clean` and `:unknown` is drawn by Julia's own IR accessors, which are internal \ +and differ by minor version: a statement's line comes from `Base.IRShow.getdebugidx` on 1.12 and \ +from `codelocs`/`linetable` on 1.11, and a callee spells itself `Core.TypeEgal{T}` on 1.14-DEV \ +where earlier versions say `Type{T}`. A verdict that moves with the compiler is not yet a covenant +""" reach reach_script Reach Reached Unresolved verdict isclean combine dependents + """ Unresolved diff --git a/src/record.jl b/src/record.jl index 690e129..49bd5d1 100644 --- a/src/record.jl +++ b/src/record.jl @@ -8,6 +8,13 @@ # and the write side — which is a function call, not an inlined store — does the counting. Nothing # in the body changes, and nothing outside `record` pays for any of it. +@experimental """ +the record's shape and its collection knobs are both still moving: `Record` gained a field after \ +`write_record` already had a file format, so a file written by an earlier version reads back with \ +that field empty, and `paths` and `timing` are refused together because the pair segfaulted 2 runs \ +in 4 where each alone crashed 0 in 4 +""" record recording Record Hit Attribution attribute experimental_fraction merge_records write_record read_record assert_clean TimingBackend timing_backend + """ Hit diff --git a/src/verify.jl b/src/verify.jl index c921f0b..09242dd 100644 --- a/src/verify.jl +++ b/src/verify.jl @@ -9,6 +9,12 @@ # Julia writes at exit, because a test that has to wait for the process to end cannot assert # anything. +@experimental """ +the counts come from two interfaces Julia does not document — `ccall(:jl_write_coverage_data, …)` \ +to flush from the running process, and the `.cov` line format to read them back — so a Julia \ +release can change what this reports without anything here changing +""" verification Verification coverage coverage_enabled unverified stale_marks flush_coverage + """ Verification diff --git a/test/test_dogfood.jl b/test/test_dogfood.jl index 8d9f74e..4a48781 100644 --- a/test/test_dogfood.jl +++ b/test/test_dogfood.jl @@ -1,33 +1,124 @@ # The package runs its own check on itself. Every public name of ExperimentalAPI is either -# documented or declared @experimental — including the release layer, which is declared because -# its file format is a guess, not because declaring it was convenient. +# documented or declared @experimental — including four whole layers that are declared because +# what they rest on is a guess, not because declaring them was convenient. -using ExperimentalAPI: ExperimentalAPI, audit, experimental, test_surface +using ExperimentalAPI: ExperimentalAPI, audit, experimental, isexperimental, test_surface using Test @testset "ExperimentalAPI accounts for its own surface" begin test_surface(ExperimentalAPI) end -@testset "the release layer says what it is" begin +# Which layers say they are not settled, and the fact each one's reason has to name. The anchor is +# a measured detail rather than the word "experimental", because a reason that could be written +# without doing the measurement is the placeholder this package exists to refuse. +const YOUNG_LAYERS = Dict( + :release => ( + "schema", + [ + :snapshot, + :read_snapshot, + :write_snapshot, + :compare, + :compare_methods, + :isbreaking, + :stamp, + :Diff, + :MethodDiff, + ], + ), + :record => ( + "segfault", + [ + :record, + :recording, + :Record, + :Hit, + :Attribution, + :attribute, + :experimental_fraction, + :merge_records, + :write_record, + :read_record, + :assert_clean, + :TimingBackend, + :timing_backend, + ], + ), + :macros => ("changed twice", [Symbol("@entered")]), + :reach => ( + "getdebugidx", + [ + :reach, + :reach_script, + :Reach, + :Reached, + :Unresolved, + :verdict, + :isclean, + :combine, + :dependents, + ], + ), + :verify => ( + "jl_write_coverage_data", + [ + :verification, + :Verification, + :coverage, + :coverage_enabled, + :unverified, + :stale_marks, + :flush_coverage, + ], + ), +) + +@testset "the young layers say what they are" begin declared = Set(mk.name for mk in experimental(ExperimentalAPI)) - @test declared == Set([ - :snapshot, - :read_snapshot, - :write_snapshot, - :compare, - :compare_methods, - :isbreaking, - :stamp, - :Diff, - :MethodDiff, - ]) - for mk in experimental(ExperimentalAPI) - @test occursin("schema", mk.reason) # the reason is the real one, not a placeholder + @test declared == Set(Iterators.flatten(last(v) for v in values(YOUNG_LAYERS))) + for (layer, (anchor, names)) in YOUNG_LAYERS, n in names + mk = only(ExperimentalAPI.marks(ExperimentalAPI, n)) + @test occursin(anchor, mk.reason) @test !isempty(strip(mk.reason)) end end +@testset "the settled core is not declared experimental" begin + # The control the testset above cannot be: an equality against a hand-written set is satisfied + # by marking every name and updating the set to match. These are the names the front page + # promises answers from, and a promise is exactly what a mark withdraws. + for n in [ + Symbol("@experimental"), + :Mark, + :mark, + :marks, + :marks_on, + :isexperimental, + :experimental, + :experimental_methods, + :Probe, + :entered, + :probes, + :detecting, + :summary_text, + :marked_modules, + :Audit, + :audit, + :surface, + :stable, + :isdocumented, + :own_methods, + :contributed_methods, + :ready_to_promote, + :age, + :docstring_note, + ] + @test n in audit(ExperimentalAPI).surface + @test !isexperimental(ExperimentalAPI, n) + end +end + @testset "the extension declares its own knobs, and the parent can see it" begin # An extension is a separate module: its public names are part of the surface a user sees and # are invisible to `names(ExperimentalAPI)`. `extensions = true` is what reaches them. diff --git a/test/test_macros.jl b/test/test_macros.jl index e467a56..4b05ee8 100644 --- a/test/test_macros.jl +++ b/test/test_macros.jl @@ -196,3 +196,26 @@ end @test Base.ispublic(ExperimentalAPI, Symbol("@entered")) @test !Base.isexported(ExperimentalAPI, Symbol("@entered")) end + +@testset "@entered writes where it is told, and only there" begin + # The knob exists because the report is text on a stream: a caller who wants it in a log, or + # does not want it at all, otherwise has to redirect the whole process to get at one line. + buf = IOBuffer() + value, printed = grab() do + ExperimentalAPI.@entered buf MacroFixture.energy(2.0) + end + @test value == MacroFixture.energy(2.0) + report = String(take!(buf)) + @test occursin("@entered", report) + @test occursin("MacroFixture.energy", report) + # Control: naming an `io` MOVES the report rather than copying it. Without this the two-arg + # form would pass while still printing to the terminal. + @test isempty(printed) + + # And the default is still looked up when the block runs, not when the macro expands, so a + # redirect around the call reaches it. + _, printed2 = grab() do + ExperimentalAPI.@entered MacroFixture.energy(2.0) + end + @test occursin("@entered", printed2) +end