Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions docs/src/adopting.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,21 @@ and no idea which of them anybody meant.

## Start by looking

```julia
julia> using MyPackage, ExperimentalAPI
```@setup adopting
using ExperimentalAPI
module MyPackage
using ExperimentalAPI
public settled, unfinished, undescribed
"Settled and documented."
settled(x) = x
"Documented, and not finished."
@experimental "the tolerance is a guess" unfinished(x) = x
undescribed(x) = x
end
```

julia> ExperimentalAPI.audit(MyPackage)
```@repl adopting
ExperimentalAPI.audit(MyPackage)
```

Two numbers matter. `undocumented` is the backlog. `dangling` should be zero on day one, because
Expand Down
23 changes: 15 additions & 8 deletions docs/src/analysing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,22 @@ CurrentModule = ExperimentalAPI
[`entered`](@ref) and [`record`](@ref) say what a run *did*. [`reach`](@ref) says what a caller
*could* do — before running it, and including through code that never names the marked thing.

```julia
julia> r = ExperimentalAPI.reach(analyse, Tuple{Model,Float64});

julia> ExperimentalAPI.verdict(r)
:depends
```@setup analysing
using ExperimentalAPI
module MyModel
using ExperimentalAPI
public energy, inner, sweep, analyse
@experimental "convergence not established below β ≈ 0.1" energy(β::Float64) = -log(2cosh(β)) / β
inner(β::Float64) = energy(β) * 2
sweep(βs::Vector{Float64}) = (t = 0.0; for β in βs; t += inner(β); end; t)
analyse(βs::Vector{Float64}) = round(sweep(βs); digits = 4)
end
```

julia> r.reached
1-element Vector{ExperimentalAPI.Reached}:
ExperimentalAPI.Reached(MyModel.energy via analyse → sweep → inner → energy)
```@repl analysing
r = ExperimentalAPI.reach(MyModel.analyse, Tuple{Vector{Float64}});
ExperimentalAPI.verdict(r)
r.reached
```

The model is Lean's `sorry`: a proof that uses one is not a proof, however many layers down it
Expand Down
80 changes: 58 additions & 22 deletions docs/src/checking.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,23 @@ This is the part the rest of the package exists for.
A mark says the shape is unsettled. That is never a reason to say nothing about what the name
does, so a marked name with no prose is a finding exactly as an unmarked one is.

```julia
julia> ExperimentalAPI.audit(Archeion)
Public surface of Archeion — 36 names
documented 33
experimental 0
undocumented 3 ← no docstring
FTPSTransport, pull_file, push_dir
```@setup checking
using ExperimentalAPI
module Archeion
using ExperimentalAPI
public ingest, fetch_record, FTPSTransport, pull_file, push_dir
"Ingest a document."
ingest(x) = x
"Fetch one record."
fetch_record(x) = x
struct FTPSTransport end
pull_file(t::FTPSTransport, p) = p
push_dir(t::FTPSTransport, p) = p
end
```

```@repl checking
ExperimentalAPI.audit(Archeion)
```

The list is never truncated. A coverage report that elides its tail reads as if the tail were
Expand Down Expand Up @@ -110,15 +120,28 @@ see exactly which names it would have to argue about. Empty means the two agree.

## The method-level half

```julia
julia> ExperimentalAPI.audit(Downstream).contributed_methods
4-element Vector{Method}:
fetch_value(::Ising, ::Energy) …

julia> ExperimentalAPI.unaccounted_methods(Downstream) # neither a docstring nor a mark
2-element Vector{Method}:
```@setup checking2
using ExperimentalAPI
module Upstream
struct Ising end
struct Energy end
fetch_value(m, q) = 0.0
end
module Downstream
using ExperimentalAPI
using ..Upstream: Upstream, Ising, Energy
public Widget
struct Widget end
"Documented, on somebody else's generic."
Upstream.fetch_value(::Ising, ::Energy) = 1.0
@experimental "extrapolated; no reference value" Upstream.fetch_value(::Ising, ::Int) = 2.0
Base.show(io::IO, ::Widget) = print(io, "<widget>")
end
```

```@repl checking2
ExperimentalAPI.audit(Downstream).contributed_methods
ExperimentalAPI.unaccounted_methods(Downstream) # neither a docstring nor a mark
```

Docstrings are keyed by signature, so [`isdocumented`](@ref)`(::Method)` is a real question and
Expand All @@ -138,15 +161,28 @@ under the ones Julia adds next. `require_methods = :all` widens it.
The mark records where it was written, and `--code-coverage` records a count per line. Joining the
two answers the worst case a marked definition can be in:

```julia
julia> ExperimentalAPI.unverified(MyPackage) # marked AND never executed by the suite
1-element Vector{ExperimentalAPI.Mark}:
ExperimentalAPI.Mark(MyPackage.never_called, "shipped without ever being called")
```@setup checking3
using ExperimentalAPI
module MyPackage
using ExperimentalAPI
public exercised, never_called
"Run by the suite."
@experimental "reference value not cross-checked" exercised(x) = x + 1
"Never run."
@experimental "shipped without ever being called" never_called(x) = x * 0
end
MyPackage.exercised(1)
```

julia> ExperimentalAPI.coverage(MyPackage, :half_exercised)
0.6
```@repl checking3
ExperimentalAPI.unverified(MyPackage) # marked AND never entered by this process
ExperimentalAPI.coverage(MyPackage, :exercised)
```

`unverified` reads the probe, which is exact and needs no coverage run. `coverage` is the partial
fraction and answers `missing` here, because a documentation build has no `--code-coverage`
counters — which is the next paragraph's point, shown rather than described.

[`coverage`](@ref) answers `missing`, never `0.0`, when the run has no coverage data: a run
without `--code-coverage` has nothing to say, and reporting zero would flag every marked
definition in every ordinary run. The counters are flushed from the running process rather than
Expand Down
58 changes: 39 additions & 19 deletions docs/src/declaring.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,22 @@ Why a name is not settled is knowledge only the author has. A reader can see tha
odd; they cannot see that it is odd *because* a refactor upstream has not landed. So the reason
is required, and the mark carries it everywhere the name goes:

```julia
julia> ExperimentalAPI.mark(Archeion, :ingest)
Archeion.ingest — experimental
reason: signature will be wrapped once the write-back refactor settles
since: v0.1.4
tracking: https://github.com/QAtlasHub/Archeion.jl/issues/12
declared: /…/Archeion.jl/src/ingest.jl:153
```@setup declaring
using ExperimentalAPI
module Archeion
using ExperimentalAPI
public ingest
@experimental(
"signature will be wrapped once the write-back refactor settles",
since = v"0.1.4",
tracking = "https://example.invalid/issues/12",
ingest(config; doc = nothing) = config,
)
end
```

```@repl declaring
ExperimentalAPI.mark(Archeion, :ingest)
```

`since` and `tracking` are optional and go between the reason and the subject:
Expand All @@ -62,23 +71,35 @@ shape they need.
## What it attaches to

`function`, short-form `f(x) = …`, `struct`, `mutable struct`, `abstract type`,
`primitive type`, `macro` (recorded as `Symbol("@name")`), `const`, and plain assignment.
`primitive type`, `macro` (recorded as `Symbol("@name")`), `const`, plain assignment, a definition
wrapped in `@inline` and its neighbours, and a method on **another module's** generic —
`Base.show(io, ::Widget) = …`. That last one is not on `names(m)` and never can be, so it is
reported by [`contributed_methods`](@ref) rather than by the name audit.

Anything else is **refused with a message naming the alternative**, never guessed at:

```julia
julia> @experimental "why" Base.sum(x::Int) = x
ERROR: ArgumentError: @experimental: `Base.sum` defines a name owned by another module,
which is not part of this module's public surface
```@example declaring
try
@eval module Refused
using ExperimentalAPI
module Sub
g(x) = x
end
@experimental "why" Sub.g
end
catch e
showerror(stdout, e isa LoadError ? e.error : e)
end
```

The refused cases and why:

| | |
|---|---|
| `module M … end` | Julia requires it as a direct top-level statement, so it cannot be wrapped — use the name-list form |
| `Base.foo(x) = …` | adds a method to a name this module does not own; it is not on your surface |
| `@somemacro …` | the macro cannot know which name the expansion defines — name it explicitly |
| a bare `Sub.g` | names somebody else's generic without saying **which** method — mark the definition |
| `begin f(x)=x; g(x)=x end` | two names, one mark; marking the first and dropping the second is a covenant that omits a definition |
| `@somemacro …` | outside the annotating allowlist the macro cannot know which name the expansion defines |

Guessing in any of these cases would produce a mark on the wrong symbol, which is worse than no
mark: `audit` would then report it as `dangling` and the author would be debugging this package
Expand Down Expand Up @@ -109,17 +130,16 @@ A mark says nothing about visibility. A name still has to be `export`ed or decla
be part of the surface, and a mark on a name that is neither promises nothing to anyone —
[`audit`](@ref) reports it as `dangling`:

```julia
```@example declaring
module M
using ExperimentalAPI
@experimental "not settled" helper(x) = x # never exported, never public
end
nothing # hide
```

```julia
julia> ExperimentalAPI.audit(M).dangling
1-element Vector{Symbol}:
:helper
```@repl declaring
ExperimentalAPI.audit(M).dangling
```

That check needs no reference to be right. The module is disagreeing with itself.
Expand Down
68 changes: 39 additions & 29 deletions docs/src/observing.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,19 @@ DocTestSetup = quote
end
```

```@setup observing
using ExperimentalAPI
module MyModel
using ExperimentalAPI
public energy, step, sweep, simulate
@experimental "convergence not established below β ≈ 0.1" energy(β::Float64) = -log(2cosh(β)) / β
step(β::Float64) = energy(β) + tanh(β)
sweep(βs::Vector{Float64}) = (t = 0.0; for β in βs; t += step(β); end; t)
simulate(βs::Vector{Float64}; steps::Int = 1) = (t = 0.0; for _ in 1:steps; t += sweep(βs); end; t)
end
βs = collect(0.05:0.05:2.0)
```

# Observing

A docstring can say a name is unfinished. It cannot tell you that *this run* went through it.
Expand Down Expand Up @@ -44,16 +57,14 @@ Three properties, each of them a decision:

[`entered`](@ref) returns the same thing the summary prints, as a `Vector{`[`Entry`](@ref)`}`:

```julia
julia> ExperimentalAPI.entered()
1-element Vector{ExperimentalAPI.Entry}:
ExperimentalAPI.Entry(Main.energy, "convergence not established below β ≈ 0.1")
```@repl observing
MyModel.energy(0.5)
ExperimentalAPI.entered(MyModel)
```

The display above is a transcript rather than a doctest on purpose: `Entry` prints its module, and
Documenter evaluates doctests in a sandbox whose module does not print as `Main`, so a doctest here
would show a line no reader ever sees at their own REPL. The fields it is read for do not depend on
where it ran, so those are checked:
Asked about a module rather than about the whole process, because the process-wide answer includes
every marked package loaded — on this page, the fixtures the documentation built. The fields it is
read for do not depend on where it ran, and those are checked as a doctest:

```jldoctest
julia> using ExperimentalAPI
Expand Down Expand Up @@ -116,21 +127,21 @@ table above.

## Recording: counts, paths and time

```julia
r = ExperimentalAPI.record() do
simulate(model; steps = 10_000)
```@example observing
r = ExperimentalAPI.record(; paths = false) do
MyModel.simulate(βs; steps = 500)
end
nothing # hide
```

```julia
julia> r
Record — 1 marked definition entered in 0.42s
MyModel.energy ×10000 — convergence not established below β ≈ 0.1
inclusive 0.31s exclusive 0.28s
via record → sweep → step → energy
recorder overhead ≈ 4.1%
```@example observing
r
```

`paths = false` here because a captured path is the whole call stack, and on this page that starts
at `_start` and runs through `makedocs`. The timings are `missing`: a documentation build does not
load `Profile`, and `record` reports what it measured rather than a zero.

[`record`](@ref) returns a `Vector`-like of [`Hit`](@ref), so `isempty(r)` and `r[1].count` read
the way they look — with the properties an empty vector could not carry:

Expand All @@ -147,25 +158,24 @@ the way they look — with the properties an empty vector could not carry:
[`@entered`](@ref) is the same question asked about an expression, and it knows two things a
closure cannot — the source text of the call and the line it is written on:

```julia
julia> ExperimentalAPI.@entered sweep(model; βs = 0.05:0.05:2.0)
┌ @entered sweep(model; βs = 0.05:0.05:2.0) at sweep.jl:42
│ MyPkg.correlator × 500 — edge cases at zero separation untested
│ MyPkg.energy ×10000 — convergence not established below β ≈ 0.1
└ 15 of 17 observable marked definitions were not entered
0.42713…
```@example observing
report = IOBuffer()
value = ExperimentalAPI.@entered report MyModel.sweep(βs)
print(String(take!(report)))
```

The footer counts every observable marked definition **loaded in the process**, not only the ones
in your package.

It returns the value of the expression, so it drops into existing code the way `@time` does — with
one divergence `@time` does not have, since `record` takes a function and the expression therefore
runs inside a closure: a `return` inside it returns from the closure, and `@entered y = f(x)` binds
`y` inside the closure. Write `y = @entered f(x)`. The last line is what makes a clean answer mean
anything:

```julia
julia> ExperimentalAPI.@entered publish(result)
┌ @entered publish(result) at sweep.jl:57
└ entered nothing marked — 17 observable marked definitions were loaded
```@example observing
ExperimentalAPI.@entered report round(1.0; digits = 2)
print(String(take!(report)))
```

"Entered nothing" and "nothing is marked anywhere" are different states, and a package that has
Expand Down
13 changes: 12 additions & 1 deletion examples/walkthrough.jl
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,19 @@ record(() -> Ising.normalise(βs, 0.5); paths=false, timing=false)
#
# [`@entered`](@ref) asks the same question about a single expression, and returns its value, so
# it drops into existing code the way `@time` does.
#
# Written here as `@entered <io> expr` and printed, because Documenter's `@example` renders a
# block's **value** when it has one and its captured output only when the value is `nothing` —
# measured with a control, a plain `println` followed by `99` loses its line the same way. Run
# from a terminal, `ExperimentalAPI.@entered Ising.report(βs)` prints the report itself.

report = IOBuffer()
value = ExperimentalAPI.@entered report Ising.report(βs)
print(String(take!(report)))

#-

value = ExperimentalAPI.@entered Ising.report(βs)
value

# ## How often, and how much of the run
#
Expand Down
Loading