Repository navigation
Python: library callback summaries -- a library calling back into client code (stacked on #1873) - #1884
Merged
swapnilpaliwal-sd merged 12 commits intoOct 10, 2026
Conversation
pytest fills a test's parameter with whatever the fixture of that name returned or yielded, and no call site spells that call, so the parameter stayed untyped: every method called on it was ambiguous_unknown, and everything derived from it (a local assigned from one of those calls, a with block) went with it. Across ten public suites 1,529 such calls were unresolved; a suite whose fixtures are annotated already resolved its own. value-flow.dl now treats the runner's call as one more argument reaching a parameter: param_arg_type from the fixture's value type (its return, its yield, or its declared return). The fixture serving a parameter comes from a syntax-only copy of the runner's lookup (class, own module, nearest conftest): the existing py_fixture_injection reads module_member_method for star-imported fixtures, which is resolution, and its nearest-conftest MAX would then be a cyclic aggregate. Star-imported and pytest_plugins fixtures still reach their tests through the injection edge; they only stay untyped. Measured with a behavioural oracle (break each of 600 sampled functions, record which test files fail) on ten held-in repositories: impact --tests recall 0.668 -> 0.698, every-failing-file-selected 46.7% -> 50.5%, precision unchanged; the three repositories carrying the pattern move, the seven without it are byte-identical. Engine suite 43/43 with identical case results; query cases 310/310. New case fixture-value-types-the-parameter fails on the base engine on its derived-local check. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…(T, wrapper) returns wrapper functools.update_wrapper(wrapper, wrapped) hands back `wrapper`, and typing.cast(T, x) hands back `x`; both are documented, neither module is staged in a client-only run. A decorator written `return update_wrapper(wrapper, f)`, or typed as `return t.cast(F, update_wrapper(wrapper, f))`, was therefore opaque: every method it decorates was decorator_replaced_target and no call through the attribute reached anything. That shape occurs in five of sixteen public repositories measured. A catalogue, py_returns_arg(DottedName, Position) in builtins.dl, lists the two functions and the argument each returns; call_chain.dl follows such calls (to any depth) to the returned name, matched through the import that binds the callee: a module import, its alias (`import typing as t` is MODULE_IMPORT_ALIAS, which the copy.copy rule beside it also misses), or a from-import. On ten held-in repositories, path's found rate on runtime-proven chains goes 0.720 -> 0.744 (the repository with the shape: 0.52 -> 0.76); test selection and the other verbs unchanged. Engine suite 43/43 with identical case results; query cases 313/313; case decorator-returning-update-wrapper fails on the base engine on both decorator checks. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…y the import walk impact's `at import` rung names every test file importing a module whose BODY reaches the change through calls that run, because a module that raises while being imported fails every file importing it. A module body's calls inside `if __name__ == "__main__":` are real edges, but they run only when the file is executed as a script, never on import. Walking them named every importer of a module that ends in a demo block. The fact export now records guard_only(module, callee) where every call site the module body has to that callee lies inside a top-level main guard, and up_running does not take those edges. Ordinary reachability is unchanged; only the import hop is. Measured with the mutation oracle on ten held-in repositories, the at-import rung was the second-noisiest (precision 0.16 over 720 selected test files), and 637 of those 720 were for targets that never run on import. On the repository carrying the shape its at-import selections drop 713 -> 305 and test-selection precision 0.335 -> 0.373 with recall unchanged; the control repository whose at-import selections were all correct is unchanged. Query cases 315/315 python, 264/264 typescript, 333/333 java; case main-guard-does-not-run-on-import fails on the base engine on the guarded check. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…_"`) is a guard too `if sys.platform != "win32" and __name__ == "__main__":` is false on import whatever the other conjunct is, so the block cannot run then; an `or` could, and is not read as a guard. The first guard rule missed the conjunction, which on the repository carrying the shape was where almost all of the remaining false at-import routes came from. There: at-import selections 305 -> 55, of which 53 fail under the mutation oracle; test- selection precision 0.373 -> 0.446. Recall 0.810 -> 0.794: a few failing files had been selected only through an at-import route that cannot run, right by accident; their real route is one the graph does not see. Control repository unchanged. IMPACT_VERSION 65, so fact caches written under 64 are not reused. Case extended with the conjunctive spelling; query cases green. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…hat builds the object, not from every constructor impact takes a hop from a protocol member (a dunder no call site names) to whoever constructs its type: you do not build a context manager without entering it. That holds for __enter__/__exit__, __bool__, __hash__, __call__ (precision 0.61-0.80 under the mutation oracle), and not for showing a value (__repr__, __str__, __format__), pickling or copying it (__getstate__, __setstate__, __reduce__, ...) or deleting from it (__delitem__, __delattr__, __del__): those run only when some code asks, and a constructor deep inside other code says nothing about that. Measured on ten held-in repositories, __repr__ alone was 124 selected test files at precision 0.05, across eight of them. For those members the hop is now taken only from test code that constructs the type: a test that builds the object is the one that asks for its repr. Dropping the hop outright removed 403 false files but left 13 targets with a failing test and an empty answer; this form keeps every answer non-empty. Ten held-in repositories (this commit with the two main-guard ones): test-selection precision 0.408 -> 0.454, recall 0.698 -> 0.694, empty answers unchanged at 84/535. Case on-demand-dunder-hop-from-tests fails on the base engine on the __repr__ check; its control (__eq__ keeps every constructor) passes on both. Query cases 318/318 python, 264/264 typescript, 333/333 java. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…setter and deleter The engine emitted a PROPERTY_READ edge for reading a @Property and nothing for writing or deleting one, though `obj.x = v` runs `@x.setter` and `del obj.x` runs `@x.deleter` exactly as a read runs the getter. A setter therefore had no caller: an impact on it named no test, and everything the setter calls was cut off from whoever assigns. Setters occur in five of sixteen public repositories measured; the torture fixture documented the gap as a known miss. type_property_accessor finds the setter / deleter on the class that wins the name in the receiver's MRO (they share the getter's binding, so mro_lookup's single answer is not enough); property_write_edge keys on the attribute access's STORE / DEL context; the edge is exported as call kind PROPERTY_WRITE (schema vocabulary for Python, the kind TypeScript already uses for its accessors), mapped to the `property` rung. Torture: the CPython oracle confirms the new edge (agree 587 -> 588, missing 47 -> 46, extra unchanged); f43's known-miss marker is removed with its docstring rewritten at the same line count; goldens updated. Engine suite 43/43. Query cases 322/322 python, 264/264 typescript, 333/333 java; case property-setter-is-a-call fails on the base engine on all three write/delete checks. All six commits together, ten held-in repositories: test-selection recall 0.668 -> 0.700, precision 0.403 -> 0.452, impact source-caller recall 0.673 -> 0.686, path found 0.720 -> 0.752, context recall@5 0.789 -> 0.799; no repository regresses beyond 0.003 on any of them. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…operand of a 3+ way union counts Two gaps in "Optional[X] IS X", the rule that types a value annotated Optional[X] or X | None: - It existed for parameters, returns and fields but not for a LOCAL variable's annotation, so `ctx: Optional[Context] = ...` and `x: Foo | None = ...` left every call on the local unresolved while the same annotation on a parameter resolved. - `|` is left-associative: `A | B | None` is `(A | B) | None`, so A and B sit two levels below the annotation, under a nested union, and the depth-1 element rule saw only that nested union (which names no type) and None. Every union of three or more operands lost its members, on parameters, returns, fields and locals alike. binding_declared_nominal gains the Optional/union clause; union_operand walks nested PEP 604 unions and feeds their operands to type_ref_element, resolved and by name. Containers are untouched (union_operand is rooted at an Optional/union kind only). Optional/union locals occur in 15 of 16 public repositories measured (96 written with `|`, 50 with Optional[]/Union[]). Ten held-in repositories, against the previous commit: path found 0.752 -> 0.756, impact source-caller recall 0.686 -> 0.687 (one repository 0.795 -> 0.807); the new callers are a union's dispatch set and are labelled `one of a set`, resolved-caller precision unchanged at 0.979. Engine suite 43/43, torture unchanged; query cases 325/325; case optional-and-union-annotations fails on the base engine on both the local and the three-operand checks. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… a test file imports the conftests above it The import facts named a Python module only by its path from the repository root, so in a src layout `src/app/core.py` was `src.app.core` and no `import app.core` matched it: a src-layout repository had almost no test-to-source imports, which the import walk (a function that runs while a module is imported breaks every file that imports it) and the loads-the-change filter both read. A module is now also named from its package root, the first directory above it that is not a package. pytest imports every conftest.py from the rootdir down to a test file's directory before the file, so what a conftest imports is imported for each test file beneath it; a test file now imports those conftests. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…o it only where the module sits under the dropped segments findModule dropped leading segments of an import it could not match exactly and then matched what was left exactly or by suffix. `from clikit.testing import CliRunner` dropped `clikit` and found the project's own `testing` package (or `app.testing` by suffix); `from fw.db import models` found the project's `models.py`. A dependency's module whose last segments collide with one of the project's resolved INTO the project, the import bound nothing, and every call through it was unresolved instead of a named boundary. The drop exists for an import more qualified than the module (`import myproject.framework` naming a `framework` in a `myproject/` directory with no __init__.py). The analyzer now records the directories above each module's top package, and a dropped prefix must be what the module actually sits under. Python engine goldens: 33-url-view-forms, 37-model-signals and 38-app-wiring move sites from ambiguous_unknown to boundary_lib (`external:path`, `external:models.CharField`, `external:call_command`); edges, framework edges and the torture harness are otherwise unchanged. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… hands back to the objects the client gives it The engine reads a dependency's signatures, never its bodies, so a hop the LIBRARY makes back into client code was invisible: a test client's get() runs the open() a client subclass overrides and then calls the application it was built around; a CLI runner's invoke(cli) runs cli.main(), which runs the command's callback; a decorator wraps a def into a library object that later calls it; ExitStack.enter_context(cm) runs cm.__enter__. On a 16-repository Python oracle a library frame was the first broken hop behind most test-impact misses of the web-framework subject, and every one of them stopped at the library. graph/python/libsum/libsum.py derives the hand-backs from the library's SOURCE (the project's own environment, its stdlib through pyvenv.cfg, or AXIOMCODE_PY_SITE): a flow-insensitive pass per function records which parameter-rooted value meets which call or protocol syntax, composed to a fixpoint across the library's own calls (self-calls, typed parameters and fields, constructors, closures, decorator factories). It writes, for the library classes and functions the client can name, five tables beside the client IR: the hand-backs (callee, root, position, path, member, via, declared type), constructor stores, re-exports, what a decorator turns a def into, and class ancestors. Cached by the installed distributions and the client's imports, so a refresh that adds no import reuses them. Staged through a new client-extra.map (facts beside the IR that are not parser schema); absent = empty = no change. framework-behavior/library-callbacks.dl joins them to the client: a member a client type inherits from its library base, a super() past the client classes, a library function or class the client imported, a member of a library object the client built (followed through locals, helpers' returns and fixtures) or one a library decorator made of a def. Each hand-back resolves to the client member of the object that arrives -- the receiver, an argument, a field the library constructor filled, a field the client set -- and only when none answers, to the client subclasses of the type the library declares there. It ADDS lib_callback_edge (caller, target, callee, member, via, how) and lib_callback_site, and leaves call_chain_edge alone. No library or framework is named in a rule. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
… hop `library callback` impact reads the Python engine's lib_callback_edge as fw_edge "library callback": a test that drives a library now reaches the client member the library hands back to, with the rung `library callback` (ranked with `registered`: a library's behaviour joined by a member name, never a resolved call). lib_callback_site marks a call site the engine knows enters a library callable as a library receiver, so it is never a by-name seed: a test client's `client.put()` no longer reaches the project's own `put` by name. tests/cases/python/library-calls-back: a fake site-packages with a test client, a CLI runner and command decorator, a template environment whose loader arrives through **options and a function calling a member on its argument; each positive check fails on the base engine. Controls: the same member name on a class never handed to a library, an object handed to a library function that never calls that member, a same-named member on a class that is not a subclass of the declared library type. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
…ependent impact names
path adds lib_callback_edge to the hops no call site expresses, worded `library_callback via <callee> -> <member>`
and verified against the engine's table, so a test reaches the application its test client calls; impact lists
the client method that calls the library as a [framework] dependent of the member handed back
("a library it calls hands this back (<callee> calls <member>)"), which is what path's note points at.
Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the Python PR work: this branch sits on the local candidate of #1873 (8 Python impact-loop commits, same fixes) and was measured with #1878's parser patch applied. Review/merge those first; the last 4 commits are this PR.
What was missing
The Python engine reads a dependency's signatures, never its bodies, so when a library calls back into client code the call graph stops at the library frame: a test client's
get()runs theopen()a client subclass overrides and then the application it was built around; a CLI runner'sinvoke(cli)runscli.main(), which runs the command's callback; a library decorator turns adefinto an object that later calls it;ExitStack.enter_context(cm)runscm.__enter__.Measured with a runtime tracer that records the library frames between two client frames, over the 16-repository Python mutation oracle: of 3,397 missed (target, test file) pairs, 979 have a runtime chain whose only hops missing from the graph are library hops.
cli.main()-> override; a def a library decorator wrapped)A pair can need more than one mechanism; 979 of the 3,397 missed pairs are fixable by library hops alone. (One subject's 17 pairs are its own generated template code, not a library.)
Design
graph/python/libsum/libsum.pyderives summaries from the library's source in the project's environment (.venv/venv/$VIRTUAL_ENV, stdlib viapyvenv.cfg, orAXIOMCODE_PY_SITE). A flow-insensitive pass per function records which parameter-rooted value meets which call or protocol syntax; a fixpoint composes them across the library's own calls (self-calls, annotated params and fields, constructors, closures, decorator factories). Output, for the library classes/functions the client can name: hand-backs(callee, root, pos, path, member, via, declared type), constructor stores, re-exports, decorator results, class ancestors. Nothing is solved over library bodies at index time. Cached in~/.cache/axiomcode/libsumby the installed distributions + the client's imports. No environment = no rows = the engine behaves exactly as before. C extensions yield nothing (declared blind spot).templates/client-extra.map(facts beside the IR that are not parser schema), so the staging guard's schema rules are untouched.framework-behavior/library-callbacks.dl(only ADDS relations; no library or framework named in a rule) joins them: a member a client type inherits from its library base,super()past the client classes, a library function/class the client imported, a member of a library object the client built (followed through locals, helper returns and fixtures) or one a library decorator made of adef, and a hand-back to a member the client object inherits from the library (composes). Each hand-back resolves to the client member of the object that actually arrives (receiver, argument, field filled by the library constructor, field the client set); only when none answers, to the client subclasses of the type the library declares there. Emitslib_callback_edgeandlib_callback_site.lib_callback_edgeasfw_edge "library callback"(its own rung, ranked withregistered), lists the caller as a[framework]dependent, and treatslib_callback_siteas a library receiver (no by-name seed: a test client'sput()no longer reaches the project's ownputby name). path walks the hop too.findModuledropped leading import segments and matched the rest exactly or by suffix, so a dependency's module sharing its last segment with a project module (x.testing,x.db.models) resolved INTO the project and its import bound nothing. A dropped prefix must now be the directories the module sits under. This alone moves no test-impact number (verified with a parser-only arm), but it gated the largest gain below.Measured (16-repo Python mutation oracle; base = #1873 candidate + #1878)
library callbackrung precision: 0.26-0.41 (the closure past the application object is the existing graph's over-approximation; "reaching is not failing").Tests
tests/cases/python/library-calls-back(fake site-packages): 8 positive checks fail on the base engine and pass here; 3 controls (same member name never handed over; handed over but never called; same member on a non-subclass of the declared type).python3 tests/run.py --langpython 339/339, java 333/333, typescript 269/269, javascript 307/307, csharp 208/208;tests/fastpath.pyok.Open
obj.attrread runs the wrapped def) is not modelled yet (one subject, 3 pairs).callback_registered); the residual (library base template overrides, object protocols) needs a library-frame runtime trace on the JS corpus to size before building.