Repository navigation
plugin: compile the query rules once per machine, never while a query waits (#1606) - #1634
Merged
Merged
Conversation
swapnilpaliwal-sd
requested review from
JaredHLZhang,
Whua689 and
suyashpaliwal26
as code owners
September 28, 2026 09:33
… waits (#1606) The compiled query programs (impact.dl, path*.dl) were cached inside the plugin's own dl/.cache, so every plugin copy recompiled identical rules, and the build compiled them only after publishing the graph, so the first query raced the compile and paid it (43-79 s). - dl_program: cache in ${XDG_CACHE_HOME:-~/.cache}/axiomcode/queries, keyed by the rules and the platform; an older dl/.cache binary is still used. A query with no binary answers from the Souffle interpreter and starts one detached background compile under a lock. - axiomcode-build starts that compile when the build starts, beside the engine, and only reports its state after publishing. - impact: the untyped-receiver caller sample is ordered by file and line, not by symbol id, so two builds of one tree show the same sample. - tests/query_rules.py and a python case with a control.
This was referenced Sep 28, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1606
What was wrong
dl/impact.dl,dl/path*.dl) were cached inside the plugin, indl/.cache/. Every plugin copy recompiled byte-identical rules: a plugin update installs a new versioned directory, and a second checkout or a worktree is another copy. On this machineimpact.dlalone takes 43-56 s to compile.axiomcode-buildcompiled them at the very end, after the graph was published. Since index: publish each language's graph as soon as it is solved (#1555) #1618 publishes the main graph as soon as it is solved, the first query arrives while that compile has not started, compiles the rules itself, and waits. Measured: 68-79 s for the firstimpacton a fresh index.impact's sampled list of callers on an untyped receiver was ordered by symbol id. Ids are not stable between two builds of the same tree, so the sample changed from build to build.The change
dl_program.py: compiled programs go to${XDG_CACHE_HOME:-~/.cache}/axiomcode/queries/, beside the engine's Soufflé cache.AXIOMCODE_QUERY_CACHEoverrides the location, and the plugin directory is the fallback when there is no writable HOME. The key is the rules hash, as before (CI's--print-idis unchanged), plus the platform, because the binary is built with-march=native. A binary an older plugin left indl/.cache/is still used.program()answers from the Soufflé interpreter, which runs the same rules and gives the same answer, and it starts one detached background compile. The compile runs in its own session, under a per-program lock in the cache, so a hook'stimeout=14cannot kill it half-way. A failed compile is recorded once, as before (.nocompile). A binary shipped in the engine package still wins over everything else.axiomcode-buildstarts that compile when the build starts (dl_program.py --background), in parallel with the engine. After publishing, it only reports the state (--no-wait, e.g.impact.dl=compiling) and never blocks. This builds directly on index: publish each language's graph as soon as it is solved (#1555) #1618's publish path: index: publish each language's graph as soon as it is solved (#1555) #1618 made the graph available early, and this makes the rules available by then, or makes their absence cost nothing. The facts export (impact --warm) after publish is unchanged.axiomcode-impact: the untyped-receiver sample (text and--jsonbyname_callers) is ordered by file, line, then name.reference/impact.md(both copies) describes the new cache and the interpreter fallback.Tests
tests/query_rules.py(13 checks, no engine, needs soufflé and c++). Checks: a first query answers from the interpreter in 0.1 s and says so; one background compile lands in the user cache and nothing is written in the plugin; the next query runs the binary; a second plugin copy reuses it; control: rules one line apart do not; a legacydl/.cachebinary is used; three simultaneous queries leave exactly one binary and no lock, .cpp or .tmp; the interpreter and the binary derive the same rows;warm()reports.python/untyped-receiver-sample-in-source-order: the sample lists the first four callers in source order and leaves out the fifth. Control: every by-name caller is still a row. Before this change the case fails (its sample iszeta, gamma, beta, alpha).tests/run.pyon 0.1.5: python 103/103, csharp 26/26, java 137/138. The one Java miss isbean-registered-or-defined-by-a-class, from java: Spring Boot registrations make beans, and impact names who registers or is injected with them (#1392, #1396, #1414) #1629: the local parser build predates java: Spring Boot registrations make beans, and impact names who registers or is injected with them (#1392, #1396, #1414) #1629's parser change, and CI builds its own. Before the rebase: java 134/134.publish_order.pyall passed,latency.py --no-engine18/18,surfaces.pyok,manifests.pyok.refresh_races.pybuilds a TypeScript fixture, so it was not run locally (JS/TS runs are left to CI).Smoke (real copies, fresh index, cold rules cache, solve cache off)
A 182-file Python web app, a 403-file Java server and a 254-file C# shop, one
impacttarget each. "Before" is origin/0.1.4 with #1618. The dl_program/axiomcode-build code is identical at 0.1.5. Machine load average ~30.First query the moment the graph is published, then the second query:
The remaining 10-13 s on Python and Java is the graph's facts export, which runs right after publish. The rules were already compiled (Python) or ran in the interpreter (Java, C#).
An existing graph with no compiled rules (a plugin update or a new copy):
Controls:
impact --page allon 4 targets across the 3 projects is identical before and after, except the sample line. On every project the interpreter's answer is byte-identical to the compiled binary's.impactof a JSON helper with 7 untyped-receiver callers, on two builds of the same tree gives two different samples before and the same sample after.Not yet measured on the full corpus or on held-out projects: no held-out or regression-gate run for this change.