Repository navigation
index: publish each language's graph as soon as it is solved (#1555) - #1618
Merged
Merged
Conversation
A repository in several languages was solved one language after another, and nothing was published until the last one finished. The main language's graph sat finished while another language's rules compiled, and every query in between waited on the build lock. - bin/axiomcode `all --progress <file>` appends "<lang> ok <graph>" or "<lang> failed" as each language finishes; without --language the largest language is solved first. - axiomcode-build runs the engine in the background (own process group, stopped with the build), solves the main language first and the rest by file count, publishes the main graph (index, stamp, file table, baseline, pointer) the moment it is solved, and moves every other graph into .axiomcode/lang as it arrives. .axiomcode/out/building lists the languages still to come. An engine without --progress is published at the end as before. - ax_fresh: status and query read the pending list; an answer from a current main graph is given at once and carries one line naming the languages it cannot see yet, or whose code comes from their previous graph. build-seconds records the time to the main graph. - ax_contract.ensure_graph: a first query waits for graph.sqlite, not for the whole build; the build's output goes to a log copied to stderr, so stdout carries only the answer. - tests/publish_order.py: C# main with Python and Java beside it, the others held back by a wrapped engine; one-language control.
swapnilpaliwal-sd
requested review from
JaredHLZhang,
Whua689 and
suyashpaliwal26
as code owners
September 28, 2026 08:15
This was referenced Sep 28, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1555
What was wrong
A repository in several languages is solved one language after another, and nearly all of each step is that language's Souffle compile.
axiomcode-buildpublished nothing until the whole engine run ended, so the main language's graph sat finished while another language compiled, and every query made meanwhile waited on the build lock. The issue measured a C# graph written at 131 s and first queryable at 5,304 s.The change
bin/axiomcode all --progress <file>appends<lang> ok <graph>or<lang> failedas each language finishes. Without--language, the largest language (by IR size) is solved first.axiomcode-buildsolves the main language first and the others by file count. It runs the engine in the background in its own process group, which is stopped along with the build. The main graph is published the moment it is solved (index, stamp, file table, baseline, pointer). Every other language's graph moves into.axiomcode/langas it arrives..axiomcode/out/buildinglists the languages still to come. An engine without--progressis published at the end, as before. If the main language fails, the build still fails and the previous graph stays.ax_fresh:statusandqueryread the pending list. An answer from a current main graph comes back at once, with onegraph refresh:line naming the languages it cannot see yet, or the ones whose code comes from their previous graph.build-secondsrecords the time to the main graph. On a clean tree with pending languages, the stale-while-revalidate query path now carries the note too.ax_contract.ensure_graph: a first query waits untilgraph.sqliteexists, not until the whole build has finished. The build writes to a log that is copied to stderr, so stdout carries only the answer.Tests
tests/publish_order.py, 19 checks, all passing. The repository is C# with Python and Java beside it. A wrapped engine passes on the main language as soon as it is solved and holds back the rest. The checks cover: publish order, a query answered while the others are held (with the note), status, a Java name refused at once, the finished build, a rebuild, a failing main language, a first query with no graph, and a one-language control with no pending list and no note.graph.sqlitenever appears while the other languages are held (the build is still running, with no graph), and the solve order iscsharp java python.tests/run.py: csharp 14/14, python 71/71, java 119/119.latency.py --no-engine18/18,surfaces.py8/8 verbs,directive.py12/12.Smoke (real copies, fresh index)
graph.sqliteappeared only at 291 s, after a JavaScript rule compile that then failed. After, the graph appeared at 25 s andimpactreturned 30 rows at 69 s (44 s of that was a one-time query-rule compile) withgraph refresh: the javascript graph is still being built, while the JavaScript compile was still running.impactanswers except the sampled list of untyped-receiver callers. That sample differs between any two builds (pre-existing).Deep scan deferred: the held-out corpus measurement was not run for this change.
This supersedes the local work branches
fix/1555-index-per-language,fix/1555-index-per-language-availability,fix/1555-lang-availabilityandfix/1555-per-lang-graphs(never pushed); it continuesfix/1555-per-language-graph.