Skip to content

index: publish each language's graph as soon as it is solved (#1555) - #1618

Merged
swapnilpaliwal-sd merged 1 commit into
0.1.4from
fix/1555-per-language-graph
Sep 28, 2026
Merged

swapnilpaliwal-sd merged 1 commit into
0.1.4from
fix/1555-per-language-graph

Conversation

@swapnilpaliwal-sd

Copy link
Copy Markdown
Contributor

Fixes #1555

What was wrong

A repository in several languages is solved one language after another, and nearly all of each step is that language's Souffle compile. axiomcode-build published nothing until the whole engine run ended, so the main language's graph sat finished while another language compiled, and every query made meanwhile waited on the build lock. The issue measured a C# graph written at 131 s and first queryable at 5,304 s.

The change

  • bin/axiomcode all --progress <file> appends <lang> ok <graph> or <lang> failed as each language finishes. Without --language, the largest language (by IR size) is solved first.
  • axiomcode-build solves the main language first and the others by file count. It runs the engine in the background in its own process group, which is stopped along with the build. The main graph is published the moment it is solved (index, stamp, file table, baseline, pointer). Every other language's graph moves into .axiomcode/lang as it arrives. .axiomcode/out/building lists the languages still to come. An engine without --progress is published at the end, as before. If the main language fails, the build still fails and the previous graph stays.
  • ax_fresh: status and query read the pending list. An answer from a current main graph comes back at once, with one graph refresh: line naming the languages it cannot see yet, or the ones whose code comes from their previous graph. build-seconds records the time to the main graph. On a clean tree with pending languages, the stale-while-revalidate query path now carries the note too.
  • ax_contract.ensure_graph: a first query waits until graph.sqlite exists, not until the whole build has finished. The build writes to a log that is copied to stderr, so stdout carries only the answer.

Tests

  • New tests/publish_order.py, 19 checks, all passing. The repository is C# with Python and Java beside it. A wrapped engine passes on the main language as soon as it is solved and holds back the rest. The checks cover: publish order, a query answered while the others are held (with the note), status, a Java name refused at once, the finished build, a rebuild, a failing main language, a first query with no graph, and a one-language control with no pending list and no note.
  • The same test against origin/0.1.4 fails its first two checks. graph.sqlite never appears while the other languages are held (the build is still running, with no graph), and the solve order is csharp java python.
  • tests/run.py: csharp 14/14, python 71/71, java 119/119. latency.py --no-engine 18/18, surfaces.py 8/8 verbs, directive.py 12/12.
  • No JavaScript or TypeScript suite was run locally; CI covers them.

Smoke (real copies, fresh index)

  • A 461-file C# library with 4 JavaScript files. Before, with the installed build, the C# solve finished at 11 s, but graph.sqlite appeared only at 291 s, after a JavaScript rule compile that then failed. After, the graph appeared at 25 s and impact returned 30 rows at 69 s (44 s of that was a one-time query-rule compile) with graph refresh: the javascript graph is still being built, while the JavaScript compile was still running.
  • Control, a 154-file C# library in one language: graph at 14 s before and after, no note, and identical impact answers except the sampled list of untyped-receiver callers. That sample differs between any two builds (pre-existing).

Deep scan deferred: the held-out corpus measurement was not run for this change.

This supersedes the local work branches fix/1555-index-per-language, fix/1555-index-per-language-availability, fix/1555-lang-availability and fix/1555-per-lang-graphs (never pushed); it continues fix/1555-per-language-graph.

A repository in several languages was solved one language after another,
and nothing was published until the last one finished. The main
language's graph sat finished while another language's rules compiled,
and every query in between waited on the build lock.

- bin/axiomcode `all --progress <file>` appends "<lang> ok <graph>" or
  "<lang> failed" as each language finishes; without --language the
  largest language is solved first.
- axiomcode-build runs the engine in the background (own process group,
  stopped with the build), solves the main language first and the rest by
  file count, publishes the main graph (index, stamp, file table,
  baseline, pointer) the moment it is solved, and moves every other
  graph into .axiomcode/lang as it arrives. .axiomcode/out/building
  lists the languages still to come. An engine without --progress is
  published at the end as before.
- ax_fresh: status and query read the pending list; an answer from a
  current main graph is given at once and carries one line naming the
  languages it cannot see yet, or whose code comes from their previous
  graph. build-seconds records the time to the main graph.
- ax_contract.ensure_graph: a first query waits for graph.sqlite, not for
  the whole build; the build's output goes to a log copied to stderr, so
  stdout carries only the answer.
- tests/publish_order.py: C# main with Python and Java beside it, the
  others held back by a wrapped engine; one-language control.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working csharp C# java Java python Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant