Java: fields become nodes, via a new LanguageConfig.value_types - #3099
Open
ehrlichandreas wants to merge 2 commits into
Open
Java: fields become nodes, via a new LanguageConfig.value_types#3099ehrlichandreas wants to merge 2 commits into
ehrlichandreas wants to merge 2 commits into
Conversation
The Go extractor dispatched on four node types: function_declaration, method_declaration, type_declaration and import_declaration. Go's grammar also has const_declaration and var_declaration, so a package-level constant never became a node, and reading one was not recorded as a relation. Measured on a 59-file Go project, asking for fifteen constants each read in exactly one function: none of the fifteen constants was a node, while all fifteen of the reading functions were. `graphify query <constant>` answered "No matching nodes found" every time. The graph held the answer and offered no way in. Three parts: - const_declaration and var_declaration become nodes, marked with value_kind so later passes can find them, with a `contains` edge from the file. - Reading such a name inside a function body emits a `references` edge with context `value_use`. Without it the node exists but stays unreachable. Restricted to names declared as package-level values, so a local identifier produces nothing. - Names not declared in the file being extracted go to a new raw_value_refs bucket, bound afterwards by _bind_cross_file_value_refs. This mirrors raw_calls: an extractor sees one file and reports what it cannot resolve rather than guessing. Of the fifteen constants above, three were read from a sibling file, and those three stayed unreachable until this pass existed. Bound only for an unambiguous name, the same god-node guard the call resolver uses. raw_value_refs[].caller_nid is rewritten by both id_remap and sym_remap, as raw_calls[].caller_nid already is - left stale the edge would dangle on its source. After the change the same fifteen questions are answered 15/15. The graph grows from 327 to 395 nodes and 797 to 976 edges on that project, with no measurable change in build time (0.76 s both ways). tests/test_go_value_nodes.py covers both directions: a declared constant becomes a node, an unused one gets no reader, a cross-file read is bound, and a local variable neither becomes a node nor binds to one. Four of the six fail without this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same gap as the Go one in the previous commit, in the shared engine rather than a bespoke extractor. The dispatch knew classes, functions, imports and calls. A Java constant is a field_declaration and none of those, so it never became a node and reading one was not recorded as a relation. Measured on a 14-file Java project using five constants each read in exactly one method: none of the five was a node, while all five reading methods were. `graphify query` answered 1/5; after this change, 5/5. LanguageConfig gains value_types and value_kind. Both default to empty, so every other language behaves exactly as before and opts in with one line when it wants to. Only _JAVA_CONFIG sets them here (field_declaration), because that is the one language this was measured on. The engine emits the same two things the Go extractor does: a node per declared name, marked value_kind, and a `references` edge with context `value_use` for a read inside a method body. Names not declared in the file go to raw_value_refs and are bound by _bind_cross_file_value_refs, which is language-agnostic and already in place from the Go commit. The value branch deliberately does not return. A field_declaration also carries its type, which the field-type-reference pass and the Java receiver-type table read from the same subtree; returning there broke eight Java tests that have nothing to do with values. Full suite: 15 pre-existing failures before and after (missing optional tree-sitter grammars), no new ones. tests/test_java_value_nodes.py adds five cases, three of which fail without this change, including one that pins the opt-in: a Python module constant must stay out of the graph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #3098 - please merge that one first. This branch contains its
commit; the diff below is only the Java part.
Problem
Same gap as #3098, one layer up. The shared engine dispatches on
classes, functions, imports and calls:
A Java constant is a
field_declarationand none of those. It neverbecomes a node, and reading one is not recorded as a relation.
Measured on a 14-file Java project using five constants each read in
exactly one method: none of the five was a node, while all five
reading methods were.
graphify queryanswered 1/5.Change
LanguageConfiggainsvalue_typesandvalue_kind. Both default toempty, so every other language behaves exactly as before and opts in
with one line when it wants to. Only
_JAVA_CONFIGsets them here, tofield_declaration, because Java is the language this was measured on.The engine then emits what the Go extractor already does after #3098: a
node per declared name marked with
value_kind, and areferencesedgewith context
value_usefor a read inside a method body. Names notdeclared in the file being extracted go to
raw_value_refsand arebound by
_bind_cross_file_value_refs, which is language-agnostic andcomes from #3098.
One detail worth flagging for review: the value branch deliberately does
not return. A
field_declarationalso carries its type, which thefield-type-reference pass and the Java receiver-type table read from the
same subtree. Returning there broke eight Java tests that have nothing
to do with values, and that is how I found it.
Result
Go is unchanged at 15/15 and TypeScript at 5/10 (its own separate issue:
9 of its 10 names are already nodes, so the loss there is in traversal,
not extraction).
Tests
tests/test_java_value_nodes.py, five cases. Three fail without thischange. One of them pins the opt-in from the other side: a Python module
constant must stay out of the graph, because Python does not set
value_types.Full suite: 15 pre-existing failures before and after (missing optional
tree-sitter grammars), no new ones.
ruff checkclean on the touchedfiles.
Scope
Java only, on purpose. Of the 30 extractors, three mention a constant
node type at all, so the same gap likely exists in most of them - Rust
has
const_itemandstatic_itemand neither is handled. Now that themechanism is config-driven, each language is one line plus its own
measurement, which I would rather do per language than in one sweep.