Skip to content

Resolve generated definitions to their source annotation - #102

Closed
JesseHerrick wants to merge 1 commit into
mainfrom
generated-definition-locations
Closed

Resolve generated definitions to their source annotation#102
JesseHerrick wants to merge 1 commit into
mainfrom
generated-definition-locations

Conversation

@JesseHerrick

@JesseHerrick JesseHerrick commented Sep 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Problem

Go-to-definition on a callable that exists only in compiled form lands on the owning module's defmodule line. For Spark entity modules it was also the wrong file, because the lexical parent is not where the code was generated:

code_interface do   ->  deps/ash/lib/ash/resource/dsl.ex:5   (defmodule Ash.Resource.Dsl)
define :register    ->  deps/ash/lib/ash/resource/dsl.ex:5

Fix

The compiler already records the answer, in two chunks Dexter was already partly reading:

  • every Docs chunk entry carries a source annotation for its callable. beam.Function now keeps it as Line instead of skipping it.
  • the CInf (compile info) chunk records the file the module was compiled from. beam.ReadSourcePath reads it.

Together they name a real position:

Ash.Resource.Dsl.code_interface/1        ->  deps/ash/lib/ash/resource/dsl.ex:1811
Ash.Resource.Dsl.CodeInterface.Define... ->  deps/spark/lib/spark/dsl/extension.ex:1608

The annotation names the site the code was created at rather than the module it landed in, which is what makes it useful: a symbol created while expanding use X is attributed to the use line, so the answer points at the code that generated it.

Notes

  • No IndexVersion bump: this adds a definition fallback and changes no index schema.
  • Following @before_compile hooks is deliberately not in this PR. It is a different mechanism — a deferred callback rather than a second use — with its own edge cases.

Note

Medium Risk
Changes LSP go-to-definition fallback behavior and path rebasing logic; mistakes could send editors to wrong files/lines, but index authority and conservative line filtering limit blast radius.

Overview
Go-to-definition for callables that exist only in compiled form (e.g. Spark/Ash generated functions) no longer stops at the owning module’s defmodule line—or the wrong file for entity modules.

The BEAM reader now keeps per-callable source lines from Docs chunk annotations on beam.Function.Line (non-integer annos are dropped to line 0 without losing the entry) and adds beam.ReadSourcePath to read the compile-time :source path from the CInf chunk (string, binary, or charlist encodings).

The LSP builds file + line targets via generatedSymbolLocations: index-backed source paths win; otherwise compile info is used and rebased onto the project root using the app segment from the recorded path (so Spark-generated Ash modules can point at Spark sources). Callables annotated at line ≤1 are skipped so the existing module-row fallback remains for @before_compile/transformer output.

Definition handling tries these locations before the prior generated-module fallback in both bare-function resolution paths.

Reviewed by Cursor Bugbot for commit 5c755a7. Bugbot is set up for automated code reviews on this repo. Configure here.

@JesseHerrick
Resolve generated definitions to their source annotation
5c755a7 
A callable that exists only in compiled form had no navigable position, so
go-to-definition fell back to the owning module's defmodule line. For Spark
entity modules that was also the wrong file, because the lexical parent is not
where the code was generated.

The compiler already records the answer. Every Docs chunk entry carries the
source annotation for its callable, and the CInf chunk carries the file the
module was compiled from. Together they name a real position:

  Ash.Resource.Dsl.code_interface/1        -> deps/ash/.../dsl.ex:1811
  Ash.Resource.Dsl.CodeInterface.Define... -> deps/spark/.../extension.ex:1608

The annotation names the site the code was created at, which is why this works:
a symbol created while expanding "use X" is attributed to the "use" line, so the
answer points at the code that generated it rather than at the consumer.

Callables annotated with the module line are left to the existing module-row
fallback. That annotation means a transformer or a before_compile hook produced
the code away from any single position, and a line-1 hit is indistinguishable
from a real one.

The recorded source path is rebased through the app directory before use: it was
recorded wherever the artifact was built, which for a dependency is often a
different checkout.
@JesseHerrick JesseHerrick self-assigned this Sep 21, 2026
JesseHerrick added a commit that referenced this pull request Sep 29, 2026
@JesseHerrick @claude
Combine generated definition lines with compile info from #102
c6563fe 
Drop the stale-BEAM guard. Dexter cannot compile the project, so a BEAM
older than its source is the usual state while editing, and its line is
closer than the module line.

From #102:
- Keep the Docs chunk anno as beam.Function.Line, and use it when a module
  has no debug info. The CInf chunk's :source (beam.ReadSourcePath) says
  which file that line is in.
- Send a generated module with no source row, such as a Spark entity
  module, to the file it was compiled from, rebased onto the project's
  lib or deps when it was built elsewhere.

Move the definition-line code into generated_definition.go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@JesseHerrick JesseHerrick mentioned this pull request Sep 29, 2026
Merged
@JesseHerrick

JesseHerrick commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

Folded into #110 (c6563fe). The Docs anno line and the CInf source reader are kept. The Docs anno is the fallback when a module has no debug info, and CInf sends sourceless generated modules, such as Spark entity modules, to their rebased deps file.

@JesseHerrick JesseHerrick mentioned this pull request Sep 30, 2026
Closed
JesseHerrick added a commit that referenced this pull request Oct 3, 2026
@JesseHerrick
Send generated functions to the line that declared them (#110)
f840dd2 
Closes #108. Supersedes #102, which is folded in here. Go-to-definition
for DSL-generated functions such as Ash code interfaces. It does not
special-case any framework. The Ash side is ash-project/ash#2971, which
is merged and will ship in the next Ash release after v3.33.11. With
released Ash, a code interface goes to its `define` line too, from the
generic declaring-call search described below.

## Summary

A function that a macro generates has no source definition, so
go-to-definition fell back to the top of its module. The compiled
module's `Dbgi` chunk records the location of each def:

- Its `:line` is the line in the module's own file that was compiled
when the def was made. Usually this is the macro call or the line where
a before-compile hook ran, but a generator can expand the def at any
line.
- A `file: {path, line}` entry comes from `@file` or from `quote
location: :keep`. When `path` is the module's own source, this line is
where the code asked for the function. When `path` is another file, it
is the generator's own implementation, and `:line` is used instead.

Released Ash records `line: 1, file:
{"deps/ash/lib/ash/code_interface.ex", N}` for every generated def.
After ash#2971, Ash expands each generated def at its `define` line, so
the debug info is `line: <define line>, file:
{"deps/ash/lib/ash/code_interface.ex", N}`. The foreign-file rule above
picks up `:line`. Ash did not use `@file`, because the function body's
lines still come from `code_interface.ex`, so stacktraces would show the
user's file with Ash's line numbers. Their approach keeps stacktraces
unchanged.

## Changes

- `beam.ReadDefinitionLines` walks `{:debug_info_v1, :elixir_erl,
{:elixir_v1, map, specs}}` and reads only `file`, `relative_file`, and
`definitions`. It steps over clause bodies without allocating. Clause
ASTs can nest very deep, so the ETF reader now steps over a term with a
count of the terms still to skip instead of recursion: any nesting costs
no stack, a count larger than the bytes left is rejected as corrupt, and
the old depth guard is gone. This is also about 1.8x faster on large
Dbgi chunks; Docs parsing speed is unchanged. Modules without Elixir
debug info (`debug_info: false`, Erlang modules, no chunk) return an
error and keep the fallback.
- `generatedDefinitionResultsFor` uses a recorded line only when all of
these conditions are true:
- The debug info was compiled from the file being opened (same absolute
path, or the relative path matches as a suffix, for a project that was
moved or is reached through a symlink).
- The line is after the module's own line. A line at or before it adds
nothing to the module result.
In all other cases, the result stays as before. A BEAM older than the
source still gives its line. Dexter cannot compile the project, so a
stale BEAM is the usual state while you edit, and the line from the last
compile is closer than the module line. Only edits to the declaring file
move it, and the next compile makes it exact. The line is not corrected
for those edits: every way to do that meant guessing at the compiled
text, and could send the editor to a wrong line that looked right.
- From #102: `beam.Function.Line` keeps the Docs chunk anno, and
`beam.ReadSourcePath` reads `:source` from the `CInf` chunk. The Docs
anno is often the same line as the Dbgi `:line`, but not always: Ash's
code interfaces give the docs line 1. It is used when a module has no
debug info, and `CInf` says which file the line is in.
- From #102: a generated module with no source row, such as a Spark
entity module, goes to the file it was compiled from. When that path
does not exist here (a BEAM built elsewhere), it is rebased onto
`lib/<app>`, `deps/<app>/lib/<app>`, or `deps/<app>`. The app name comes
from the recorded path, because Spark builds Ash's entity modules in
Ash's ebin from Spark's source. If no file is found, the lexical parent
stays the answer.
- Bare-call definition and call-hierarchy preparation call it directly.
Qualified definition, the references declaration, and `dexter lookup`
get it through `LookupName` (shared since #107). A recorded line is the
function's own definition, so `LookupName` returns it for a strict
(`ExactModule`) lookup too. Without a recorded line, strict and fallback
lookups do not change.
- The debug info and compile info are read on the first definition
request that needs them. It is memoized on the module's
generated-function cache entry, so the BEAM stamp invalidates it with
everything else.

### Also in this PR

- **Declaring call.** When the only line the BEAM records is the module
line (a `@before_compile` hook, or released Ash), the module's body is
searched for the call whose first argument is the function's name as an
atom. When several calls spell the name, as an Ash action and the code
interface that runs it do (`update :publish` and `define :publish`), the
macro whose calls name the most of the module's generated functions
wins: `define` names every interface, an action names only itself. A tie
keeps the module line.
- **Clauses.** A function with a clause per DSL call (`route :get`,
`route :post`) goes to every clause, from the line Dbgi records for each
clause.
- **Past the end of the file.** A recorded line the current text does
not have (`quote line: 99`) is dropped.
- **Module names.** Go-to-definition on a module that exists only as a
BEAM (`Module.create`) goes to the file and line the compiler recorded.
A module that a macro made with `defmodule unquote(name)` and a name it
computed records no module line, so it goes to its first function's
line.
- **Imports.** A bare call to a generated function of an imported module
resolves.
- **Defs in macro quotes.** The parser no longer indexes a def inside a
`quote` in a macro other than `__using__` as a function of the macro's
module. `defmacro route ... quote do def handle` made the index claim
that the DSL defines `handle/2`, so a consumer that imports the DSL
resolved `Consumer.handle` into the macro's body before the BEAM was
asked. What `__using__` injects and a quote in a helper function are
still indexed.
- **Older Elixir.** Elixir 1.17 and earlier record the module line under
`line`, not `anno`; both are read. A `location: :keep` macro defined
above its caller in the same file is no longer taken for an `@file`
stamp.
- **Worktree moves (fix for #111).** `git worktree move` renames the
directory and then rewrites its `.git` file in place, so a watcher can
read it empty and take the worktree for a plain directory: the fsnotify
backend reported every file in it, and the FSEvents backend ran a full
reconcile. A `.git` file that names no git directory yet now makes the
directory a pending top, which is checked again once git is done. This
was the cause of the flaky
`TestFSNotifyWatcherSkipsWorktreeAddedWhileRunning` on Linux; both
backends now have a deterministic test for it.
- **Compressed BEAMs.** A BEAM compiled with the `compressed` option (a
gzip stream, used by some Erlang dependencies) is decompressed instead
of rejected.
- **Integration tests.**
`TestDefinition_GeneratedFunctionsFromCompiler*` compile a DSL fixture
of each shape with `mix` (with and without debug info, and with a stale
source) and run in the integration job.

## Performance

A cold read on real Ash modules takes 0.8–2 ms (inflate plus walk;
`Repro.Chat`'s Dbgi is 19 KB compressed and 350 KB inflated). It happens
once per BEAM stamp and only on a definition miss for a generated
function. The parser change below bumps `IndexVersion` (and the daemon
contract), so each workspace rebuilds its index once after the upgrade,
through the parallel full build.

## Validation

- `go test ./...` and `make lint` pass.
- New `internal/beam` tests: own-file location, foreign location, plain
`:line`, macros and private defs, stripped or Erlang debug info,
truncation at every seventh byte, a clause body 200,000 ETF levels deep,
and a test that compiles real modules with `elixirc` (skipped if Elixir
is not installed) for `@file`, `location: :keep`, and plain quotes. It
also checks that the Docs anno and the `CInf` source agree with the
debug info.
- New LSP tests: qualified and bare definitions and call hierarchy go to
the recorded line, for both the `@file` shape and the Ash shape (`:line`
with a generator `file`). `LookupName` returns the recorded line with
and without `ExactModule`, and a strict lookup without a recorded line
keeps the module. A stale BEAM goes to the line it recorded, and never
to a line past the end of the file. A foreign location with no useful
`:line`, and debug info from a different source file, keep the module
line. The Docs anno gives the line when there is no Dbgi, and does not
when there is no `CInf` source. A sourceless generated module goes to
its rebased `deps` file from Dbgi or Docs, and to its lexical parent
when that file is missing.
- End to end on the repro app from #108 against Ash `main` (9fa5088,
includes ash#2971), with `lspprobe` for definition and `dexter lookup
--strict` for the CLI. Both give the same lines:

| Probe | main | this branch |
|---|---|---|
| `Chat.get_room_by_slug!` | `chat.ex:1` | `chat.ex:7` |
| `Chat.create_room` | `chat.ex:1` | `chat.ex:8` |
| `Repro.Chat.Room.rename` (resource `code_interface`) | `room.ex:1` |
`room.ex:19` |

End to end on a project with no dependencies and a DSL in the project,
for macros that are not Ash (`main` gives line 1 of `user.ex` for all of
them except `hello`):

| Generator | this branch |
|---|---|
| plain `quote` (`field :email`) | `user.ex:4` |
| `location: :keep` | `user.ex:6` |
| `@file {file, line}` | `user.ex:8` |
| `__using__` injection | `dsl.ex:5` (unchanged, the `def` in the quote)
|
| nested `defmodule` from a macro | `user.ex:10` |
| `Module.create` in the generator's file (no source row) | `dsl.ex:43`
|

The same results hold for a copy of the project that was not recompiled,
for an umbrella copied the same way (`apps/<app>/lib`), and for a
project in a directory named with `é` and `日本`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@JesseHerrick JesseHerrick

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JesseHerrick