Fix LSP workspace discovery, graph invalidation, and document close

docs/architecture/infrastructure.md

@@ -9,7 +9,7 @@ This map records infrastructure implementation observed in `internal/module`, `i
 | `module` | Per-source artifact container, phase reset, and published semantic/IR handoff state. |
 | `project` | Compilation context, module registry, paths, imports, snapshots, fingerprints, semantic lookup, metrics. |
 | `pipeline` | Concurrent module loading, import graph construction, phase barriers, phase advancement, invalidation. |
-| `driver` | Public compile entrypoints and source/overlay selection. |
+| `driver` | Public compile entrypoints and disk/source override selection. |
 | `diagnostics` | Phase/module-scoped diagnostic storage, codes, labels, source cache, rendering, suggestions, highlighting. |
 | `graph` | Directed adjacency, reverse edges, topology algorithms, FIFO worklists. |
 | `problems` | Reusable construction of common semantic diagnostics. |
@@ -144,9 +144,9 @@ can survive a parse-only reset; later pipeline phases rebuild dependent artifact
 ## Inputs, outputs, and loading
 
 `driver.CompileFile` is the ordinary entrypoint. It resolves an absolute input path,
-reads disk content when `overlay == nil`, or uses the overlay string otherwise. It
-creates or obtains the module, marks it as entry, runs `pipeline.Run`, and returns
-the module. Read/path/pipeline failures are added to a load-scoped diagnostic bag.
+reads disk content when `sourceOverride == nil`, or uses the source override string
+otherwise. It creates or obtains the module, marks it as entry, runs `pipeline.Run`,
+and returns the module. Read/path/pipeline failures are added to a load-scoped diagnostic bag.
 `driver.AddSource` registers virtual content without running the pipeline.
 
 `project.DiscoverSourceFiles` expands explicit files and directories, accepts the
@@ -197,7 +197,7 @@ The pipeline then runs these project jobs:
 
 | Job | Input | Output / barrier |
 | --- | --- | --- |
-| Load | entry, prelude, file/overlay content | registered modules, ASTs, imports, graph, `Load`/`Parsed` diagnostics. |
+| Load | entry, prelude, disk/source override content | registered modules, ASTs, imports, graph, `Load`/`Parsed` diagnostics. |
 | Analysis-through | ordered modules and import readiness | phases through `Analyzed`; symbols, THIR, CFG, durable analysis facts and diagnostics. |
 | Usage | analyzed modules | usage diagnostics and `Usage` module/project barrier. |
 | Entrypoint check | entry module scope and function type | optional `ErrInvalidEntrypoint`. |
@@ -346,11 +346,11 @@ both locations. It does not store or emit diagnostics itself beyond the supplied
 back to bounded file scanning.
 
 `diagnostics.SourceCache` stores split source lines under an RW mutex. The bag accepts
-in-memory source through `AddSourceContent`, allowing diagnostics for overlays. The
-emitter supports ANSI/HTML strings and stderr output, source snippets, labels,
-code hints, tab expansion, and optional syntax highlighting. `SyntaxHighlighter`
-performs lightweight line tokenization; `suggest.go` supplies Levenshtein-based,
-ambiguity-aware nearest-name suggestions with stable priorities.
+in-memory source through `AddSourceContent`, allowing diagnostics for source
+overrides. The emitter supports ANSI/HTML strings and stderr output, source
+snippets, labels, code hints, tab expansion, and optional syntax highlighting.
+`SyntaxHighlighter` performs lightweight line tokenization; `suggest.go` supplies
+Levenshtein-based, ambiguity-aware nearest-name suggestions with stable priorities.
 
 ## Graphs and worklists
 

docs/architecture/lsp.md

@@ -8,8 +8,8 @@ Key implementation files:
 
 - `server.go` owns the session loop, method dispatch, diagnostic publication, and
   file URI/path conversion.
-- `state.go` owns mutable server state, document overlays, compiler snapshots,
-  diagnostic generations, debounce workers, and module reuse.
+- `state.go` owns mutable server state, document source overrides, compiler
+  snapshots, diagnostic generations, debounce workers, and module reuse.
 - `types.go` defines the LSP request, response, notification, position, range,
   document, diagnostic, hover, completion, rename, and workspace-edit payloads.
 - `jsonrpc.go` frames JSON-RPC messages and serializes writes.
@@ -42,8 +42,8 @@ The supported lifecycle and document methods are:
   and publishes component diagnostics immediately.
 - `textDocument/didChange`: converts the URI, takes the first full-text change,
   stores its version, and schedules diagnostics after the debounce delay.
-- `textDocument/didClose`: removes the overlay and version, then publishes the
-  resulting component diagnostics.
+- `textDocument/didClose`: decodes `textDocument.uri`, removes the source override
+  and version, then publishes the resulting component diagnostics.
 - `textDocument/hover`, `definition`, `completion`, and `rename`: invoke the
   corresponding `ServerState` handler and return its result.
 - `shutdown`: returns a successful null result.
@@ -75,17 +75,18 @@ completion requests.
 
 `ServerState` contains:
 
-- `RootDir`, `Cache`, `LastCtx`, and `LastMetrics` for workspace configuration,
-  open-document text, the latest compiler context, and metrics.
+- `RootDir`, `SourceOverrides`, `LastCtx`, and `LastMetrics` for workspace
+  configuration, open-document text, the latest compiler context, and metrics.
 - `modules` for retained compiled modules used by incremental recompilation.
 - `workspace` for the source/import index.
 - `documentVersions` for LSP document versions included in diagnostic
   notifications.
-- `diagGeneration` for invalidating diagnostic snapshots after any overlay
-  mutation.
+- `diagGeneration` for invalidating diagnostic snapshots after any source
+  override mutation.
 - `diagVersion` for per-file debounce cancellation.
 - `diagWG` and `diagErr` for waiting on workers and retaining their first error.
-- `mu` for state, cache, compiler snapshot, workspace, and debounce metadata.
+- `mu` for state, source overrides, compiler snapshot, workspace, and debounce
+  metadata.
 - `publishMu` for the publication/mutation boundary.
 
 `NewServerState` initializes the maps. Most state reads and writes use `mu`.
@@ -102,17 +103,17 @@ The request loop itself is sequential, but diagnostic refreshes run in goroutine
 The protocol writer remains safe when those workers publish concurrently with the
 request loop.
 
-## Document overlays and generations
+## Document source overrides and generations
 
 Peeper uses full synchronization. A document snapshot is the complete text from
 `didOpen` or the first `didChange` content item. `applyDocumentSnapshot` canonicalizes
-the path, stores text in `Cache`, and stores the supplied LSP version. Closing the
-document deletes both entries, so later reads fall back to disk.
+the path, stores text in `SourceOverrides`, and stores the supplied LSP version.
+Closing the document deletes both entries, so later reads fall back to disk.
 
-Every open, change, or close increments `diagGeneration`, including removal of an
-overlay. `diagnosticSnapshotLocked` recompiles while state is locked, copies the
-selected file list, captures current document versions, and records the current
-generation in `diagnosticSnapshot`.
+Every open, change, or close increments `diagGeneration`, including removal of a
+source override. `diagnosticSnapshotLocked` recompiles while state is locked,
+copies the selected file list, captures current document versions, and records
+the current generation in `diagnosticSnapshot`.
 
 `currentCompiledModule` reuses `LastCtx` only when `lastCtxGeneration` equals the
 current `diagGeneration`, the module is parsed, and its content hash matches
@@ -151,8 +152,8 @@ forms become invalid-parameter errors for request handlers or are ignored by
 notifications.
 
 `pathToURI` normalizes backslashes and path cleaning, preserves UNC authorities,
-and emits a `file:` URI. Internal cache, workspace, module, and version keys use
-`project.CanonicalPath`; URI conversion is kept at the protocol boundary.
+and emits a `file:` URI. Internal source override, workspace, module, and version
+keys use `project.CanonicalPath`; URI conversion is kept at the protocol boundary.
 
 LSP positions are zero-based and use UTF-16 code units. `offsetAtPosition` walks a
 line and counts one or two UTF-16 units per decoded rune. `positionAtOffset` does
@@ -164,12 +165,25 @@ zero range.
 
 ## Workspace index and incremental compilation
 
-`workspaceIndex.rebuild` discovers source files under `RootDir` and adds cached
-source files within that root. Cached content wins over disk content in
+`workspaceIndex.rebuild` discovers source files under `RootDir` and adds source
+overrides within that root. Source overrides win over disk content in
 `workspaceContent`. Each indexed `workspaceModule` records canonical path,
 project/import context, content hash, import/export fingerprints, and resolved
 local import targets.
 
+Each rebuild uses `workspaceFiles` and canonical `project.DiscoverSourceFiles`
+to enumerate disk sources, including files added inside previously empty nested
+directories. Unchanged files still reuse their content/parse evidence; directory
+membership is not inferred from stamps of directories already containing sources.
+With no source overrides, the canonical sorted, deduplicated result is returned
+directly.
+
+Filtered file contexts also supply membership checks. Import lookup contexts are
+created on demand per project and reused only within that rebuild.
+Project lookup maps retain both loaded projects and checked negative results.
+Membership comparison includes counts and keys before content reads; a same-count
+replacement invalidates graph/components even if its content cannot be read.
+
 Changed or new files are lexed and parsed only for index construction. The index
 builds a directed import graph and groups modules into weakly connected
 `workspaceComponent`s. A component stores sorted files and roots. Roots are
@@ -208,11 +222,11 @@ If source-project resolution fails, it records a load diagnostic and retains the
 context.
 
 For a configured root, it rebuilds the workspace, computes dirty files, seeds
-reusable modules, adds cached sources, and tries the synthetic component entry.
+reusable modules, adds source overrides, and tries the synthetic component entry.
 If that entry produces the requested module, the context is retained and returned.
-Otherwise cached sources are added and the requested file is compiled with its
-current overlay, if present. Reusable diagnostics are activated, then the context
-and modules are retained.
+Otherwise source overrides are added and the requested file is compiled with its
+current source override, if present. Reusable diagnostics are activated, then the
+context and modules are retained.
 
 `retainCompiledContext` updates `LastCtx`, `LastMetrics`, and
 `lastCtxGeneration`. `diagnosticSnapshotLocked` selects a component's files when
@@ -255,10 +269,10 @@ rather than maintaining independent AST parent maps for the same operation.
 
 ## Hover and rendering
 
-`HandleHover` reads current overlay/disk text, converts the LSP position, resolves
-a subject from the current compiled module, renders it, and returns Markdown plus
-its source range. `resolveHoverSubject` checks attributes, imports, types,
-declarations, selectors, symbols, and expressions in that order.
+`HandleHover` reads current source override or disk text, converts the LSP
+position, resolves a subject from the current compiled module, renders it, and
+returns Markdown plus its source range. `resolveHoverSubject` checks attributes,
+imports, types, declarations, selectors, symbols, and expressions in that order.
 
 The normalized `hoverSubject` can represent a symbol, expression type, resolved
 type, declaration, import, or attribute. Type syntax is resolved through
@@ -299,8 +313,8 @@ LSP positions.
   after the cursor, deduplicates names, and sorts results.
 - Qualified completion returns enum variants or public symbols from an imported
   module.
-- Operation completion builds a sentinel source, compiles it with other cached
-  overlays, resolves the base type, and offers fields, interface methods, receiver
+- Operation completion builds a sentinel source, compiles it with other source
+  overrides, resolves the base type, and offers fields, interface methods, receiver
   methods, intrinsics, adaptable local operations, and public imported operations.
   It creates edits for method-call or pipe syntax and can preserve existing
   arguments.
@@ -317,7 +331,7 @@ ordering by sort text, label, and item kind.
 - Compiler contexts are never published as diagnostics after their generation is
   invalidated.
 - Query handlers do not reuse a compiled module whose generation or content hash
-  no longer matches the current document overlay.
+  no longer matches the current document source override.
 - Full-sync changes replace complete document snapshots; the server does not apply
   text ranges.
 - Workspace reuse is constrained by content and import/export fingerprints; a

internal/driver/compiler.go

@@ -21,9 +21,9 @@ func NewCompilerContext(cfg project.Config, diag *diagnostics.DiagnosticBag) *pr
 	return ctx
 }
 
-// CompileFile compiles the entry file from overlay when non-nil, otherwise it
-// reads the source from disk.
-func CompileFile(ctx *project.CompilerContext, path string, overlay *string) *module.Module {
+// CompileFile compiles the entry file from sourceText when non-nil.
+// Otherwise it reads the source from disk.
+func CompileFile(ctx *project.CompilerContext, path string, sourceText *string) *module.Module {
 	if ctx == nil {
 		return nil
 	}
@@ -39,15 +39,15 @@ func CompileFile(ctx *project.CompilerContext, path string, overlay *string) *mo
 		return nil
 	}
 	content := ""
-	if overlay == nil {
+	if sourceText == nil {
 		data, err := os.ReadFile(absPath)
 		if err != nil {
 			loadDiag.Add(diagnostics.NewError("read input file: " + err.Error()))
 			return nil
 		}
 		content = string(data)
 	} else {
-		content = *overlay
+		content = *sourceText
 	}
 	if module, ok := prelude.ModuleForFile(ctx, absPath, content); ok {
 		module.IsEntry = true
@@ -69,17 +69,17 @@ func CompileFile(ctx *project.CompilerContext, path string, overlay *string) *mo
 }
 
 // AddSource registers a virtual/in-memory module in the compiler context.
-func AddSource(ctx *project.CompilerContext, path string, content string) {
+func AddSource(ctx *project.CompilerContext, path string, sourceText string) {
 	if ctx == nil {
 		return
 	}
 	absPath, err := filepath.Abs(path)
 	if err != nil {
 		return
 	}
-	if module, ok := prelude.ModuleForFile(ctx, absPath, content); ok {
+	if module, ok := prelude.ModuleForFile(ctx, absPath, sourceText); ok {
 		ctx.AddModule(module)
 		return
 	}
-	ctx.AddModule(ctx.NewModuleForFile(absPath, content))
+	ctx.AddModule(ctx.NewModuleForFile(absPath, sourceText))
 }

internal/driver/compiler_test.go

@@ -19,20 +19,20 @@ func TestCompileFileSourceSelection(t *testing.T) {
 		t.Fatalf("write source: %v", err)
 	}
 	empty := ""
-	nonempty := "fn overlay() -> i32 { return 2; }\n"
+	nonempty := "fn sourceText() -> i32 { return 2; }\n"
 	tests := []struct {
-		name    string
-		overlay *string
-		want    string
+		name       string
+		sourceText *string
+		want       string
 	}{
 		{name: "disk", want: disk},
-		{name: "empty overlay", overlay: &empty, want: ""},
-		{name: "nonempty overlay", overlay: &nonempty, want: nonempty},
+		{name: "empty source override", sourceText: &empty, want: ""},
+		{name: "nonempty source override", sourceText: &nonempty, want: nonempty},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			ctx := project.NewWithConfig(project.Config{RootDir: root, Extension: peeper.SourceExt}, diagnostics.NewDiagnosticBag())
-			mod := CompileFile(ctx, path, tt.overlay)
+			mod := CompileFile(ctx, path, tt.sourceText)
 			if mod == nil {
 				t.Fatalf("CompileFile returned nil")
 			}

internal/lsp/benchmark_test.go

@@ -24,63 +24,64 @@ func BenchmarkIncrementalWorkspace(b *testing.B) {
 		fixture := createBenchFixture(b, fixtureName)
 		b.Run(fixtureName, func(b *testing.B) {
 			runBenchCase(b, "cold_compile", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				return fixture.entry
 			})
 			runBenchCase(b, "warm_no_change_open", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				sourceText := fixture.entryImport + fixture.entryBody
+				state.applyDocumentSnapshot(fixture.entry, &sourceText, nil)
 				_, _ = state.recompile(fixture.entry)
 				return fixture.entry
 			})
 			runBenchCase(b, "function_body_edit", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				updated := "const leafLimit: i32 = 1;\nfn LeafValue() -> i32 { return 2; }\nfn StableValue() -> i32 { return 7; }\n"
 				state.applyDocumentSnapshot(fixture.leaf, &updated, nil)
 				return fixture.entry
 			})
 			runBenchCase(b, "earlier_function_growth", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				updated := "const leafLimit: i32 = 1;\nfn LeafValue() -> i32 {\n\tlet value = leafLimit;\n\treturn value;\n}\nfn StableValue() -> i32 { return 7; }\n"
 				state.applyDocumentSnapshot(fixture.leaf, &updated, nil)
 				return fixture.entry
 			})
 			runBenchCase(b, "private_constant_edit", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				updated := "const leafLimit: i32 = 2;\nfn LeafValue() -> i32 { return leafLimit; }\nfn StableValue() -> i32 { return 7; }\n"
 				state.applyDocumentSnapshot(fixture.leaf, &updated, nil)
 				return fixture.entry
 			})
 			runBenchCase(b, "export_shape_edit", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				updated := "const leafLimit: i32 = 1;\nfn LeafValue() -> i32 { return leafLimit; }\nfn StableValue() -> i32 { return 7; }\nfn AddedValue() -> i32 { return 9; }\n"
 				state.applyDocumentSnapshot(fixture.leaf, &updated, nil)
 				return fixture.entry
 			})
 			runBenchCase(b, "import_set_edit", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				updated := fixture.entryImport + "import \"bench/extra\";\n" + fixture.entryBody
 				state.applyDocumentSnapshot(fixture.entry, &updated, nil)
 				return fixture.entry
 			})
 			runBenchCase(b, "unrelated_component_edit", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				_, _ = state.recompile(fixture.entry)
 				_, _ = state.recompile(fixture.unrelated)
 				updated := "fn main() -> i32 { return 2; }\n"
 				state.applyDocumentSnapshot(fixture.unrelated, &updated, nil)
 				return fixture.unrelated
 			})
 			runBenchCase(b, "multi_main_first_root", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				return fixture.entry
 			})
 			runBenchCase(b, "multi_main_second_root", fixture, func(state *ServerState) string {
-				state.Cache = map[string]string{}
+				state.SourceOverrides = map[string]string{}
 				return fixture.unrelated
 			})
 		})