Parser event ir by tompng · Pull Request #1768 · ruby/rdoc

tompng · 2026-08-08T19:19:22Z

Currently, RDoc's module/class resolve is parse-order dependant, and may fail.

# file1.rb
require 'file2'
require 'file3'
module Foo
  class Bar::Baz < Blah
  end
end

# file2.rb
class Bar; end

# file3.rb
module Foo
  class Blah; end
end

Parse steps

Parse all ruby file, convert to an intermediate representaiton
Resolve constant/class/module namespace
Create CodeObject

Future possibilities

Server mode can be more stable

The Prism visitor now emits an intermediate representation: a flat stream of plain-data records (scope open/enter/exit events, methods, constants, comments, directives, ...) instead of mutating the store directly. A new CodeObjectBuilder replays the records in emission order and contains all CodeObject creation and name resolution logic, unchanged. Lexical state (module nesting syntax, visibility cursor, comment consumption, token extraction) stays on the emitter side; everything that reads or writes the store moves to the builder. Scopes are identified structurally by ids so that the builder reproduces the exact open-time container identity of the previous single-pass implementation, including scopes that resolve to no documentable container, whose inner records are dropped. This is a behavior-preserving refactoring step toward two-phase name resolution: a later change can collect the IR of all files first and resolve names against the complete declaration table instead of the parse-order store state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Batch documentation runs now parse every file into IR first and replay the IR of Ruby files afterwards, in the original file order. This is a step toward resolving names against the declarations of all files instead of the store state at parse time. Output for Ruby-only code bases is unchanged: the replay performs the same store mutations in the same order, only later in time. In a code base mixing C and Ruby sources, mutations of the C parser now happen before all Ruby mutations instead of interleaved in file order. A single #scan call still parses and builds immediately; the server's per-file reparse path keeps that behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A new NamespaceResolver runs before CodeObjects are built: it simulates the lexical scopes of the IR of every Ruby file in the batch and computes the declaration table (namespaces, constants and module aliases) with a fixed-point iteration, because resolving one declaration's owner can depend on the names another declaration introduces. Each IR record is annotated with its resolved full names: the declared class or module, the superclass, def receivers, constant owners, include/extend targets and constant alias targets. CodeObjectBuilder consumes the annotations and contains no name resolution of its own. Namespaces that no file has built yet are created ahead of the build as ignored ghosts; a ghost revives through the same mechanism as a namespace created inside a :stopdoc: region when some file contributes documentable contents, and stays out of the documentation otherwise. Resolution is a pure function of the declaration table, with two visible consequences. Names resolve against the declarations of all files regardless of the file order, so a superclass or mixin declared in a file built later is found. Within one file, forward declarations are visible as well; test expectations of position-dependent resolution are updated. Details ported from the single-pass behavior: a module named as a superclass is upgraded to a class (including for the implicit Object superclass), `class Cipher < Cipher` resolves the right-hand side to an outer namespace rather than the class the clause is defining, and the resolver ignores documentation suppression when collecting declarations: those declarations still define real Ruby constants, so they participate in name resolution. An implicit namespace - the owner of `class B::C` when no B is declared anywhere - is never invented while a real declaration could resolve the name: in real Ruby such code raises NameError unless something defined B first, so preferring a real declaration matches every load order that works. Nested undeclared roots stay pending until the table is stable and are then pinned as implicit namespaces, outermost first, so that one invented namespace can serve the deeper pendings. The kind of a namespace is tracked as class, module or unknown; an unknown kind falls back to a module when the namespace is created. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This pull request restructures the Ruby parser to emit an intermediate representation (IR), defers Ruby CodeObject construction until after all files in a batch are parsed, and preloads namespaces across the batch so constant/class/module name resolution becomes independent of file processing order.

Changes:

Add IR-based Ruby parsing (parse_ir + build_ir) and defer Ruby builds in RDoc::RDoc#parse_files to enable cross-file namespace preloading before building.
Introduce RDoc::Parser::Ruby::NamespaceResolver (batch fixed-point solver) and RDoc::Parser::Ruby::CodeObjectBuilder (replays IR to build CodeObjects).
Expand/adjust tests to cover order-independent resolution and additional parse_files edge cases.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/rdoc/rdoc_rdoc_test.rb	Adds parse_files regression tests for order-independent resolution and other multi-file edge cases.
test/rdoc/parser/ruby_test.rb	Updates expectations/sorting and clarifies assertions for new resolution behavior.
lib/rdoc/rdoc.rb	Defers Ruby parsing/building and adds batch build step with namespace preloading and improved parse error hint factoring.
lib/rdoc/parser/ruby.rb	Splits Ruby parsing into IR emission + build replay; emits IR records instead of mutating CodeObjects during AST walk.
lib/rdoc/parser/ruby_namespace_resolver.rb	New batch namespace resolution pass to create “ghost” namespaces before building.
lib/rdoc/parser/ruby_code_object_builder.rb	New IR replay engine that materializes comments/directives and constructs CodeObjects.
lib/rdoc/parser.rb	Ensures the new Ruby builder/resolver files are required when loading the parser subsystem.
lib/rdoc/code_object/class_module.rb	Adds `namespace_ghost` flag to support pre-created namespaces.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+  def build_ir(resolve: true)
+    NamespaceResolver.new(@store).preload_namespaces([self]) if resolve
+    builder = CodeObjectBuilder.new(@top_level, @store, @options, @stats, @preprocess, track_visibility: @track_visibility)
+    builder.run(@ir)
+    @top_level
  end


+    if payload[:startdoc] && !container.ignored?
+      # Compatibility: `module Net #:nodoc:` followed by :stopdoc:/:startdoc:
+      # regions is a common pattern that expects :startdoc: to make the
+      # container documentable again. Containers ignored here were created
+      # in a suppressed region and need documentable contents to revive.
+      container.start_doc
+      container.force_documentation = true
+    end


tompng and others added 3 commits August 7, 2026 19:51

Copilot AI lite review requested due to automatic review settings August 8, 2026 19:19

tompng requested a deployment to fork-preview-protection August 8, 2026 19:19 — with GitHub Actions Waiting

tompng marked this pull request as draft August 8, 2026 19:19

Copilot started reviewing on behalf of tompng August 8, 2026 19:19 View session

Copilot AI reviewed Aug 8, 2026

View reviewed changes

Navigation Menu

Uh oh!

Parser event ir - #1768

Parser event ir#1768
tompng wants to merge 3 commits into
ruby:masterfrom
tompng:parser_event_ir

tompng commented Aug 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tompng commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parse steps

Future possibilities

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tompng commented Aug 8, 2026 •

edited

Loading