Skip to content

Port rewatch to OCaml - #8780

Open
cknitt wants to merge 93 commits into
compiler-request-isolationfrom
rewatch-ocaml-port
Open

cknitt wants to merge 93 commits into
compiler-request-isolationfrom
rewatch-ocaml-port

Conversation

@cknitt

@cknitt cknitt commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Stacked on #8864, which the in-process follow-up #8781 builds on.

This PR replaces the Rust implementation of rewatch, ReScript's build system and CLI, with an OCaml port as the rescript executable on Linux, macOS, and Windows. The Rust implementation still ships as rescript-rust for comparison and as a fallback.

The compiler and almost all supporting tools are OCaml. Moving the build system to OCaml gives the project one implementation language. It also shrinks the dependency surface: about 143 crates become 14 OPAM packages. And it lets the build system call the compiler directly later on. In this PR bsc still runs as a separate process for every parse and compile, exactly as with Rust rewatch. The follow-up PR stacked on this one moves compilation in-process onto OCaml domains, which is where the performance gains are.

Behavior

The port follows Rust rewatch's architecture and algorithms. It does not translate the Rust code line by line. On the rewatch/testrepo fixture, both implementations run the same bsc invocations for clean, unchanged, and single-edit builds (1031 / 4 / 6). They also produce identical file sets and byte-identical generated JavaScript, .cmi, .cmj, ASTs, and namespace maps. The shared rewatch/tests suite runs against both executables.

Deliberate differences from Rust rewatch

  • rescript.json maps interpreted by the build system reject duplicate keys instead of silently using the last one.
  • A namespace-entry without a namespace is an error instead of being ignored.
  • rescript clean also removes out-of-source JavaScript outputs.
  • The CLI uses Cmdliner:
    • help is rendered as a man page;
    • --no-timing no longer takes a value;
    • help takes precedence over version, and version is also accepted after a subcommand.
  • --filter supports the common subset of Rust's regex syntax. Unsupported constructs such as (?i) are rejected with an error.
  • Configuration errors use the compiler-style path: message form (path:line:column: for JSON syntax errors), and they name the field and the accepted values instead of the parser's internal types.
  • format -v and compiler-args -v write their logs to stderr, so stdout stays formatted source or JSON.
  • OpenTelemetry export is not implemented.
  • A failing --after-build command makes the build fail.

Rust rewatch bugs that the port does not have

  • Watch events used the opposite filter semantics from source discovery.
  • Incremental dependency replacement kept obsolete reverse edges.
  • Deleting generated JavaScript did not regenerate it (Rewatch should recreate file after re-run of watch #7728).
  • Removing a source directory could leave its previous outputs behind.
  • Dependency fingerprints ignored inherited JSX and experimental options.
  • Malformed configuration or dependency states could panic or hang the scheduler.
  • SIGHUP/SIGTERM sent to rescript left compiler children running.
  • A copy into lib/ocaml that failed partway left a truncated .cmi or .cmj for the next build; the port removes it, so the next build publishes it again.
  • The project-context debug line printed Rust's internal Debug representation. Both implementations now print the same plain sentence (a small change in rewatch/src/project_context.rs).

Like Rust, rescript.json must be standard JSON: comments, unquoted keys, and NaN are rejected.

Performance

These numbers come from interleaved clean builds of rewatch/testrepo on a 10-core Linux container, comparing against release Rust:

Rust OCaml
Clean build (median of 7) 4697 ms 4965 ms 1.06×
Peak process-tree RSS 1872 MiB 1940 MiB 1.04×
Retained-watch single edit 140 ms 123 ms 0.88×

The remaining clean-build overhead is in subprocess handling: every bsc child gets reader threads for its pipes. The in-process follow-up removes it.

Packaging

The platform packages contain both rescript.exe (OCaml) and rescript-rust.exe. Linux binaries are linked statically. The luv/libuv dependency is built from a small overlay in .github/opam-repository that fixes static linking on arm64 musl. License notices for the bundled OCaml libraries ship in each platform package.

Reviewing this PR

Of the about 29k added lines, 12.9k are the implementation in rewatch-ocaml/. Most of the rest is tests: 5.4k unit tests and 5.7k shell tests. There are also 2.6k lines of license notices, one copy per platform package, and the infrastructure: CI, the opam overlay, the JS launchers and the scripts. rewatch-ocaml/README.md maps each area to its modules, and rewatch-ocaml/AGENTS.md states the rules the code follows: OS code behind platform.mli, one owner for every child process, pipe, lock and temporary file, and signal handlers that only record the request.

Suggested reading order, with the Rust counterparts for side-by-side comparison:

Layer Lines Modules Rust
Entry and CLI 0.8k rescript_ocaml, cli, output, source_filter main.rs, cli.rs
Config, project and package graph 3.3k strict_json, config_decode, config, project_context, package_resolution, package_traversal, source, package_graph config.rs, project_context.rs, build/packages.rs, helpers.rs
Platform, processes and locks 2.4k platform.mli, platform_unix, platform_windows, windows_job_stubs.c, process_child, process, build_lock, file_util lock.rs; Rust uses std::process and rayon
Build pipeline 5.4k module_graph, build_preparation, build_artifacts, compiler_info, package_parse, package_compilation, compiler_scheduler, compiler_process, build_attempt, build_session, build, clean, format build.rs, build/*.rs, format.rs
Watcher 1.4k native_watcher, watch_scope, watch_snapshot, watcher watcher.rs, queue.rs

rescript build runs Build.run, then Build_preparation.run: discovery, stale-output cleanup, parallel parse and the module graph. After that come Package_parse and Package_compilation per package, and finally Compiler_scheduler.run on Process.run_dependency_graph.

Where review time is best spent:

  • Concurrency and processes:
    • the worker pool and its cancellation in process.ml;
    • process_child.ml, which owns the pipes, the reader and waiter threads, and the escalation from SIGTERM to SIGKILL for the process group;
    • the signal handling in rescript_ocaml.ml and watcher.ml.
  • Locks: build_lock.ml, which covers hard-link publication, stale-lock takeover and the liveness probes in platform_*.ml. A probe that is too lax steals a live build's lock, and one that is too strict leaves a lock that never clears.
  • Deleting files:
    • Build_artifacts.cleanup_stale, which decides which public JavaScript outputs the build owns;
    • Compiler_info.clean_package;
    • Clean;
    • File_util.remove_tree.
  • Incremental and watch builds: Build.prepare_incremental, the session kept after failed attempts, and the snapshot reconciliation in watch_snapshot.ml, which keeps changes from being missed.
  • Windows: platform_windows.ml and windows_job_stubs.c (Job Objects, inherited-handle lists, quoting for cmd.exe). They only run on Windows CI, though the unit tests type-check them on every platform.

Safe to skim: config decoding and its error messages, which check_config_acceptance.sh compares with Rust; the Cmdliner definitions; output formatting; and the small helpers.

Besides the shared rewatch/tests suite, the check_*.sh scripts run Rust and OCaml side by side, comparing exit codes and output for command validation, configuration acceptance, verbose output and interactive (pty) output.

Tests

  • tests/rewatch_ounit_tests: unit tests for configuration, package and module graphs, invalidation, compiler arguments, scheduling, locks, and the watcher. New cases cover:
    • monorepo members building with the parent workspace's settings;
    • namespace names with an entry (cross-checked against Rust's convert_case);
    • SGR stripping;
    • lock creation without hard links, and a lock left empty by a dead writer;
    • strict JSON validation, with positions;
    • configuration error wording.
  • rewatch-ocaml/tests/run.sh: focused end-to-end tests, including:
    • SIGTERM, SIGHUP, and SIGQUIT during a build;
    • dependency rebuilds after package-spec changes;
    • deterministic outputs when a listed dependency is built on its own;
    • signals during a js-post-build hook;
    • compiler-args -v keeping stdout valid JSON;
    • format reporting a package-name mismatch once;
    • watch mode exiting promptly while another build holds the build lock.
  • tests/build_tests/cli_help: compares the help output exactly, and checks that output piped through a color-capable terminal contains no escape codes.
  • Lock-owner detection accepts only the build-system executables (rescript, rescript-rust, the dune development executable), in both implementations, so a reused PID of another rescript-* tool no longer keeps a stale lock. Unit tests cover the names on both sides.

The shared rewatch/tests snapshots are unchanged from master. The Rust implementation gets two small changes:

  • the lock-owner rule above, which is needed so the two binaries respect each other's locks;
  • the plain project-context debug line.

🤖 Generated with Claude Code

cknitt added a commit that referenced this pull request Oct 10, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt
cknitt added this pull request to stack #8782 October 10, 2026 16:36
@pkg-pr-new

pkg-pr-new Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript@8780

@rescript/belt

npm i https://pkg.pr.new/@rescript/belt@8780

@rescript/darwin-arm64

npm i https://pkg.pr.new/@rescript/darwin-arm64@8780

@rescript/darwin-x64

npm i https://pkg.pr.new/@rescript/darwin-x64@8780

@rescript/linux-arm64

npm i https://pkg.pr.new/@rescript/linux-arm64@8780

@rescript/linux-x64

npm i https://pkg.pr.new/@rescript/linux-x64@8780

@rescript/runtime

npm i https://pkg.pr.new/@rescript/runtime@8780

@rescript/win32-x64

npm i https://pkg.pr.new/@rescript/win32-x64@8780

commit: 002d5dd

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.94%. Comparing base (4acb29d) to head (002d5dd).

Additional details and impacted files
@@                     Coverage Diff                     @@
##           compiler-request-isolation    #8780   +/-   ##
===========================================================
  Coverage                       80.94%   80.94%           
===========================================================
  Files                             453      453           
  Lines                           62839    62839           
===========================================================
  Hits                            50865    50865           
  Misses                          11974    11974           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown

@cknitt
cknitt force-pushed the rewatch-ocaml-port branch from ea54579 to 042e314 Compare October 12, 2026 05:57
cknitt added a commit that referenced this pull request Oct 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt
cknitt force-pushed the rewatch-ocaml-port branch from 042e314 to 2a090ed Compare October 12, 2026 06:08
cknitt added a commit that referenced this pull request Oct 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt
cknitt force-pushed the rewatch-ocaml-port branch 4 times, most recently from bfb3808 to 374d93a Compare October 14, 2026 10:28
@cknitt
cknitt force-pushed the rewatch-ocaml-port branch from 374d93a to ca31093 Compare October 14, 2026 15:51
@cknitt
cknitt removed this pull request from stack #8782 October 14, 2026 15:51
@cknitt
cknitt changed the base branch from master to compiler-request-isolation October 14, 2026 15:52
@cknitt
cknitt added this pull request to stack #8865 October 14, 2026 15:52
@cknitt

cknitt commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T18:17:38.126043Z 82047c0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca31093b2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rewatch-ocaml/watch_scope.ml
@cknitt
cknitt force-pushed the rewatch-ocaml-port branch from ca31093 to 82047c0 Compare October 14, 2026 17:56
@cknitt

cknitt commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82047c029d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rewatch-ocaml/compiler_args.ml Outdated
cknitt and others added 5 commits October 14, 2026 19:39
Signed-off-by: Christoph Knittel <ck@cca.io>
Like Rust rewatch, a package listed by its parent rescript.json builds with
the parent's output settings. Every package now uses the root's settings,
replacing the separate build owner that made outputs depend on build history.
Belt's own lib/es6 and lib/js outputs are no longer tracked.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A string namespace with a namespace-entry is Pascal-cased as by Rust's
convert_case, so "my_lib" is named MyLib again.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Children run in their own process groups, so these signals reached only
rescript and left the children and the build lock behind.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also document the exit statuses the CLI actually uses.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cknitt and others added 26 commits October 14, 2026 19:39
Follows #8756: Rust rewatch no longer passes -bs-jsx-mode and accepts both
fields with any value.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The warning cleanup on exit gives up after about two seconds instead of
waiting up to a minute.

Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watcher now logs each watch path at debug level, as Rust rewatch does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Ports the copy skip from Rust rewatch (#8769).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Mutex.protect needs OCaml 5.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
The formatter emits CRLF on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
Windows rejects locking an unbounded range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
macOS can deliver both edits in one batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christoph Knittel <ck@cca.io>
A clean test-repo build takes over 20 seconds on macOS Intel runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Rust now capitalizes only an ASCII first letter (#8797), as this port did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
The next build then publishes the .cmi or .cmj again instead of reading a
truncated one. Copies stay in place, like Rust rewatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
The test failed once on macOS Intel CI without leaving output to diagnose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Coarse file times could hide a same-size edit, so the watcher missed it and a unit test hung on Windows CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Port of Rust rewatch's schema test from #8835.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
The copies differed in timeouts and diagnostics; the shared ones wait 20 s and report what they waited for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Rust rewatch does this too since #8540.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
… change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZbCuRQ1Tn31bnKzLj5e72
Signed-off-by: Christoph Knittel <ck@cca.io>
@cknitt
cknitt force-pushed the rewatch-ocaml-port branch from 82047c0 to 002d5dd Compare October 14, 2026 19:46
@cknitt
cknitt marked this pull request as ready for review October 15, 2026 04:15
@cknitt
cknitt requested a review from cristianoc October 15, 2026 04:16

@cristianoc cristianoc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not going to read this one. Seems pretty well tested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants