feat: external + in-process plugins (4 transports incl. native dylib)

[raphaelvigee]

Make heph plugins runnable as separately-built, hot-swappable artifacts over four transports behind one logical interface — out-of-process for isolation/polyglot, in-process for native speed.

Transport	Where	Cost	Use
proto (bin:)	out-of-process, UDS	~µs/call	portable, polyglot, OS-sandboxable
shm	out-of-process, iceoryx2	low syscall	high-volume out-of-process
wasm (wasm:)	in-process, wasmtime	host-call	pure-compute, capability-sandboxed
dylib (dylib:)	in-process, stable ABI (stabby)	native	trusted/first-party, max speed

In-process and remote plugins coexist permanently; query/hostbin/group/statictarget stay in-process. Engine core is untouched — every transport registers through the existing factory hooks.

In-process stable-ABI transport (dylib:) — the perf headline

Out-of-process plugins paid ~2×: test //... over the go plugin ran ~7s vs ~3.1s compiled-in. Profiling pinned the gap on the async Mux transport (tokio mpsc + duplex wakeups + frame allocs) — not serialization, and not the ~22k note_dep/result callbacks (they fan out concurrently, hiding latency; an A/B with a spin-mailbox confirmed it).

Fix: a native stable ABI (crates/plugin-stabby, on stabby) so a plugin loads in-process as a cdylib and is called via direct vtable dispatch — no IPC, no hot-path serialization — while staying a separate, ABI-checked, hot-swappable artifact.

• StableExecutor (result/note_dep/query) crosses callbacks natively; StableProvider/StableManagedDriver cross cold methods as prost pb::Frame bytes; get takes the native executor.
• crates/plugin-go-cdylib: the go plugin as a loadable cdylib (#[stabby::export]).
• Host loader: libloading + stabby get_stabbied type-report ABI check (mismatched dylib = hard error).
• The cdylib owns its own tokio runtime (its statically-linked tokio ≠ the host's), so a driver run shelling out (go list) works.
• mux: a try_inline fast path serves synchronous note_dep without a task spawn (for the bin: path).

Result: go plugin via dylib: runs ~3.4s (vs the 3.1s compiled-in floor), fresh and cached builds correct. Full ladder: 3.1s compiled-in · 3.4s dylib · 4.7s prost-in-process · 7s out-of-process.

Config

Each provider/driver entry picks a transport (mutually exclusive); absent = in-process built-in:

- name: go
  dylib: { path: .heph3/heph-go-plugin.dylib }   # or url: with {os}/{arch}
- name: other
  bin:  { path: ... | exec: [...] | url: ... }
- name: compute
  wasm: { path: ... | url: ... }

dylib:/wasm: share one path/url artifact source (url downloaded + cached like bin:). gen-example builds + installs the cdylib; example/.hephconfig2 uses dylib:.

Milestones

• M0 — ABI schema + plugin-abi + plugin-sdk + parity tests
• M1 — plugin-remote + transports + host_service + lease
• M2 — plugin-go out-of-process (note_dep/result split); deliberate-cycle test
• M3 — shm tier (iceoryx2)
• M4 — wasm tier + launch/sandbox
• M5 — in-process stable-ABI (dylib:) transport at native speed; dylib:/wasm:/url config

Notes

• Result/input artifacts are abstract readers (Content: file/sqlite/in-mem/remote) — cross as opaque handles / eagerly-read bytes, never paths. Only driver run outputs (fresh files) cross as paths.
• Host-side dep-cycle invariant preserved: every result/note_dep registers the DepDag edge; note_dep is the cache-hit fast path.
• Perf analysis + won't-fix notes in ai-docs/PERFORMANCE.md (not committed).

🤖 Generated with Claude Code

[raphaelvigee]

Progress update

Done, tested, committed:

• M0 — ABI schema (proto/plugin/v1), plugin-abi (generated pb + rkyv hot-message mirror + parity tests), plugin-sdk skeleton, wit/heph-plugin.wit.
• M1 — full proto transport: length-prefixed Frame framing + bidirectional id-multiplexed Mux (plugin-abi, transport feature); host RemoteProvider + callback service + lease table + subprocess spawn_plugin (fd-3 inheritance); guest serve/serve_inherited; reference crates/plugin-echo. Proven in-process (4 tests) and over a real subprocess (1 test). shm/wasm are feature-gated stubs.
• M2 (provider path) — pb↔engine conversions (convert feature); artifact byte streaming + tar walk over the result() callback (the plugin-go hot read). e2e reads package.bin through the transport. All green; clippy -D warnings clean.

Remaining (each substantial, some need a decision):

• RemoteDriver — blocked on the raw_def round-trip: parse() yields a concrete Arc<dyn RawDef> that run() downcasts; over the wire it's opaque JSON. Resolvable via an hplugin::TargetDef deserialize-capable raw_def, or a guest-side parse-time concrete-def cache (one long-lived plugin process per engine makes this safe). run() also needs input-artifact materialization into the shared sandbox dir + output collection.
• plugin-go migration — wire its provider+drivers to run remote; go-toolchain e2e; deliberate-dep-cycle test (must return CycleError, not deadlock).
• note_dep edge-only ProviderExecutor API (currently falls back to result()).
• M3 — real iceoryx2 shm transport + batching + benchmark vs in-process.
• M4 — wasmtime component transport + launch sandbox modes (exec / exec_in_container / container; landlock/seccomp/.sb).

The hard architectural pieces (the bidirectional transport + the abstract-artifact byte path) are done and tested; the rest is well-scoped follow-on work.