Skip to content

fix(inspect): report Darwin topology for exiting processes - #23

Merged
steipete merged 1 commit into
mainfrom
fix/darwin-census-exiting-topology
Oct 18, 2026
Merged

steipete merged 1 commit into
mainfrom
fix/darwin-census-exiting-topology

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Problem

On Darwin, listProcesses() returns a bare row { pid, owner: "unknown" } for a process that is in its exit window, while readProcessIdentity(pid) still returns a live identity (exited: false). Once XNU starts exiting a process, libproc's proc_pidinfo(PROC_PIDT_SHORTBSDINFO) fails with ESRCH, but the kernel record (sysctl KERN_PROC_PID) still describes it as SRUN/SSTOP with P_WEXIT set. The census already resolved that ESRCH from the kernel record for zombies, but returned unknown facts for every other state.

Consumers that fail closed on unreadable topology (OpenClaw's shared Darwin census owner) therefore fail a whole census whenever any process on the host is mid-exit. On a loaded Mac Studio (load ~110), 31% of OpenClaw readDarwinProcessTree calls failed this way, and it is the remaining cause of intermittent cleanup: "uncertain" in OpenClaw's Codex app-server containment tests.

Change

  • The kernel-record shim also reports whether the record has P_WEXIT.
  • On libproc ESRCH, the census fills parent PID, process group, owner, and state from the kernel record when it is a zombie or exiting. Every other ESRCH record keeps the bare unknown row; EPERM/EACCES rows stay bare; kernel-record read errors still propagate. No new ProcessState, no change to identity.exited, Windows untouched.
  • Docs and README describe the exiting case; changelog entry under Unreleased.

Evidence

Which ESRCH rows exist. A C probe ran full proc_listallpids + short-info censuses for 60 s at load ~110: 6,500 censuses / 11.7M rows. Every libproc-ESRCH row whose kernel record still existed was SZOMB (32,749, all with P_WEXIT) or SRUN + P_WEXIT (5,402). There were zero ESRCH rows that were neither, so gating on SZOMB || P_WEXIT covers the observed class without accepting unexplained records.

Deterministic regression on the real OS boundary. The census fixture now holds its child inside the exit window. The child keeps a pty slave open with undrained output, and the fixture never reads the master. XNU's last-close drain holds the child at libproc ESRCH + SRUN + P_WEXIT for a stable ~600 ms before it becomes a zombie. The fixture verifies that exact kernel condition in C before printing exiting, fails loudly if the window is missed, and re-checks the window after the proof's selected and full censuses, so a late sample cannot pass. The new stage runs on the main thread, because a Worker spawn does not fit the window reliably under load. The existing running/stopped/zombie/gone stages keep main + Worker coverage.

  • Unfixed native code: real Darwin census and Worker boundaries fails with exiting row must retain its parent PID (actual undefined), with selected and full rows {"pid":…,"owner":"unknown"}, while the fixture confirms stillExiting: true, libprocErrno: 3, state: 2, flags: 0x2084.
  • Fixed: PROC_SAFE_NATIVE=require pnpm test test/inspect.test.ts passed 24/24 three consecutive times at load ~110. The standalone proof reports passed: 9, with censuses completing ~12 ms into a ~601 ms window.

Gates (macOS arm64, load ~100–125): PROC_SAFE_NATIVE=require pnpm check passed (104 passed, 8 skipped, pack check OK); cargo clippy --workspace --locked -- -D warnings clean; cargo test --workspace --locked passed; pnpm package:smoke passed host-only (npm and pnpm root-only installs, main and Worker); git diff --check clean. One run of the unrelated test/native-status.test.ts hit its 5 s timeout under this load and passed on rerun.

Downstream impact (OpenClaw owner readDarwinProcessTree, same owner code, 2,000 censuses each).

mode 0.4.1 this branch
natural host load 627 failed (31.35%, all unavailable) 0 failed
4 exit-churn loops in the caller's own lineage 630 failed (625 unavailable, 5 permission) 5 failed (permission)

The residual permission class exists before and after this change. A selected-lineage process gets reaped between the census's topology read and its trailing birth-identity bracket, so the row keeps topology without identity. It is tracked separately.

Not included

No version bump, tag, or release; releasing needs maintainer approval.

libproc returns ESRCH once a process starts exiting, while its kernel record
still reports SRUN/SSTOP with P_WEXIT. The census only resolved zombies from
that record, so exiting processes produced bare unknown rows. Accept kernel
topology for zombies and P_WEXIT records; other ESRCH records stay unknown.
@clawsweeper

clawsweeper Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T23:04:13.389130Z fab9abd PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. labels Oct 18, 2026
@clawsweeper

clawsweeper Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

The Darwin census reports kernel-observed topology for exiting processes that libproc can no longer inspect.

Example: Darwin returns libproc ESRCH while a retained process has kernel state SRUN and P_WEXIT set.

  • Before: The census returns only the PID and owner: "unknown".
  • After: The row includes parent PID, process group, ownership, and state: "running", without marking the identity exited.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) A focused, source-supported repair with real Darwin regression coverage; directly inspectable proof remains harness-based.
Proof confidence 🦐 gold shrimp (3/6) The supplied maintainer-authored designation exempts ordinary contributor proof gating; the body reports real macOS arm64 selected/full census results and downstream comparisons, while the inspectable regression path is a real-OS harness rather than an attached shipped-entry-point artifact.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A process exiting during a Darwin census loses readable parentage and ownership facts, causing strict process-tree consumers to reject otherwise usable observations.
Reason: The existing feature owner supplies a narrowly scoped repair for a demonstrated kernel state; the separately described trailing-identity race is outside this defect.

Merge readiness

✅ Ready for maintainer review

This PR fixes a source-confirmed Darwin census defect still present in main and v0.4.1; no actionable production correctness or security defect was found.

Priority: P2
Reviewed head: fab9abd5adcc6d847efefe213923bd69d6738a37

Before merge

None.

Findings

None.

Tests

  • Missing end-to-end proof: The exiting-window scenario is exercised only on the main thread; existing running, stopped, zombie, and gone scenarios retain Worker coverage.
Agent review details

How this fits together

The public inspection API passes census requests to native Rust, which reads Darwin process records and returns frozen JavaScript snapshots with independently observed topology and optionally verified identity.

flowchart TD
  A[Inspection caller] --> B[TypeScript census API]
  B --> C[Rust Darwin census]
  C --> D[libproc short records]
  D --> E[Kernel record fallback]
  E --> F[Birth identity checks]
  F --> G[Process snapshots]
Loading

Technical review

Best possible solution:

Retain the focused extension of the existing kernel-record fallback while preserving unknown facts, denial handling, and identity brackets.

Do we have a high-confidence way to reproduce the issue?

Current-main source deterministically returns a bare row for ESRCH plus a non-zombie kernel record; the PR supplies a real-OS fixture and reports its before-and-after failure, which this review did not execute.

Is this the best way to solve the issue?

The change fixes the existing topology owner directly, using the kernel's explicit exiting flag without adding retries, application policy, public options, or a parallel inspection path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 90b7b6b5b715.

Provenance checked

  • Darwin ESRCH topology fallback changes intended behavior with a stated reason (27f3562: The original census used validated kernel records for libproc-excluded zombies and kept unexplained observations unknown.)
  • Darwin identity kernel bridge keeps the original intent (5852b3e: The SDK-compiled bridge distinguishes retained kernel objects from proven absence; the added flag preserves that owner and its validation.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

  • Current-main defect: The released census discards readable kernel topology whenever libproc returns ESRCH and the kernel state is not SZOMB, including the reported SRUN plus P_WEXIT exit window. (native/src/inspect/darwin_census.rs:78, 90b7b6b5b715)
  • Focused implementation: The pinned diff adds the matching C and Rust exiting field and extends the existing fallback condition; denial handling, kernel-layout validation, disappearance checks, and birth-identity brackets remain intact. (native/src/inspect/darwin_census.rs:78, fab9abd5adcc)
  • Real Darwin regression scenario: The fixture holds a child in the pty exit-drain window and verifies ESRCH, P_WEXIT, and non-zombie state; the script checks selected and full census results before rechecking that the window remains open. (scripts/darwin-census-proof.mjs:70, fab9abd5adcc)
  • Supplied behavior evidence and discussion: The complete PR body reports the regression failing against unfixed native code and passing after the change, plus downstream census comparisons; discussion contains no additional owner decision or unresolved review finding. (fab9abd5adcc)
  • Original fallback intent: feat(inspect): add Darwin process census and PID selection #19 introduced independent topology and the zombie fallback while retaining unknown-fact semantics and PID-reuse checks; this PR explains the additional kernel-proven exit case. (native/src/inspect/darwin_census.rs:78, 27f3562b3a4d)
  • Exact-head checks: Twenty-two checks pass, including all four Darwin architecture and Node combinations; two FreeBSD ARM64 checks remain in progress, with no failed check attributable to this diff. (fab9abd5adcc)

Review metrics

Metric Value Why it matters
Production and regression scope Native production +7/-4 lines; regression script, fixture, and test +145/-24 lines The larger validation change creates and verifies the real Darwin exit window rather than mocking the kernel condition.

Labels

Label changes:

  • add P2: Concurrent Darwin exits lose readable topology and can make downstream process-tree consumers reject a census, with a focused fix and limited platform scope.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: Concurrent Darwin exits lose readable topology and can make downstream process-tree consumers reject a census, with a focused fix and limited platform scope.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Extend the exiting-window scenario to a Worker initialized before triggering exit, avoiding Worker startup inside the short window.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 18, 2026, 7:15 PM ET / 23:15 UTC.

@steipete
steipete merged commit 2f01cbd into main Oct 18, 2026
24 checks passed
@steipete
steipete deleted the fix/darwin-census-exiting-topology branch October 18, 2026 23:23
steipete added a commit to openclaw/openclaw that referenced this pull request Oct 19, 2026
A process inside XNU's exit window is hidden by libproc (ESRCH) while
kern.proc.pid still shows it with P_WEXIT. proc-safe 0.4.1 returned a bare
row for it, so the shared Darwin census owner failed every full census
while any process on the host was mid-exit, and Codex app-server
containment fell back to uncertain cleanup on busy Macs.

Hold a setsid child in that window with an unread pty drain, verify the
kernel window before and after inspection, and require readDarwinProcessTree
and readDarwinProcessCensus to keep the child with its parent, group and
running state. Red on proc-safe 0.4.1 (inspection unavailable); green with
openclaw/proc-safe#23, which lands here through the 0.4.2 bump.
steipete added a commit to openclaw/openclaw that referenced this pull request Oct 19, 2026
… processes exit (#176612)

Pin @openclaw/proc-safe 0.4.2. Its Darwin census now reports parentage, process group, owner, and state for processes already exiting (openclaw/proc-safe#23) and omits rows proven gone by its trailing identity read (openclaw/proc-safe#25). OpenClaw's shared Darwin owner no longer fails a full census, and Codex app-server containment no longer reports cleanup "uncertain", whenever an unrelated process on the Mac is mid-exit.

Adds an OpenClaw-boundary regression test that holds a child in XNU's exit window and requires readDarwinProcessTree and readDarwinProcessCensus to keep it with its topology. Release-age exclusions move to the 0.4.2 set (remove after 2026-10-26T00:51:31.079Z).

Admin-merged over one inherited CI failure: run 39185036717 job 117845878737, an unhandled ERR_STREAM_WRITE_AFTER_END in the extensions/browser/src/browser/chrome-mcp.cleanup.test.ts fixture teardown. That test is unchanged by this PR. The failure reproduces on main with proc-safe 0.4.1 (5/5 runs at e2b6418, 3/6 at 9d013ba), and the job ran on Linux, where proc-safe 0.4.2 is unchanged.
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Oct 19, 2026
… processes exit (openclaw#176612)

Pin @openclaw/proc-safe 0.4.2. Its Darwin census now reports parentage, process group, owner, and state for processes already exiting (openclaw/proc-safe#23) and omits rows proven gone by its trailing identity read (openclaw/proc-safe#25). OpenClaw's shared Darwin owner no longer fails a full census, and Codex app-server containment no longer reports cleanup "uncertain", whenever an unrelated process on the Mac is mid-exit.

Adds an OpenClaw-boundary regression test that holds a child in XNU's exit window and requires readDarwinProcessTree and readDarwinProcessCensus to keep it with its topology. Release-age exclusions move to the 0.4.2 set (remove after 2026-10-26T00:51:31.079Z).

Admin-merged over one inherited CI failure: run 39185036717 job 117845878737, an unhandled ERR_STREAM_WRITE_AFTER_END in the extensions/browser/src/browser/chrome-mcp.cleanup.test.ts fixture teardown. That test is unchanged by this PR. The failure reproduces on main with proc-safe 0.4.1 (5/5 runs at 85afd2a, 3/6 at 11d9c97), and the job ran on Linux, where proc-safe 0.4.2 is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant