Skip to content

fix(inspect): omit Darwin census rows reaped before the trailing identity read - #25

Merged
steipete merged 1 commit into
mainfrom
fix/darwin-census-reaped-rows
Oct 19, 2026
Merged

steipete merged 1 commit into
mainfrom
fix/darwin-census-reaped-rows

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Problem

On a loaded Mac, a Darwin census could return a row for a process that had already been reaped, with parentPid, processGroupId, and state but no identity. Consumers that require birth identity for every row in a lineage (OpenClaw's readDarwinProcessTree) then fail closed with a permission-style error even though the process simply exited.

Each Darwin row is bracketed by birth-identity reads: before, then topology, then optional command, then after. The trailing read was collapsed with .ok().flatten(), so Ok(None) (proven gone: kern.proc.pid has no record and kill(pid, 0) returns ESRCH) looked the same as an unreadable identity. The row kept bare topology instead of being omitted, which contradicts the documented contract that proven-gone PIDs are omitted.

Fix

When the trailing identity read returns Ok(None), skip the row, exactly as if topology had observed the disappearance. Everything else is unchanged:

  • Err from the trailing read (unreadable/denied) still retains topology without identity; denial never establishes absence.
  • A changed identity (different start time or parent) still drops identity and command; the PID-reuse bracket is not weakened.
  • before == Ok(None) is intentionally left alone: if the later topology read succeeds, some process holds that PID (only possible through PID reuse), so omitting it would falsely claim absence. If topology also finds it gone, the row is already omitted.

Proof

Deterministic regression (Rust, real OS): a #[cfg(test)] thread-local hook runs between topology and the trailing identity read. The test spawns /usr/bin/true, holds it as an unreaped zombie with waitid(WEXITED | WNOWAIT), asserts the selected census returns one zombie row with identity.exited, then reaps the child from inside the hook and asserts the census returns no row. With the old .ok().flatten() restored, the test fails with "a reaped PID must not keep bare topology". The hook is compiled only under cfg(test) and never reaches the addon. Because darwin_census is macOS-only and cargo test previously ran only on Ubuntu, the native workflow now runs cargo test --workspace --locked on the darwin-x64 and darwin-arm64 lanes (Node 24 entry only).

There is no external real-OS way to land a reap inside that window deterministically. The census is one synchronous native call, and the window is microseconds, so the existing pty/zombie fixture can hold a process in any state but cannot time a transition between two reads inside the call.

Statistical churn proof (Mac Studio, load average 73–98): 4 tight /bin/sh -c 'while :; do /usr/bin/true; done' loops in the caller's lineage; each iteration takes a full census and requires identity on every descendant of the caller.

Build Censuses Lineage rows Censuses with identity-less lineage rows
main (2f01cbd), instrumented 2,000 14,405 5
this PR 6,000 (3 × 2,000) 43,004 0

The instrumented main build logged why each identity-less same-owner row lost identity: all 5 were after == Ok(None) (3 running, 2 zombie at topology time), with zero before-error, after-error, or changed-identity cases. Every failing PID read back as null immediately after. At main's rate, about 15 failures were expected in 6,000 censuses (Poisson P(0) ≈ 3×10⁻⁷). Zombie lineage rows still carry identity after the fix (164–267 per 2,000-census batch).

Checks: PROC_SAFE_NATIVE=require pnpm check, cargo fmt --all --check, cargo clippy --workspace --locked -- -D warnings, cargo test --workspace --locked, actionlint, git diff --check.

Related: #23, openclaw/openclaw#176612 (draft bump to 0.4.2). No release is tagged or published here.

…tity read

Each Darwin census row brackets topology with birth-identity reads. The
trailing read was collapsed with .ok().flatten(), so a proven absence
(no kern.proc.pid record and kill ESRCH) looked like unreadable identity:
a process reaped after its topology read stayed in the census with
parent, group, and state but no identity, and lineage consumers that
require identity failed closed with a permission error.

Omit the row when the trailing read proves the PID gone, matching a
topology read that observes the disappearance. Unreadable and changed
identities keep the existing topology-only behavior, so denial handling
and the PID-reuse bracket are unchanged.

A cfg(test) hook reaps an unreaped zombie between the two reads for a
deterministic real-OS regression test; macOS native CI lanes now run
cargo test so the Darwin-only module's tests execute.
@clawsweeper

clawsweeper Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T23:45:13.804267Z b2aedc8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. labels Oct 18, 2026
@clawsweeper

clawsweeper Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

Omit Darwin census rows when the trailing identity read proves the process has disappeared.

Example: A selected zombie process is reaped after its topology is read but before its trailing identity read.

  • Before: The census returns its topology without identity, making disappearance resemble unreadable birth identity.
  • After: The census omits the row while preserving topology-only results for unreadable or changed identities.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The focused patch appears correct and has meaningful real-OS regression coverage; directly inspectable behavior evidence is strongest at the in-process native boundary.
Proof confidence 🦐 gold shrimp (3/6) This maintainer-authored PR is outside the ordinary contributor proof gate; its body reports zero identity-less lineage failures across 6,000 post-fix censuses, supported by a deterministic real-OS reap test.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: Concurrent process reaping can make lineage inspection fail with a permission-style error because a vanished process remains as a row without identity.
Reason: The maintainer-authored direction restores the documented proven-disappearance contract through a narrow change at its existing owner.

Merge readiness

✅ Ready for maintainer review

Keep this PR open: it fixes a current Darwin census race that main still contains, and review found no introduced correctness or security defect.

Priority: P2
Reviewed head: b2aedc8e50b913c8214bdd86d9f98f8c69c598dd

Before merge

None.

Findings

None.

Tests

  • Missing end-to-end proof: The body reports successful churn censuses, but provides no raw shipped-entry-point trace; the deterministic regression exercises the real native owner and OS inside the Rust test process.
Agent review details

How this fits together

The inspect subpath passes census requests to the native Darwin owner, which combines kernel topology and bracketed birth identities into process snapshots.

flowchart TD
  A[Inspect API] --> B[Darwin census]
  B --> C[Initial birth identity]
  C --> D[Topology and optional command]
  D --> E[Trailing birth identity]
  E --> F[Omit proven disappearance]
  E --> G[Return verified or topology-only row]
Loading

Technical review

Best possible solution:

Land the focused native omission fix with its deterministic regression test and aligned documentation.

Do we have a high-confidence way to reproduce the issue?

Main visibly retains topology after a trailing Ok(None); the supplied real-OS test deterministically reaches that window, and the body reports five matching failures across 2,000 main censuses.

Is this the best way to solve the issue?

This fixes the distinction at the existing census owner without retries or consumer workarounds, preserving PID-reuse comparisons, denied observations, and deadline checks.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 2f01cbdc3bb8.

Provenance checked

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

  • Exact introduced change: The pinned diff replaces trailing-read error flattening with an explicit Ok(None) omission; existing identity comparisons and command clearing remain intact. (native/src/inspect/darwin_census.rs:211, b2aedc8e50b9)
  • Main still contains the defect: Main collapses proven disappearance and unreadable identity into the same absent identity, then pushes the topology row. (native/src/inspect/darwin_census.rs:193, 2f01cbdc3bb8)
  • Absence remains independently established: The Darwin identity reader returns None only after the kernel-record lookup reports absence and kill(pid, 0) returns ESRCH; denial and incomplete observations remain errors. (native/src/identity/macos.rs:86, b2aedc8e50b9)
  • Deterministic real-OS regression: The test first verifies an unreaped zombie retains identity, then reaps it inside the census window and requires an empty result; the hook is excluded from production builds. (native/src/inspect/darwin_census.rs:258, b2aedc8e50b9)
  • Prior work distinguishes the remaining race: fix(inspect): report Darwin topology for exiting processes #23 explicitly records the residual permission-style failures caused by reaping between topology and the trailing identity read; its exiting-process topology fix does not solve this race. (2f01cbdc3bb8)
  • Policy and inspection limits: The complete root AGENTS.md was read; no nested policies, maintainer notes, or pr-simplify skill were found. The checkout remained clean and git diff --check passed. No target code or second reviewer was run. One historical stat lookup encountered an unavailable old object; the relevant original census blob and history were readable. (AGENTS.md:19, b2aedc8e50b9)

Likely related people:

  • Peter Steinberger: Raw commit 27f3562 adds native/src/inspect/darwin_census.rs:197 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 27f3562b3a4d; files: native/src/inspect/darwin_census.rs)

Labels

Label changes:

  • add P2: A narrow Darwin exit race can make lineage consumers reject otherwise usable censuses; the repair preserves existing public APIs and denial semantics.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: A narrow Darwin exit race can make lineage consumers reject otherwise usable censuses; the repair preserves existing public APIs and denial semantics.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 18, 2026, 7:55 PM ET / 23:55 UTC.

@steipete
steipete merged commit 9c217f8 into main Oct 19, 2026
24 checks passed
@steipete
steipete deleted the fix/darwin-census-reaped-rows branch October 19, 2026 00:01
steipete added a commit to openclaw/openclaw that referenced this pull request Oct 19, 2026
… processes exit (#176612)

Pin @openclaw/proc-safe 0.4.2. Its Darwin census now reports parentage, process group, owner, and state for processes already exiting (openclaw/proc-safe#23) and omits rows proven gone by its trailing identity read (openclaw/proc-safe#25). OpenClaw's shared Darwin owner no longer fails a full census, and Codex app-server containment no longer reports cleanup "uncertain", whenever an unrelated process on the Mac is mid-exit.

Adds an OpenClaw-boundary regression test that holds a child in XNU's exit window and requires readDarwinProcessTree and readDarwinProcessCensus to keep it with its topology. Release-age exclusions move to the 0.4.2 set (remove after 2026-10-26T00:51:31.079Z).

Admin-merged over one inherited CI failure: run 39185036717 job 117845878737, an unhandled ERR_STREAM_WRITE_AFTER_END in the extensions/browser/src/browser/chrome-mcp.cleanup.test.ts fixture teardown. That test is unchanged by this PR. The failure reproduces on main with proc-safe 0.4.1 (5/5 runs at e2b6418, 3/6 at 9d013ba), and the job ran on Linux, where proc-safe 0.4.2 is unchanged.
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Oct 19, 2026
… processes exit (openclaw#176612)

Pin @openclaw/proc-safe 0.4.2. Its Darwin census now reports parentage, process group, owner, and state for processes already exiting (openclaw/proc-safe#23) and omits rows proven gone by its trailing identity read (openclaw/proc-safe#25). OpenClaw's shared Darwin owner no longer fails a full census, and Codex app-server containment no longer reports cleanup "uncertain", whenever an unrelated process on the Mac is mid-exit.

Adds an OpenClaw-boundary regression test that holds a child in XNU's exit window and requires readDarwinProcessTree and readDarwinProcessCensus to keep it with its topology. Release-age exclusions move to the 0.4.2 set (remove after 2026-10-26T00:51:31.079Z).

Admin-merged over one inherited CI failure: run 39185036717 job 117845878737, an unhandled ERR_STREAM_WRITE_AFTER_END in the extensions/browser/src/browser/chrome-mcp.cleanup.test.ts fixture teardown. That test is unchanged by this PR. The failure reproduces on main with proc-safe 0.4.1 (5/5 runs at 85afd2a, 3/6 at 11d9c97), and the job ran on Linux, where proc-safe 0.4.2 is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant