Skip to content

os: start processes with posix_spawn - #5634

Open
yohimik wants to merge 5 commits into
tinygo-org:devfrom
yohimik:upstream-pr/os-exec-posix-spawn
Open

os: start processes with posix_spawn#5634
yohimik wants to merge 5 commits into
tinygo-org:devfrom
yohimik:upstream-pr/os-exec-posix-spawn

Conversation

@yohimik

@yohimik yohimik commented Aug 30, 2026

Copy link
Copy Markdown

os: start processes with posix_spawn

Implement process creation and waiting on hosted Linux and Darwin.
Darwin needs the fcntl wrapper fix in #5612 and the libSystem symbols in #5636.
Concurrent process creation and pipe creation also need #5630.

Changes

The old process layer refused ProcAttr.Files, Dir, and Sys.
os/exec supplies Files, so it could not start commands.

  • Use posix_spawn so that Go code does not run in a child after fork.
    This avoids inherited runtime and collector locks.
  • Map ProcAttr.Files to child descriptors. Close nil entries and unnamed
    standard descriptors. Apply Dir with addchdir_np.
  • On Darwin, open the absolute path /dev/null in each nil or unnamed slot,
    then close that slot. This makes the close valid even if the descriptor was
    already closed. Both actions run in the child after sources have been saved.
    There is no parent descriptor check or mutation.
  • Preserve sources before an earlier file action can replace or close them.
    This handles cycles, repeated sources, and sources below their target index.
    Temporary descriptors are above all sources and targets. They exist only
    in the child and are closed after the mappings. Parent descriptors stay intact.
  • Keep the file objects alive until the spawn completes.
  • Inherit the parent environment when Env is nil and clear the child signal mask.
  • Support SysProcAttr.Setpgid and Pgid. Refuse other nonzero fields by name
    with an error that wraps ErrNotImplementedSys.
  • Use wait4, retry EINTR, and report the process ID, exit status, and resource
    usage. Support Kill and Signal, including ErrProcessDone after a reap.
  • Mark Darwin pipe descriptors close-on-exec under syscall.ForkLock.
    Linux uses pipe2 with O_CLOEXEC.

Targets without a process model keep the previous stubs.

Tests

The existing process tests cover exit status, kill, missing programs, working
directory, process groups, refused attributes, pipe inheritance, and closed
standard descriptors.

src/os/exec_remap_test.go adds cycle, repeated-source, closed-source, identity,
and sparse mappings. It checks the child mappings, closure of nil slots and
temporary descriptors, and access to the original parent files. It also checks
an invalid source descriptor. The child uses real descriptor reads and exec
redirection, not /dev/fd metadata. These checks use /bin/bash for descriptor
numbers above 9 and explicitly skip when that shell is absent.

The Linux source-overlay run used Debian 13 with Bash 5.2.37. All five cases
ran. A separate Alpine 3.24.1 check without Bash reported SKIP for the mapping
test, even though the binary's final status was PASS. That is skipped
coverage, not a successful remapping test. The targeted Alpine harness installed
Bash with apk add --no-cache bash, then ran the same test binary. All five
cases passed with Bash 5.3.9.

The checked Linux workflow uses Alpine for release builds and Ubuntu for
hosted standard-library tests. The listed APK dependencies do not include Bash,
and their simulated dependency resolution on Alpine 3.24.1 did not add it.
No workflow change was made. A targeted Alpine test harness must install Bash
and check the individual test output for skips.

Current follow-up results on Linux arm64 use the released TinyGo 0.42.0 compiler
and Go 1.27.0. A private TINYGOROOT copy contains this branch's source files and
the released compiler's library files. Compilation uses -p 1 and
GOMAXPROCS=2.

  • Before the fix, the cycle, repeated-source, closed-source, and sparse tests fail.
    The identity case passes.
  • With the fix, tinygo test -p 1 -v -run TestForkExec os passes.
  • With the fix, the full tinygo test -p 1 os suite passes.

The first Darwin run at 8af70341 found two defects. The /dev/fd test checks
also failed with ordinary Go. Separately, Darwin's native posix_spawn returned
EBADF for a close action on an unopened descriptor. With corrected tests but
the old actions, cycle, repeated-source, closed-source, and identity still
failed at StartProcess. Sparse passed.

Commit 84032b325c9343065746e367119b92ff6bab0600 corrects both issues.
The corrected remapping tests first passed with ordinary Go 1.26.7 on Darwin
arm64 and Go 1.27.0 on Linux arm64. With the new file actions, both the focused
process tests and the full os package pass on both systems.

Darwin validation uses macOS 26.6.2, the released 0.43.0-net.1 compiler,
and Go 1.26.7. A private TINYGOROOT copy contains the new src/os. Its private
SDK assembly also has the added _posix_spawn_file_actions_addopen symbol
stub. The shared release and cache are unchanged. The compiler and runtime
remain from the release and contain the older fcntl correction.

Commands were tinygo test -p 1 -v -run TestForkExec os and
tinygo test -p 1 os, with GOMAXPROCS=2 and external 180-second limits.
Darwin completed the focused tests in 0.590 s and os in 0.795 s. Linux
completed them in 0.017 s and 0.060 s. No remapping cases were skipped.

These are released-compiler source-overlay results. They do not validate a
compiler rebuilt from #5612 and #5636, the combined fork candidate, or a new
downstream full suite. Fork PR #18
owns combined CI. Canceled validation-only
PR #16 runs are not successful validation.

Downstream context

Dispat uses the related hosted process, signal, network, and TLS work. Its
TinyGo acceptance record
describes the downstream integration history. That record is separate from
the focused results for this PR and is not a new downstream full-suite result.

Current Crier evidence

The Crier report separates the original size comparison from a new CI-built candidate test. The original stripped Linux ARM64 result is 13,829,248 bytes versus Go's 30,277,794 bytes. Two render fixtures exceed pixel tolerance.

Candidate e7d34c8c126e0eecd2ce711915f2f88d112d833a, with net 0f460803, passed all 24 fork CI checks. Its downloaded compiler artifacts, with no source overlays, build unchanged Crier 7edaff9. Linux ARM64 E2E passed 143 cases with no failures or skips. Darwin ARM64 passed 142 with no failures and one platform trust-store skip. Darwin startup now works. A separate Darwin local-TLS matrix passes certificate rejection, plaintext refusal, update to a Go 1.1.0 target, and offline rollback. Both platforms pass spawn, signal, cookiejar, os/fcntl, and network probes.

These are combined-candidate results, not proof that this PR alone supplies all features. No new pixel or stripped-size comparison, amd64 execution, or current Dispat acceptance is claimed. WaitDelay and the recorded net limits remain open. No fork release was published.

WaitDelay remains open

The manual test in testdata/os-exec-waitdelay reproduces the reported failure
with the real Go os/exec package. A shell exits while a background sleep
keeps the output pipe open for two seconds.

With a 100 ms WaitDelay, TinyGo returns exec.ErrWaitDelay after 2.004 seconds.
Standard Go returns that error after 101 ms. In a separate pipe test, closing
the reader leaves TinyGo's active read blocked until the writer closes about
two seconds later. Standard Go stops the read immediately.

TinyGo's hosted file layer uses blocking syscall.Read and syscall.Close.
After the timer expires, Cmd.awaitGoroutines closes the pipes and waits for
the copy goroutines. The blocked read prevents that wait from finishing.
The fd mapping fix does not resolve this failure.

A follow-up needs interruptible pipe I/O, coordination between close and active
I/O, and protection against descriptor reuse. It must test normal exit, context
cancellation, blocked reads and writes, and close on Linux and Darwin.
The manual test and its README include commands and measured results.

Dependencies and overlap

Other limits

  • SysProcAttr fields other than Setpgid and Pgid remain unsupported.
  • posix_spawn_file_actions_addchdir_np needs macOS 10.15 or later, while the
    configured deployment target is lower.
  • Process has a four-byte done field on targets that do not use processes.

Related pull requests

Related process work is #4471 and #4377. These related changes and their tests
do not establish that WaitDelay works.

@yohimik

yohimik commented Sep 2, 2026

Copy link
Copy Markdown
Author

Rebased on dev after the 0.42.0 release. The change applies on top of v0.42.0
as released without a conflict, and the diff is unchanged.

Use posix_spawn on hosted Linux and Darwin. Map process files, apply the
working directory and process group, and clear the child signal mask.
Use wait4 for process status and support Kill and Signal.

Mark Darwin pipes close-on-exec under ForkLock. Darwin also needs the
fcntl wrapper in PR tinygo-org#5612 and the libSystem symbols in PR tinygo-org#5636.
Concurrent spawn and pipe creation need the RWMutex fix in PR tinygo-org#5630.

Keep the process stubs on other targets and add process regression tests.
Open /dev/null in the child before closing a nil or unnamed descriptor. Save remap sources before these actions. This avoids a parent descriptor check and its race with concurrent open.

Use descriptor reads and exec redirection in remap tests. Check nil slots and temporary descriptors. The tests pass with ordinary Go on Darwin and Linux. Both TinyGo os suites pass with released compiler source overlays. Darwin also needs the addopen symbol from PR tinygo-org#5636.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant