[Build Data] Run Go build functions through a module-local, cancellable protocol

[cssbruno]

[Build Data] Run Go build functions through a module-local, cancellable protocol

Priority

High — correctness, portability, and compiler execution safety

Context

GOWDK supports imported, same-package, and inline Go functions as a source of deterministic build-time page data. The current runner in internal/buildgen/build_data_runner.go generates a temporary Go source file under the operating-system temporary directory and executes it with:

exec.Command("go", "run", path)

The result is read as JSON from stdout. The command inherits the calling process environment and has no context, deadline, output ceiling, or shared process-lifecycle policy.

This path is distinct from generated application compilation. Issue #663 addresses generated apps being placed in a nested gowdk-generated-app module, but build-data helper execution has its own module-boundary and process-management problems.

Problem

The current runner has several failure modes:

1. Application internal/ packages may be inaccessible. The generated helper source lives outside the application module tree. A build-data function imported from example.com/site/internal/content can therefore fail Go's internal visibility rule even when the .gwdk source and application package are valid members of example.com/site.
2. A build function can block the compiler indefinitely. A function that deadlocks, waits forever, starts a long network request, or launches a child process has no deadline or cancellation path.
3. User stdout conflicts with the result protocol. The runner expects stdout to contain one JSON object. Logging with fmt.Println, a library writing to stdout, or noisy initialization can make a valid function result undecodable.
4. Output is unbounded. A function can emit arbitrarily large stdout or stderr and exhaust memory in the compiler process.
5. Execution provenance is unclear. The effective module, workspace, vendoring, build tags, environment, and GOWDK version are not represented as an explicit execution contract.
6. Process cleanup is platform-dependent. Cancelling only the immediate go run process is insufficient when the Go tool or user code creates descendants.

These are compiler correctness issues, not merely performance improvements.

Goal

Introduce a module-local, versioned, cancellable build-data execution protocol that preserves ordinary Go module semantics and keeps user logging separate from machine-readable results.

Proposed direction

1. Execute from a generated helper package inside the application module

Create or stage the helper beneath a compiler-owned directory inside the project module, for example:

.gowdk/tmp/builddata/<digest>/main.go

The helper should be built and run using the application's normal module context. It must honor:

• application internal/ visibility;
• replace and exclude directives;
• vendoring mode;
• the selected go.work policy;
• build tags, GOOS, GOARCH, and toolchain selection;
• the local-versus-released GOWDK runtime resolution policy.

If temporary source cannot safely live under the project tree, use a temporary workspace/module arrangement that preserves the original module's import semantics explicitly.

2. Replace stdout JSON with a dedicated result channel

The helper should write its response envelope to one of:

• a compiler-created result file opened with restrictive permissions;
• a dedicated inherited file descriptor/handle; or
• a framed local IPC channel.

Stdout and stderr should remain user-observable logs and must not be parsed as the result payload.

A versioned response envelope should include at least:

{
  "protocolVersion": 1,
  "status": "ok",
  "value": {},
  "error": null
}

The protocol should reject unknown incompatible versions with a specific diagnostic.

3. Add context, timeout, and process-tree termination

All build-data calls should accept context.Context and run through the shared subprocess supervisor proposed separately for compiler-owned external commands.

Provide a conservative default timeout and an explicit configuration/CLI override. On cancellation or timeout, terminate the complete process tree on supported Unix and Windows environments and perform deterministic temporary-file cleanup.

4. Bound every channel

Define limits for:

• result payload bytes;
• captured stdout bytes;
• captured stderr bytes;
• helper source/package size where relevant;
• total execution duration.

When logs exceed their limit, retain a bounded prefix/suffix and report truncation rather than allocating indefinitely.

5. Make execution visible

Build reports and --timings output should record:

• function identity and source location;
• module/package used;
• protocol version;
• cache hit/miss when caching is introduced;
• duration;
• timeout/cancellation status;
• bounded log metadata without exposing secrets.

Human diagnostics should distinguish compilation failure, unsupported function signature, helper protocol failure, timeout, cancellation, malformed result, and user-returned error.

6. Avoid signature probing through repeated go run

The current implementation attempts multiple candidate signatures. Prefer resolving supported signatures through the existing Go binding/type inspection phase, then generate and execute exactly one helper invocation. Repeated compilation should not be the signature-discovery mechanism.

Suggested public behavior

Possible configuration shape:

Build: gowdk.BuildConfig{
    DataExecution: gowdk.BuildDataExecution{
        Timeout:       20 * time.Second,
        MaxResultBytes: 2 << 20,
        MaxLogBytes:    256 << 10,
    },
}

Possible CLI override:

gowdk build --build-data-timeout 30s

Exact public naming should follow the configuration-boundary decision from #677 and the shared execution model from #665/#670.

Non-goals

• Providing a security sandbox for trusted local application code.
• Allowing arbitrary remote build execution.
• Making nondeterministic network or filesystem access a recommended build-data pattern.
• Replacing normal Go package/type checking.
• Hiding user-returned build errors.

The initial goal is bounded, correct local execution. Stronger sandboxing may remain an explicit hosted/playground concern.

Test plan

Add focused integration tests covering:

• imported build function in an application-owned internal/ package;
• same-package and inline build functions;
• application replace directive resolution;
• vendored dependency resolution;
• an active go.work file and the documented GOWORK policy;
• user stdout and stderr logging without corrupting the result;
• malformed or oversized result payload;
• oversized stdout and stderr;
• timeout and parent-context cancellation;
• descendant-process termination;
• helper compile failure with source-linked diagnostics;
• user-returned error;
• protocol version mismatch;
• cleanup after success, failure, cancellation, and interruption;
• Windows and Unix process behavior;
• one invocation per already-resolved function signature.

Acceptance criteria

• Build-data helpers execute with application-module import semantics.
• Imported build functions can reside in application-owned internal/ packages.
• Application replace, vendoring, workspace, build-tag, and toolchain policies are honored consistently.
• Every helper invocation is context-aware and has a documented default deadline.
• Cancellation and timeout terminate the complete helper process tree where the platform permits it.
• Machine-readable results do not share stdout with user logs.
• Result, stdout, and stderr sizes are bounded.
• Signature selection comes from Go type inspection rather than repeated trial compilation.
• Build reports expose structured execution outcomes without leaking secrets.
• Failure diagnostics identify the function, package, source location, execution phase, and actionable cause.
• Temporary helper artifacts are cleaned deterministically.
• Tests cover Unix and Windows behavior for the supported CLI path.

Related

• Generate apps inside the project module instead of a nested gowdk-generated-app module #663 — generated application module placement
• Unify gowdk.config.go loading under one explicit execution model #665 — authoritative config execution model
• Separate runtime environment validation from compiler config loading #666 — runtime environment validation separation
• Separate built-in compiler features from executable addon extensions #670 — built-in features versus executable extensions
• Extract shared project compilation orchestration from the CLI #671 — shared project compilation orchestration
• Version and harden the executable addon bridge #672 — versioned executable addon bridge
• Define and enforce the v1 core versus optional product boundaries #677 — v1 core and optional-product boundaries