Skip to content

feat: connect orchestrator and frontend#1564

Open
Scra3 wants to merge 155 commits intomainfrom
feat/prd-214-server-step-mapper
Open

feat: connect orchestrator and frontend#1564
Scra3 wants to merge 155 commits intomainfrom
feat/prd-214-server-step-mapper

Conversation

@Scra3
Copy link
Copy Markdown
Member

@Scra3 Scra3 commented Apr 20, 2026

Definition of Done

General

  • Write an explicit title for the Pull Request, following Conventional Commits specification
  • Test manually the implemented changes
  • Validate the code quality (indentation, syntax, style, simplicity, readability)

Security

  • Consider the security impact of the changes made

Note

Connect workflow orchestrator to frontend via a new workflow-executor package and agent proxy routes

  • Introduces the packages/workflow-executor package — a standalone Node.js service that polls the Forest server for pending workflow runs and executes each step via a set of typed step executors (Condition, ReadRecord, UpdateRecord, TriggerAction, LoadRelatedRecord, Mcp, Guidance).
  • Each executor is backed by a common BaseStepExecutor providing idempotency checks, activity logging, timeout handling, and unified error-to-outcome mapping; a StepExecutorFactory selects the right implementation per step type.
  • Adds an HTTP server (ExecutorHttpServer) with JWT auth, exposing GET /health, GET /runs/:runId, and POST /runs/:runId/trigger so the frontend can read step state and submit pending data.
  • Adds WorkflowExecutorProxyRoute to the agent, exposing GET/_internal/workflow-executions/:runId and POST /_internal/workflow-executions/:runId/trigger that forward requests (including auth headers) to the executor URL configured via the new workflowExecutorUrl agent option.
  • Persistence is handled by pluggable RunStore implementations: an InMemoryStore for development and a Sequelize-backed DatabaseStore (SQLite/Postgres) with Umzug migrations for production.
  • Risk: enabling workflowExecutorUrl on the agent adds unauthenticated-to-executor proxy routes; the executor relies on JWT validation of the forwarded Authorization/Cookie headers for security.

Macroscope summarized d88b561.

matthv and others added 30 commits March 17, 2026 15:00
…premature deps, add smoke test

- Rewrite CLAUDE.md with project overview and architecture principles, remove changelog
- Remove unused dependencies (ai-proxy, sequelize, zod) per YAGNI
- Add smoke test so CI passes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… document system architecture

- Lint now covers src and test directories
- Replace require() with import, use stronger assertion (toHaveLength)
- Add System Architecture section describing Front/Orchestrator/Executor/Agent
- Mark Architecture Principles as planned (not yet implemented)
- Remove redundant test/.gitkeep
- Make index.ts a valid module with export {}

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erver (#1504)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: alban bertolini <albanb@forestadmin.com>
…ain (#1512)

Co-authored-by: alban bertolini <albanb@forestadmin.com>
- Remove McpClient.tools property, loadTools() returns local array
- Rename closeConnections() → dispose()
- Rename testConnections() → checkConnection()
- Add McpServers type export
- Rename mcpServerConfigs → toolConfigs in create-ai-provider
- Update all tests accordingly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…eability

Add sourceId to McpToolRef so that persisted execution data (pendingData,
executionParams) tracks which MCP server provided the tool. This fixes
tool lookup on re-entry (confirmation flow) when multiple servers expose
tools with the same name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… collection schema

- Replace non-null assertion with explicit McpToolNotFoundError when AI
  selects a tool name that doesn't match any available tool
- Resolve related collection name from parent schema before looking up
  the related schema in cache, fixing cases where relation name differs
  from target collection name

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge main into feature branch, resolve conflicts by taking main's
versions of router.ts, create-ai-provider.ts and their tests.
Remove deleted mcp-config-checker.ts. Bump workflow-executor internal
deps to match main.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
import PrettyLogger from './adapters/pretty-logger';
import { ConfigurationError, extractErrorMessage } from './errors';

const POSITIVE_INT = z.coerce.number().int().positive();
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium src/cli-core.ts:17

Setting MAX_CHAIN_DEPTH=0 to disable chaining throws a ConfigurationError because parsePositiveIntEnv uses z.number().positive(), which rejects zero. The ExecutorOptions interface comment explicitly says "0 disables chaining", so users following the documentation cannot actually disable chaining. Consider changing the validator to allow zero.

-const POSITIVE_INT = z.coerce.number().int().positive();
+const NON_NEGATIVE_INT = z.coerce.number().int().nonnegative();
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/workflow-executor/src/cli-core.ts around line 17:

Setting `MAX_CHAIN_DEPTH=0` to disable chaining throws a `ConfigurationError` because `parsePositiveIntEnv` uses `z.number().positive()`, which rejects zero. The `ExecutorOptions` interface comment explicitly says "0 disables chaining", so users following the documentation cannot actually disable chaining. Consider changing the validator to allow zero.

Evidence trail:
packages/workflow-executor/src/cli-core.ts line 17: `const POSITIVE_INT = z.coerce.number().int().positive();`
packages/workflow-executor/src/cli-core.ts line 153: `maxChainDepth: parsePositiveIntEnv('MAX_CHAIN_DEPTH', env.MAX_CHAIN_DEPTH),`
packages/workflow-executor/src/build-workflow-executor.ts line 43: `// Max auto-chained steps per entry (see RunnerConfig.maxChainDepth). 0 disables chaining.`
packages/workflow-executor/src/runner.ts line 55: `// next poll cycle (counted after the initial step). 0 disables chaining entirely. Default 50.`

constructor(services: ForestAdminHttpDriverServices, options: AgentOptionsWithDefaults) {
super(services, options);
// Remove trailing slash for clean URL joining
this.executorUrl = new URL(options.workflowExecutorUrl.replace(/\/+$/, ''));
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low workflow/workflow-executor-proxy.ts:24

When workflowExecutorUrl includes a path component (e.g., http://executor.com/api/v1), the path is silently discarded. At line 40, new URL(executorRelativeUrl, this.executorUrl) receives executorRelativeUrl starting with /runs/..., and the URL constructor treats absolute paths as relative to the origin, not the base path. So new URL('/runs/123', 'http://executor.com/api/v1') yields http://executor.com/runs/123, dropping /api/v1.

-    // Remove trailing slash for clean URL joining
-    this.executorUrl = new URL(options.workflowExecutorUrl.replace(/\/+$/, ''));
+    this.executorUrl = new URL(options.workflowExecutorUrl);
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/agent/src/routes/workflow/workflow-executor-proxy.ts around line 24:

When `workflowExecutorUrl` includes a path component (e.g., `http://executor.com/api/v1`), the path is silently discarded. At line 40, `new URL(executorRelativeUrl, this.executorUrl)` receives `executorRelativeUrl` starting with `/runs/...`, and the URL constructor treats absolute paths as relative to the origin, not the base path. So `new URL('/runs/123', 'http://executor.com/api/v1')` yields `http://executor.com/runs/123`, dropping `/api/v1`.

Evidence trail:
packages/agent/src/routes/workflow/workflow-executor-proxy.ts lines 24 and 39-42 (REVIEWED_COMMIT): constructor strips trailing slashes from executorUrl, then `new URL(executorRelativeUrl, this.executorUrl)` is called with `executorRelativeUrl` starting with `/runs/...`. WHATWG URL spec: https://url.spec.whatwg.org/#dom-url-url — absolute-path references resolve against the origin, not the base path. packages/agent/src/types.ts:54 — `workflowExecutorUrl?: string | null` with no path restriction. packages/agent/test/routes/workflow/workflow-executor-proxy.test.ts — all tests use URLs without path components (e.g., `http://localhost:${executorPort}`).

alban bertolini and others added 17 commits April 29, 2026 10:55
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a single command to start the agent and workflow executor together
in dev. The executor starts 5s after the agent to ensure it is ready.
Logs are prefixed [agent]/[executor] via concurrently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auto-reloads the executor on source changes without needing a manual rebuild.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…collision

EXECUTOR_AGENT_URL and EXECUTOR_DATABASE_URL replace AGENT_URL and DATABASE_URL
in the executor CLI to avoid conflicts with the agent's own env vars in deployments
where both processes share an environment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…XECUTOR_ prefix)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or executor

Inline VAR=value command syntax is not reliably inherited by tsx watch child
processes. Using set -a + assignment ensures the vars are properly exported.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sh, not yarn

yarn runs scripts via sh -c, which expands $EXECUTOR_AGENT_URL before bash
even sources .env. Escaping with \$ defers expansion to the inner bash subshell.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ecutor start

Polls EXECUTOR_AGENT_URL every second with curl until the agent responds,
instead of relying on a fixed sleep.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n runner

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eful shutdown

Without exec, concurrently sends SIGTERM to bash which exits without forwarding
the signal to tsx. exec replaces bash with tsx so the signal handler in
build-workflow-executor.ts can run stop() cleanly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ndling

tsx watch spawns a child process and intercepts signals before they reach
the Node.js handler, so shutdown logs are never emitted. Plain tsx runs
the script in-process so SIGTERM/SIGINT reach the onSignal handler cleanly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove duplicate AGENT_URL/DATABASE_URL block, keep EXECUTOR_ prefixed vars
and add WORKFLOW_EXECUTOR_URL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ing into proxy path

context.url includes the /forest prefix when mounted via Koa Router (prefix is
prepended to nested routes, not stripped). Reconstructing from context.params.runId
and context.querystring is prefix-agnostic and works across all frameworks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…fix leaking into proxy path"

This reverts commit 5d112f8.
…x in proxy path

context.url includes /forest when mounted via Koa Router. Reconstructing from
context.params.runId and context.querystring is prefix-agnostic across all frameworks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread packages/agent/src/routes/workflow/workflow-executor-proxy.ts
alban bertolini and others added 7 commits April 30, 2026 11:43
…n schemas

Remove .strict() from all StepDefinition variant schemas so fields sent
by the orchestrator but not declared locally (e.g. automaticExecution on
condition/guidance steps) are silently stripped instead of throwing.

AvailableStepExecutionSchema and the execution-level schemas remain
strict to guarantee the mapper output is well-formed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the user completes an awaiting-input step manually from the frontend,
StepSummaryBuilder now tells the AI that the proposed action may not
reflect what was actually done, instead of presenting the executor's
proposal as fact.

Detection: pendingData exists + idempotencyPhase is undefined (side
effect never started) + stepOutcome.status is 'success'.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed signal

Replace idempotencyPhase === undefined with executionResult === undefined
in StepSummaryBuilder. The previous check produced false positives for
non-mutating steps (load-related-record, skipped paths, trigger-action via
saveFrontendResult) that never set idempotencyPhase but do set executionResult.

Any normal executor completion always writes executionResult before reporting
the step as done, making its absence the correct invariant.

Add regression tests covering all false-positive paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lder test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cordStepExecutor

getAvailableRecordRefs, selectRecordRef, toRecordIdentifier, getCollectionSchema,
and findField were only used by the 4 record executors that already extend
RecordStepExecutor. Moving them keeps BaseStepExecutor generic and confines
record-domain logic to the correct layer.

Also extract idempotencyPhase into MutatingStepExecutionData so non-mutating
step types (condition, read-record, guidance, load-related-record) no longer
carry a field that can never be set on them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…TCH body

The server validator (Joi) only accepts { status } — extra fields cause a
400 ValidationFailedError. The error message is still logged locally via
stepErrorMessage on failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nt camelCase deserialization

The JSON:API deserializer in agent-client converts all attribute keys to camelCase
(card_number → cardNumber). ReadRecordStepExecutor looks up values by original field name,
so snake_case fields came back undefined and were stripped from executionResult.

Add restoreFieldNames() in AgentClientAgentPort.getRecord to reverse the camelCase
mapping using the original field names from the query, ensuring executors receive
values keyed by the field name they requested.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread packages/workflow-executor/src/adapters/agent-client-agent-port.ts Outdated
alban bertolini and others added 3 commits April 30, 2026 16:41
…d and values

Same camelCase deserialization issue as getRecord: apply restoreFieldNames to
related records using both primaryKeyFields and requested fields so that
extractRecordId finds the correct PK values and callers receive snake_case keys.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lues

Same camelCase deserialization issue as getRecord/getRelatedData: agent-client's
JSON:API deserializer converts response keys to camelCase. Restore original names
using the input values keys, which are already in the caller's original format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants