Managed workflows orchestration (feature proposal)

[lnmunhoz]

Hi @webdevcody,

My idea is to introduce workflow orchestration to Nebula for worktrees.

A workflow could define a sequence of steps inside the .nebula/workflows folder. Nebula would then run a chain of specialized agents, with each agent completing its part before handing the work to the next. A workflow might refine a specification, create an implementation plan, write the code, run tests, review the changes, and open a pull request.

This would also make Nebula useful as an autonomous runtime on a VPS. For example, I could ask Claude to select one of my triaged GitHub issues and run a predefined workflow that strengthens the specification, implements the change, verifies it, reviews the result, and prepares a pull request with minimal supervision.

Nebula already provides much of the runtime needed to support this model, so workflow orchestration feels like a natural extension. I am proposing it as an idea to experiment with, learn from, and evolve based on real-world use.

Please give it a try, skip to the How to test section and use the skill to invoke it. Look at the .nebula folder I created a couple of sample workflows to start with.

To test, from Claude invoke:

/nebula-workflow reviewed "Work on task xyz"

As "reviewed" being the workflow being triggered for this example, defined in the .nebula/workflows folder.

---

Draft for discussion: this is an unfinished prototype, not a request to merge yet. It explores whether NEBULA should coordinate an ordered sequence of AGENTS using its existing WORKTREES, SESSIONS, DAEMON, and SQLITE STORE.

Contents: Why · What to try · Progress preview · How it works · How to test · Risk · Technical overview · Discussion and unfinished work

💡 Why

NEBULA already provides the pieces for working with several AGENTS, but coordinating a planner, implementer, and reviewer still requires manually starting SESSIONS and passing work between them. The idea here is to describe that order once, kick off a task, and let NEBULA manage the handoffs while keeping progress visible.

The proposed MANAGED WORKFLOW creates one WORKTREE for a task and a fresh SESSION for each stage. Run state belongs in the SQLITE STORE; repository files describe the configuration. This keeps execution state with the DAEMON that already owns the SESSIONS and makes a run inspectable after a restart.

This PR is intended to make the idea concrete enough to discuss the ownership, configuration, and user experience before treating any interface as stable.

✨ What to try

• Choose a sequence. The examples include default (planner → implementer), reviewed (planner → implementer → reviewer), and review (reviewer only).
• Compose AGENTS. Named TOML definitions support reusable AGENT files or inline configuration, including provider, MODEL / EFFORT, and instructions. Claude and Codex are supported kinds.
• Start and inspect a run. /nebula-workflow wraps discovery and kickoff. The CLI also exposes catalog, inspect, start, list, status, report, pause, and resume.
• Follow progress. Shift+O opens the WORKFLOWS PANEL with the current step, next step, and completed/remaining counts. The FOOTER shows the running count and opens the panel when clicked.
• Recognize the work. Managed WORKTREES use a ◆ icon and task labels; generated SESSION labels show the stage name. New branch names use the task slug instead of a random run ID.

👀 Progress preview

Excerpt from the recorded isolated TUI test, using a STUB AGENT and the two-stage definition. The PNG remains local and is not hosted in this draft.

WORKTREES · 2        WORKFLOWS · 1            SESSIONS · 1
● main              ◆ Fix login              ● planner claude
◆ Fix login         demo · workflow
                    Running
                    Step 1/2 · planner
                    Next: implementer
                    0/2 done · 2 left

FOOTER: ◆ 1 running · ⇧O

The panel includes unfinished runs across the open WORKSPACE, including paused and blocked runs, followed by the ten newest completed runs. Steps left includes the current unfinished step. Background updates do not move FOCUS or switch the selected SESSION.

🔄 How it works

flowchart TD
    A["Kickoff from an AGENT SESSION on main"] --> B["Resolve and validate workflow + AGENTS"]
    B --> C["DAEMON freezes definition in SQLITE STORE"]
    C --> D["Create one WORKTREE from pinned main commit"]
    D --> E["Launch current stage SESSION"]
    E --> F["Store explicit result and artifact"]
    F --> G{"Result completed AND SESSION FINISHED?"}
    G -- "No" --> H["Wait, or block for intervention"]
    G -- "Yes" --> I{"More stages?"}
    I -- "Yes" --> E
    I -- "No" --> J["Completed; inspect the WORKTREE"]
    C -. "Persisted transitions" .-> K["WORKFLOWS PANEL + FOOTER"]
    F -. "Progress updates" .-> K

Loading

The DAEMON watcher checks runs every two seconds. A SESSION finishing is insufficient by itself: the assigned AGENT must also report an explicit completed result. Each subsequent stage gets the previous results and uses the same WORKTREE. The resolved definition is frozen at kickoff, so editing configuration affects new runs only.

Pause stops scheduling and leaves the current SESSION running. Missing SESSIONS, timeouts, reported blockers, or ambiguous launch recovery block progress for intervention. A DAEMON restart restores scheduling state; it does not automatically resume a provider conversation. There is no automatic commit, merge, or deployment.

🧪 How to test

Use the DAEMON/TUI built from this PR, with its .nebula/workflows, .nebula/agents, and /nebula-workflow skill available in your test PROJECT. The configured AGENTS need working provider authentication and supported MODEL / EFFORT values.

1. Open an AGENT SESSION inside NEBULA, in the PROJECT's ROOT WORKTREE on main.

2. Invoke the skill in that SESSION:

   /nebula-workflow reviewed Add a hello.py script that prints hello and a unittest for it
   

   This selects planner → implementer → reviewer. Use default instead of reviewed for planner → implementer only.

3. Press Shift+O to open the WORKFLOWS PANEL (Ctrl+Q first if the TERMINAL is locked). Follow the current step, next step, and remaining count; respond to provider prompts if requested.

Expected: one new managed WORKTREE, a fresh SESSION for each stage, and progress from 0/3 done · 3 left to completion. Inspect the resulting files in that WORKTREE. The DAEMON handles handoffs after each stage reports its result and finishes.

Already validated at 5f62819: 922 tests passed, plus formatting, clippy, and repository memory/glossary checks. Integration coverage uses STUB AGENTS; real-provider approval and recovery behavior still needs testing.

⚠️ Risk

Verdict: 🔴 Do not merge as-is. This draft introduces autonomous stage scheduling and persistent/protocol changes that need upstream design review and integration work.

Area	Level	Why
Security and production merge	High	The DAEMON can create WORKTREES and launch successive AGENT SESSIONS. Commands share the existing local IPC trust boundary; SESSION IDs do not authenticate against another process running as the same OS user.
Performance	Medium	A two-second watcher and workflow summaries add work. The panel caps displayed completed history, but stored history and snapshot loading still need a retention/scaling decision.
Fit with the codebase	Medium	Scheduling reuses the existing DAEMON, SQLITE STORE, and SESSION machinery, but adds orchestration policy and a fifth panel whose long-term scope is open.

Rollback: reverting the code removes the feature but does not undo created WORKTREES, branches, provider actions, or persisted workflow records. Restart with matching binaries; this draft does not define a database downgrade procedure.

🔧 Technical overview

• Configuration and entry points. Configuration loader, CLI, and kickoff skill. .nebula/workflows/<selector>.toml supplies the stable selector; name is an optional display label. Reusable roles live under .nebula/agents/. Explicit legacy JSON definitions remain readable.
• Persistence and scheduling. Workflow model, store, and watcher. MIGRATION 24 adds run snapshots and unique SESSION associations. Launch intent is checkpointed before external work; ambiguous recovery blocks rather than creating a duplicate SESSION. SQLite replaces the original idea of mutable state files under .nebula.
• Progress UI. Panel and summary/navigation logic. Compact summaries travel in the initial snapshot and persisted-transition updates; full artifacts remain in the store. The branch uses PROTOCOL VERSION 41 and requires matching DAEMON/TUI binaries.
• Further reading and validation. Workflow documentation, configuration examples, handoff integration tests, and TUI integration test. Validation evidence and its limits are listed above.

💬 Discussion and unfinished work

• Ownership and scope. Should the DAEMON own MANAGED WORKFLOW orchestration? Is one shared WORKTREE with fresh sequential SESSIONS the right initial boundary?
• Configuration. Are named TOML workflows plus reusable/inline AGENTS the right authoring model? Which fields and MODEL / EFFORT defaults should become stable?
• Human intervention and review. Define approval, cancellation, recovery, and reviewer rejection behavior before expanding autonomy. The current sequence has no automatic review/fix loop, retry policy, parallel stages, or dependency graph.
• Kickoff and visibility. Decide how ordinary-terminal kickoff and non-main starting points should work, and refine the WORKFLOWS PANEL, history retention, labels, and shortcut.
• Integration and cleanup. GitHub reports merge conflicts against upstream main. This branch has not incorporated its latest changes, including ROOT WORKTREE visibility changes. Resolve those conflicts and reconcile protocol/migration numbering before merge, validate real-provider recovery, and clean up prototype documentation/defaults. The older “Try it” paragraph still says PROTOCOL VERSION 40; the implemented version is 41.

[webdevcody]

Let me spend some time reviewing / thinking about this. I will say with each new modal release, the claude and codex harnesses are able to follow specific .md workflows very well, so I'm debating if adding this into nebula directly introduces more bloat than I'd want, especially since claude code and codex can easily run for hours and follow the provided worked flow.

[lnmunhoz]

@webdevcody I appreciate you taking into consideration.

For context, the inspiration for this came from this video.

I have not yet run these workflows just using Claude or Codex, so my view may change with more experience. However, I see value in having Nebula orchestrate the workflow instead of relying on one agent to manage the entire process within a single conversation.

The main advantages are:

• Each stage can run in a fresh task, reducing context overload and preventing earlier details from distracting the next agent.
• Nebula can enforce the sequence of steps, making the workflow more consistent and predictable.
• Different stages can use different agent harnesses, allowing each task to run in the most suitable environment.
• Progress is easier to inspect because each task has a clear purpose and boundary.

The approach also has trade-offs:

• Passing work between tasks creates coordination overhead.
• Important context may be lost unless each stage produces a clear handoff.
• A fixed workflow may be less flexible when the agent discovers that the original plan needs to change.
• Debugging failures across several tasks may be harder than inspecting one continuous session.

I agree that a single coordinating agent could manage the full workflow. My concern is that an agent may eventually skip a step, deviate from the process, or stop early. Encoding the workflow in Nebula provides stronger guarantees while preserving the option to choose a different harness for each stage.