Experiment: Restructure project and flatten dependency graph - #396
Draft
hidde-jan wants to merge 52 commits into
Draft
Experiment: Restructure project and flatten dependency graph#396hidde-jan wants to merge 52 commits into
hidde-jan wants to merge 52 commits into
Conversation
Drop the MQTT-based Fin capability, its controller, and the associated message/protocol models, along with the ENABLE_FINS wiring, mocks, and tests. See docs/adr/FIN-EXTERNAL-INTEGRATION-ANALYSIS.md for why: a push-based, broker-mediated protocol turned out to be fragile and hard to reason about for job delivery/ack semantics, retries, and job ownership. No transitional adaptation of the MQTT capability to the stepwise Context is kept here since it's being deleted outright - only the final, MQTT-free state matters for review. Also drops the paho.mqtt.golang dependency and the mosquitto broker config/deployment (docker-compose, Dockerfile config) that only existed to support it. The replacement pull-based HTTP/JSON protocol is proposed in docs/adr/FIN-WEBHOOK-PROTOCOL-PROPOSAL.md and implemented in a later, separate series of commits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the MQTT-based Fin protocol/setup docs and the MQTT Fin example playbook, and finish scrubbing stale MQTT references from the remaining docs (README, concepts, core-components, getting-started, installation-configuration, soarca-extensions). Fin documentation for the new HTTP/JSON protocol will be added alongside its implementation in a later commit series. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move capability execution from a pairwise (command x target) model to a stepwise one, and fix capability routing to use the CACAO-correct identifier field. Stepwise dispatch: - Replace capability.Context's singular Command/Target/Authentication fields with Commands []cacao.Command and Targets []ResolvedTarget (a target paired with its own resolved auth info, since different targets in a step may legitimately use different authentication). - The action executor now calls a capability once per step, passing the step's full commands[]/targets[] arrays, instead of once per (command, target) pair. This fixes the case where a step has zero targets: previously nothing ran at all (the pairwise loop had nothing to iterate), which silently violated CACAO's single-outcome-per-step model; now a step always produces exactly one outcome. - Adapt every native capability (ssh, http, powershell, openc2, manual) to the new Context shape, each internalizing its own commands/targets iteration. ssh, http, powershell, and openc2 all perform commands against a target, so a step declaring zero targets for one of these has nothing to run against: they skip execution (no commands run, empty result, nil error) rather than fabricating a dummy target, matching prior single-target behavior for the common case. manual is target-independent (a human ticket resolved once per step, not per target) and is unaffected by target count. Routing by agent.Type: - CACAO documents agent.type as the identifier field and agent.name as display-only, but the executor routed on agent.Name (executor.capabilities[agent.Name]) and never read Type at all. Fix: route strictly on agent.Type. Each capability keeps registering under its own distinct type value (soarca-ssh, soarca-http-api, soarca-openc2-http, soarca-powershell, soarca-manual) - a step routes to exactly one capability via its agent's type, matching the chosen one-command-type-per-step granularity rather than introducing a second, internal dispatch-by-commands[].type mechanism alongside it. - Fix examples/manual-playbook.json, which had drifted to an unused second agent definition. Update executer.md to describe the corrected routing model and clarify that `name` is a free-text label with no role in routing. This lands ahead of the new HTTP/JSON Fin protocol, which needs both: one call per step (matching planned Fin job granularity), and agent type to be a meaningful, real routing key so a Fin capability pool can be addressed by its own distinct type value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
StepId is stable across re-executions of the same step (while-loops, cyclic on_completion graphs), but the reporting cache and public API keyed step results by StepId alone. A second start-report for the same StepId was rejected and dropped (fire-and-forget IStepReporter calls), silently losing data whenever a step ran more than once in the same execution. Add StepExecutionId (a fresh UUID minted once per step invocation) to execution.Metadata. The decomposer's new newStepMetadata() helper is the single mint point, called for every step dispatch including each while-loop iteration. Thread the field through: - IStepReporter / IDownStreamReporter: ReportStepStart/ReportStepEnd now take the full execution.Metadata instead of a bare executionId. - Reporting cache (downstream_reporter/cache): StepResults is now keyed by StepExecutionId instead of StepId, so re-executed steps get their own entry instead of colliding. - Public API (StepExecutionReport / reporter_parser): exposes the new field and the corrected keying. - TheHive integration: updated for interface conformance only (no deeper use of StepExecutionId in TheHive's own data model yet). The manual capability's interaction registry deliberately keeps keying pending commands on (ExecutionId, StepId) rather than StepExecutionId -- one pending manual command per step is intentional and unrelated to this cache bug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New pkg/core/registry provides Registry[V], a small, generic, concurrency-safe, two-level (outerKey -> innerKey -> value) in-memory store: Register/Get/Remove/List, with automatic pruning of an outer key once it has no remaining inner entries. This generalizes the register/lookup/expire pattern already used by the manual-command interaction queue (keyed by execution id -> step id), so the next commit can refactor that queue to use it instead of its own hand-rolled nested map, and the future Fin job queue (keyed by capability type -> job id) can reuse it too instead of duplicating the same bookkeeping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-execution race, switch continue to PUT
Implements the manual capability changes specified in
docs/adr/EXECUTION-MODEL.md: fixes the pending-interaction race between
Execute() and its async cleanup goroutine, rekeys the pending-interaction
registry from (ExecutionId, StepId) to (ExecutionId, StepExecutionId),
and replaces the generic POST /manual/continue endpoint with a PUT on
the specific pending-command resource.
Re-execution race fix:
- ManualCapability.Execute() registered a pending interaction and
blocked on user input, while the entry's removal happened in a
separate goroutine reacting to ctx.Done(), with nothing guaranteeing
removal completed before Execute() returned. A step re-executed
quickly after the previous invocation returned (e.g. the next
iteration of a while-condition loop body) could collide with its own
predecessor's not-yet-finished cleanup. Fixed by adding a synchronous
Deregister() call right after the blocking wait resolves, keeping the
async cleanup goroutine as an idempotent backstop.
StepExecutionId rekeying:
- The registry's "one pending manual command per StepId" behaviour was
an accident of implementation - StepId was simply the only
per-invocation identifier available - not a deliberate design
constraint. It broke down as soon as a step could legitimately be
invoked more than once with independently-pending state: sequential
re-executions (while-condition loops, cyclic on_completion graphs)
or, once implemented, parallel branches converging on the same step.
- pkg/models/api/manual.go: added StepExecutionId to
InteractionCommandData (StepId kept for display).
- pkg/core/capability/manual/interaction/interaction.go: register,
look up, and remove pending commands by StepExecutionId instead of
StepId.
- New test proving two pending commands with the same StepId but
different StepExecutionId coexist and resolve independently.
POST /manual/continue -> PUT /manual/{exec_id}/{step_execution_id}:
- Resolving a pending command is now a PUT on the same resource GET
identifies, instead of a generic RPC-style endpoint duplicating its
own ids in the body. ManualOutArgsUpdatePayload drops
ExecutionId/PlaybookId/StepId/StepExecutionId entirely - the path is
the sole source of truth for which command is being resolved.
- pkg/api/api.go: route changed accordingly.
- pkg/api/manual/manual_api.go: PostContinue renamed to PutContinue;
parses exec_id/step_execution_id from the path (mirroring
GetPendingCommand), looks up the pending command first (clean 404 for
an unknown resource, and supplies PlaybookId for the response without
needing it in the body), then hands the full, looked-up
execution.Metadata to the storage layer's PostContinue.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ndpoint
Update the hand-written manual API reference and regenerate Swagger
docs (`make swagger swagger-docs`) to reflect the previous commit's
manual capability changes:
- docs/content/en/docs/core-components/api-manual.md: describes
GET/PUT /manual/{exec_id}/{step_execution_id} (replacing
{step-id} and POST /manual/continue), the smaller PUT request body,
and why a pending command may share a step_id across invocations.
- docs/content/en/docs/core-components/modules.md: updates the
protocol diagram to the same routes (also fixes a pre-existing
{step-id} staleness left over from the earlier StepExecutionId work).
- docs/static/openapi/{swagger.json,swagger.yaml}: regenerated from the
updated Swagger annotations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce pkg/models/fin, the data models for the new HTTP/JSON pull-based Fin protocol from docs/adr/FIN-WEBHOOK-PROTOCOL-PROPOSAL.md, replacing the MQTT-based one removed earlier on this branch. This is models only (no registry, queue, API, or capability wiring yet) — those follow in later commits. - Capability: one executable capability a Fin declares at registration (Type is the routing key playbook authors reference; Description/ Version/StepExample are display/documentation only). - Record: the persisted (MongoDB-backed) view of a registered Fin — FinId, a hash of its fin_token (never the plaintext credential), declared Capabilities, and RegisteredAt/LastSeen. Fin registrations are meant to survive a SOARCA restart, unlike the job queue (planned in-memory, since execution state itself is not persisted/resumed today either). - Job/StepInfo/Command/Target: the poll-response payload a Fin receives — one job per step, commands[]/targets[] as arrays (0, 1, or many targets), each target carrying its own resolved authentication. Adds StepExecutionId (not in the original proposal draft) so Fin jobs key consistently with the rest of the execution model. - JobResult/TargetResult/JobState: the result a Fin submits — a single aggregated (state, variables) pair that SOARCA's step machinery consumes, plus optional per-target diagnostic detail for reporting/audit only. - RegisterRequest/Response, PollRequest/Response, ResultRequest, StatusPingRequest/Response, ListResponse: the wire-level request/ response shapes for the register/poll/result/status-ping/list endpoints, matching the proposal's documented JSON payloads. - Sentinel error types for the auth/lookup failure modes the protocol needs to distinguish (invalid registration/fin token, unknown fin/job, job not leased to the calling fin). Includes JSON-shape regression tests pinning the wire contract against the documented protocol payloads.
Introduce the two data-holding services the new HTTP/JSON Fin protocol needs, per the design decision that Fin registrations persist across a SOARCA restart but the job queue does not (matching the fact that in-flight execution state itself isn't persisted/resumed either): - internal/database/fin: IFinRepository, the persistence contract for fin.Record (register/get/find-by-token-hash/list/touch/unregister), implemented against the existing generic database.Database interface so it works unmodified against MongoDB. Extends internal/database/mongodb's dbtypes union and adds a fin_registry_collection alongside the existing playbook collection. - internal/database/finmemory: a non-persisted fallback implementation of the same interface for running without a configured database, mirroring internal/database/memory's existing playbook fallback. - pkg/core/capability/fin/queue: the in-memory Fin job queue. A future FinCapability will Enqueue one Job per step invocation and block (bounded by the step's own deadline) for a result; independently polling Fin processes registered under the job's CapabilityType Claim it via long-poll, Submit a result, or ExtendLease via a status ping. Expired leases (a Fin that stopped polling/responding) are swept and requeued in the background so another Fin registered under the same type can pick the job up. No HTTP endpoints or capability wiring yet - those follow in later commits. All new code has unit test coverage (the repository tests use a hand-rolled database.Database test double rather than a live MongoDB, consistent with there being no existing unit tests of the MongoDB-backed playbook repository either).
Implement the REST surface of the new pull-based Fin protocol on top
of the registry/queue services added previously, wiring pkg/models/fin
directly as the request/response wire types (no separate api-package
DTOs needed - that package was already designed as the wire format):
- POST /fin/register registration-token gated; issues a
new fin_id/fin_token pair
- POST /fin/poll fin-token gated; long-polls the job
queue for a job matching the calling
Fin's registered capability types,
204 No Content on timeout
- PUT /fin/jobs/:job_id fin-token gated; submits a job's
result, only if leased to the caller
- PATCH /fin/jobs/:job_id/status fin-token gated; extends a job's
lease (status-ping liveness/lease
renewal, §2.4/§2.5); the response
shape has room for a future
cancellation instruction, always
empty for now since cancellation
itself isn't implemented yet
- DELETE /fin/:fin_id fin-token gated; a Fin may only
delete its own registration
- GET /fin/, GET /fin/:fin_id admin/dashboard discovery reads,
not Fin-authenticated
Adds pkg/core/capability/fin/token for fin_token generation/hashing/
constant-time comparison (registration tokens and fin_tokens are only
ever compared/looked-up by hash, never stored or logged in plaintext).
RequireFinToken is a route-group-scoped gin middleware (not global),
resolving the Authorization: Bearer token to its fin.Record via
IFinRepository.FindByTokenHash and making it available to handlers -
deliberately kept separate from the existing JWT-role-based gauth
middleware used for the rest of the API, since Fins authenticate with
their own shared-secret scheme, not user JWTs. Wiring this handler and
its config (FIN_REGISTRATION_TOKEN, poll/lease defaults) into
controller.go is a later stage.
Regenerates docs/static/openapi/swagger.json/.yaml for the new
annotated endpoints/types.
Full unit test coverage over the handlers (registration, auth
middleware, poll/claim/submit/status-ping round trips and their
failure modes, unregister ownership check, list/get) using the
in-memory repository fallback and the real in-memory queue.
Implements capability.ICapability on top of the Fin job queue: Execute converts a step invocation into one fin.Job (commands, targets+auth, step metadata, a lease sized off the step's own timeout) and blocks on queue.Queue.Enqueue until a claiming Fin submits a result or the deadline elapses. A JobResult with State == JobStateFailure is turned into a Go error so it still flows through the executor's normal on_completion/on_failure branching. Fin capability types are declared dynamically at Fin registration time, so they can't be pre-populated into the executor's static capabilities map the way builtins are. Rather than complicating that map, Executor grows an optional finFallback capability (Executor.SetFinFallback), consulted whenever agent.Type doesn't match any statically-registered capability - FinCapability is wired as that fallback, and reads capability.Context's new Agent field to know which capability type to route the job under. Passing no fallback (the default) preserves today's exact "capability ... is not available in soarca" error behavior, so existing callers are unaffected until Fin support is actually wired up in the next stage.
…tes) - Controller gains a finRepo, toggled between Mongo (SetupFinRepository) and in-memory (finmemory.New()) alongside the existing playbookRepo, using the same DATABASE env var. - A package-level mainFinQueue (queue.Queue) is shared across every action.Executor built by NewDecomposer() (one per execution) and the Fin API handlers, so all Fin jobs land in one queue regardless of which execution enqueued them. - NewDecomposer() wires fincapability.New(mainFinQueue, guid) into the action executor via SetFinFallback, so any agent.Type not matching a built-in capability routes to a registered Fin declaring that type. - initializeCore() reorders setupDatabase() before intializeAuthenticationMiddleware() and registers routes.FinPublic before the admin auth middleware (Fins authenticate via fin_token, not JWT) with a warning comment against reordering; routes.FinAdmin is registered alongside the rest of the admin-gated API. - New FIN_REGISTRATION_TOKEN/FIN_POLL_INTERVAL_SECONDS/ FIN_LONG_POLL_TIMEOUT_SECONDS/FIN_JOB_LEASE_SECONDS env vars replace the removed ENABLE_FINS/MQTT_BROKER/MQTT_PORT entries in .env.example. - Verified end-to-end with a real running binary (in-memory DB): POST /fin/register -> 201 with fin_id/fin_token/poll config, GET /fin (admin) lists the registered Fin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- New docs/content/en/docs/soarca-extensions/fin-protocol.md replaces the removed MQTT-based protocol page: endpoints table, auth model (registration token vs fin_token vs admin JWT), registration/poll/ result/status-ping/unregister/discovery walkthroughs with example payloads, lease/retry semantics, and a sequence diagram. Links from soarca-extensions/_index.md. - New examples/fin-playbook.json: a worked playbook combining a native SSH capability step with a step routed to an externally-registered Fin capability (agent type custom-ssh-fin), demonstrating how the two coexist in one playbook. - Verified end-to-end against a live running binary: registered a fin, submitted examples/fin-playbook.json, triggered it, polled and claimed the Fin-routed job, submitted a result, and confirmed both the step and execution report as successfully_executed. - Fixed two small pre-existing inaccuracies noticed while documenting the protocol: pkg/models/fin/protocol.go doc comments referenced stale /api/v1/fins/* paths instead of the actual registered /fin/* routes, and fin.Record.LastSeen's doc comment overclaimed active TTL-based expiry from routing that isn't implemented - stale Fins are simply never polling, so jobs queued under their capability types go unclaimed until the step's own timeout elapses (the job lease mechanism, not LastSeen, is what actually bounds that wait). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New pkg/api/fin/fin_e2e_test.go (package fin_test, avoiding an import
cycle) wires the real fincapability.Capability, a real queue.Queue, and
the real FinHandler HTTP routes together the same way
internal/controller/controller.go wires them in production, with no
mocks standing in for either side of the protocol.
TestFullFinProtocolFlow drives the whole lifecycle this protocol exists
for: register a fin declaring a capability type -> a concurrently
running Execute() call enqueues a job under that type -> POST /fin/poll
claims it -> PUT /fin/jobs/{job_id} submits a result -> Execute()
returns the submitted variables to the (simulated) step machinery.
This is the automated regression-test counterpart of the manual
register -> trigger -> poll -> result -> report smoke test run earlier
against a live binary with examples/fin-playbook.json.
Existing coverage this builds on was already solid going in (queue
98%, fin capability 92.6%, api/fin 73.5%, finmemory 100%) - this adds
the one thing that was previously only verified manually: the full
stack working together end-to-end, not just each layer in isolation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- DELETE /fin/ (fin-token gated) now unregisters the calling fin itself -
the fin_id is inferred from the token, so it's never sent in the path
and there's no ownership check to get wrong.
- DELETE /fin/{fin_id} is repurposed as an admin/dashboard action (same
auth as GET /fin/ and GET /fin/{fin_id}) to forcibly remove any fin's
registration, e.g. one that's stale/offline and will never unregister
itself.
- Updated fin-protocol.md and route/handler doc comments accordingly.
- go.mod/go.sum: incidental `go mod tidy` cleanup (drops an unused
testcontainers-go dependency added but never wired up in an earlier,
since-abandoned exploration).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously a step routed to the Fin fallback capability always waited
out its full step timeout (or a 1-minute default) even when no Fin had
ever registered for its capability type, or every registered Fin had
gone stale. Both cases are knowable up front, so there is no reason to
wait:
- fincapability.Capability gains an IFinRepository dependency; before
enqueuing a job it checks whether any registered Fin declares the
step's capability type, and whether at least one of those is "live"
(LastSeen within a staleAfter threshold, default 2x
FIN_LONG_POLL_TIMEOUT_SECONDS). If not, it returns
fin.ErrNoCapableFin / fin.ErrOnlyStaleCapableFins immediately instead
of enqueuing. A nil repository (or a failing List() call) fails open
to the previous wait-based behaviour, so this is purely additive.
- fin.Record gains a computed, non-persisted Stale bool, populated by
GET /fin/ and GET /fin/{fin_id} using the same staleness threshold,
so GUIs/operators can flag Fins that are registered but likely no
longer polling without SOARCA needing to actively expire them.
Wired into internal/controller/controller.go: NewDecomposer passes
controller.finRepo/soarcaTime/staleAfter into fincapability.New, and
newFinHandler computes the matching StaleAfterSeconds for
fin_handler.Config so both checks agree on the same threshold.
Swagger docs regenerated for the new Stale field.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The LastSeen doc comment and §2.4 of FIN-WEBHOOK-PROTOCOL-PROPOSAL.md
still described the original proposal (stale Fins auto-expired from the
routing table via TTL), which is not what got implemented: registrations
are never removed automatically. Update both to describe what's actually
in place - stale Fins are flagged (Record.Stale / GET /fin[/{fin_id}])
for operator visibility, and fincapability.Capability's fail-fast check
uses the same staleness threshold to refuse new jobs when no live Fin
could possibly claim them, rather than an automatic reaper.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fin.ErrRegistrationTokenInvalid has existed since the Fin protocol's data models were first defined but was never actually constructed anywhere - Register() used a plain string literal instead. Construct and return it now, matching the pattern used for the fin_token/job-related sentinel errors elsewhere in this package (ErrFinTokenInvalid, ErrJobNotFound, etc.), and use its Error() text for the API response instead of duplicating the message as a separate literal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The unregister/delete split (DELETE /fin/ vs DELETE /fin/{fin_id}) and the
LastSeen staleness doc-text fix had updated Go source annotations but
swagger.json/yaml were never regenerated with make swagger-docs. Sweep
those up now. Also includes the command_b64 field being added to
fin.Command in the following commit (make swagger-docs was run once,
covering both).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cacao.Command.CommandB64 (used by the built-in powershell capability as a base64 alternative to Command for complex/quoted scripts) was silently dropped when building fin.Command for a Job - a Fin-based capability relying on command_b64 would receive nothing. Add the field to fin.Command and forward it in toFinCommands. Also fix a stale doc comment that claimed Type/Content/Headers were dropped when the struct already carried them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t poll Previously only /poll updated a Fin's LastSeen. A Fin with limited concurrency running a long job (extending its lease via periodic status pings per §2.5, not polling again until it's free) would go stale by that measure alone - wrongly triggering fin.Capability's fail-fast rejection of new jobs for its capability type (ErrOnlyStaleCapableFins) and showing as stale in List/Get, even though it is actively alive and working. SubmitResult and StatusPing now also call repository.Touch(), tolerating failure the same way Poll already does (log and continue - liveness bookkeeping must never block the Fin-facing response). Updated LastSeen's doc comment on fin.Record to describe all three touch points. Regenerated swagger docs for the comment change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Manual and Fin are the only capabilities that enforce cacao.Step.Timeout as a hard wall-clock deadline (SSH/HTTP/OpenC2/PowerShell don't enforce it at all today - a separate, pre-existing gap, out of scope here). Both independently hardcoded a 1-minute fallback for when a step omits timeout entirely. 1 minute is unrealistically short as a deadline for what these two capabilities actually do: a human-in-the-loop Manual approval, or a Fin job that may involve real external work (e.g. multi-target SSH). The CACAO spec does not mandate this fallback value - its __ACTION_TIMEOUT__ constant (60000ms/1 minute) is merely a default value playbook authors may optionally assign to a step's own timeout, not a mandated platform-level fallback for when the property is omitted altogether; the spec only says implementations SHOULD consider implementing a maximum allowed timeout, without prescribing a number. We considered treating timeout as an inactivity/liveness signal instead (resettable by Fin status pings) given the spec's "used to determine when a step is no longer responsive" wording, but settled on keeping it a simple hard deadline - simpler to implement/reason about, and matches precedent (e.g. GitLab CI job timeouts). This keeps StatusPing's existing role unchanged (queue-side claim lease only, not the overall wait). Added pkg/utils.DefaultStepTimeout(), reading the new DEFAULT_STEP_TIMEOUT_SECONDS env var (default 600s/10 minutes), and switched both manual.go and fin_capability.go to call it instead of their own separate hardcoded constants - one shared, configurable default instead of two independent ones. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduces a clean greenfield storage package to replace the generic
internal/database layer with strongly-typed interfaces:
- PlaybookStore: Create, Update, Get, Delete, List, ListMeta
- FinStore: Create, Get, GetByTokenHash, List, Touch, Delete
- Sentinel errors: ErrNotFound, ErrConflict
Memory backend:
- sync.RWMutex-backed maps
- Full compliance with all interface methods
- 15 passing tests covering all CRUD operations
MongoDB backend:
- URI-only config (database name parsed from URI)
- No hardcoded credentials or collection names
- Configurable connect timeout
- Integration tests (skipped by default, ready to run with live MongoDB)
- Proper error mapping to sentinel errors
All interfaces and implementations tested and verified to compile.
Use connstring package to extract database from URI
Parse the URI using the official driver connstring package
instead of manual net/url parsing. This ensures we extract
the database name the same way the driver does.
Simplify MongoDB config to URI only
- Remove ConnectTimeout field from Config struct
- All timeout configuration now comes from URI parameters
(connectTimeoutMS, serverSelectionTimeoutMS, timeoutMS, etc.)
- Config is now just { URI: string }
- Cleaner, single source of truth for all MongoDB settings
- All tests passing
Upgrade to mongo-driver v2
- Update mongo-driver dependency to v2.8.2
- Remove connstring import (no longer needed)
- Simplify Config by removing manual database name parsing
- Use mongo.Connect without context (v2 API)
- Database selection now deferred to client.Database("") which extracts from URI
- All tests passing
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace controller database hooks with storage.PlaybookStore - Wire controller startup to internal/storage memory and MongoDB stores - Update playbook, trigger, fin, and playbook-action handlers to use new storage interfaces - Update fin capability liveness checks to use storage.FinStore - Refresh mocks and route tests for the new storage API - Remove old repository plumbing from controller wiring
This commit splits the app in two: 1. A runtime, that handles playbook execution and status 2. A server, which hooks up the runtime to incoming HTTP request Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce dependency structs for FIN (HandlerDependencies and Dependencies) and update constructors to accept them. Change Config.StaleAfter from an int (seconds) to time.Duration and remove buried default constants so callers set the TTL explicitly. Update server wiring, capability and API code, queue docs, models comments, and all tests to match the new signatures and types. This refactor centralises DI, clarifies the stale duration type, and updates callers/tests to preserve behavior by passing explicit StaleAfter values. Rename repository to store Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
- Add getter methods to Runtime for all fields (PlaybookStore, FinStore, Cache, Interaction, FinQueue) - Update transport layer to use getters instead of direct field access - Verify boundary: runtime has no pkg/api imports, transport owns HTTP handlers - Add comprehensive boundary tests (TestRuntimeBoundary, TestTransportBoundary, TestConfigBoundary) - Confirm config split: runtime gets only Storage+Cache, transport gets Server+HTTP+Auth+Fin+TheHive+CORS - All tests pass This ensures runtime remains embeddable in non-HTTP contexts and prevents API logic leakage into core orchestration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create internal/services/interfaces.go with all service contract definitions: Core: - ExecutionRuntime: playbook orchestration (StartExecution, ResumeManualStep, GetExecutionStatus) Operational (independent from execution): - FinOperations: agent lifecycle (register, poll, submit, heartbeat, unregister, list, get, delete) - ManualOperations: manual step resolution (list, get, continue pending) Application (thin orchestrators): - TriggerService: coordinate playbook execution (ExecutePlaybook, ExecuteUploadedPlaybook) - PlaybookService: CRUD operations (list, get, create, update, delete, listMetas) - ReporterService: execution reporting (list executions, get report) Each service clearly documents: - Dependencies (what it needs) - Responsibility (what it owns) - Method contracts (no implementation yet) Dependency notes included showing proper layering: HTTP handlers -> services -> runtime -> stores/queues This phase establishes the boundary contracts before implementation. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
- Add internal/services/execution Runtime service implementing the ExecutionRuntime contract - Move async decomposer execution/wait logic out of trigger handlers - Keep manual resume and execution status on the execution runtime service - Refactor trigger handler and API wiring to depend on the service interface - Update HTTP transport to construct and inject the execution service - All affected packages compile and test cleanly This is the first step toward separating core execution from transport concerns. Remove arbitrary 3s timeout from StartExecution, respect context timeout only - Replaced hardcoded time.NewTimer(3 * time.Second) with context-only cancellation - Removed 'fmt' and 'time' imports (no longer needed) - Removed 'Empty' struct and init() function (no longer used) - Service now properly cancellable via context timeout (caller controls deadline) - Eliminates magic numbers and improves testability All tests pass. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Extracted FIN operations into two service interfaces: - FinRegistry: manages FIN registration, admin lifecycle (register, unregister, list, get, delete) - FinWorkService: manages leased work items (poll, submit result, heartbeat) Implementation: - Created internal/services/fin/registry.go with FinRegistry implementation - Created internal/services/fin/work.go with FinWorkService implementation - Added ValidateToken method to FinRegistry for auth middleware - Refactored pkg/api/fin/fin_api.go handler to depend on service interfaces - Updated RequireFinToken middleware to validate tokens via registry - Updated internal/transport/httptransport/server.go to construct FIN services - Added missing error types to pkg/models/fin models (ErrRegistrationDisabled, etc) Tests: - All 15 FIN API tests pass - All transport and controller boundary tests pass - No regressions in existing functionality Key architectural points: - Services own business logic (lease management, token validation, capability matching) - Handlers remain thin HTTP adapters (parse requests, map HTTP semantics) - FIN registry isolated from work service (no cross-dependencies) - Lease semantics stay in FinWorkService (runtime doesn't know about leases) - opaque job_id maps to execution_id/step_execution_id at service boundary
Extracted manual interaction handling behind a ManualInbox service:
- Added internal/services/manual/inbox.go as a thin adapter over interaction storage
- Refactored pkg/api/manual/manual_api.go to depend on services.ManualInbox
- Updated pkg/api/api.go and internal/transport/httptransport/server.go wiring
- Kept the REST path shape /manual/{exec_id}/{step_execution_id} for resource clarity
- Aligned ManualInbox with the existing interaction mock/service boundary
Tests:
- pkg/api/manual passes
- pkg/api and transport/controller boundary tests pass
This keeps manual resolution logic out of the HTTP handler while preserving the current API shape.
Added service adapters for the remaining read/write handlers: - internal/services/playbook wraps the playbook store behind services.PlaybookService - internal/services/reporter wraps execution informer access behind services.ReporterService Refactors: - pkg/api/playbook now depends on the playbook service abstraction - pkg/api/reporter now depends on the reporter service abstraction - pkg/api/api.go composes the new services from existing controller/informer inputs - trigger integration tests updated to the current ExecutionRuntime-based API Validation: - targeted route/transport/controller tests pass - trigger integration suite compiles and passes This continues the transport/service split while leaving the remaining trigger orchestration cleanup for the next step.
Introduce internal/services/trigger Service to handle playbook execution and variable merging (ExecutePlaybook, ExecuteUploadedPlaybook, DecodeVariables, DecodePlaybook) and a ValidationError type for payload validation failures. Move variable decoding/merge logic out of the HTTP handler. Wire the new trigger service into pkg/api and update the trigger handler to use services.TriggerService, handling ValidationError responses. Update integration test to construct the trigger service. Removes duplicate merging/JSON logic from the handler and centralises validation/error handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Create bootstrap package with Container holding all pre-built services - Move service construction from server.go into bootstrap.New() - Refactor server.go to accept bootstrap.Container and delegate to it - Update controller to pass config.Config to transport layer instead of Options - Add ApiWithServices() and ReporterWithService() to pkg/api for injected services - Update tests to use new signatures and bootstrap patterns - server.go now only handles route registration, middleware, and startup - pkg/api/api.go now mostly defines routes, with service construction in bootstrap - All transport and controller tests passing - All trigger integration tests passing This completes Phase 5 of the service extraction refactoring. The HTTP transport layer is now a thin adapter that receives pre-built services from the bootstrap layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…port The previous Phase 5 commit only shuffled service construction into a bootstrap container but left the actual execution engine wiring (capabilities, executors, reporters) on the HTTP Server — meaning the HTTP transport still implemented decomposer_controller.IController and database.IController. This commit completes the separation: - Create internal/bootstrap/workflow_factory.go with WorkflowFactory - Owns NewDecomposer(): capability wiring (SSH, HTTP, OpenC2, PS, Manual, FIN) - Owns initializeTheHiveReporting() - Implements decomposer_controller.IController - Built inside bootstrap.New() — no dependency on HTTP transport - Rewrite internal/transport/httptransport/server.go - Only imports: bootstrap, config, runtime, api, gauth, gin - No capabilities, executors, reporters, TheHive, storage - Implements no application interfaces (no IController, no GetPlaybookStore) - SetupServer() delegates entirely to pre-built services from container - Update internal/bootstrap/bootstrap.go - Remove DecomposerFactory interface (server no longer implements it) - Remove Runtime/Config/DecomposerFactory from Container (transport doesn't need them) - bootstrap.New() takes only runtime + config; creates WorkflowFactory internally - Update pkg/api/api.go - Add NewTriggerHandler/NewManualHandler constructors - Add PlaybookRoutesWithService/ReporterRoutesWithService (service-based) - Remove ApiWithServices/ManualWithService/ReporterWithService duplicates - Keep legacy Database/Reporter/Api helpers for tests still using controllers - Update boundary test to assert the new invariant (server must NOT have GetPlaybookStore or NewDecomposer) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Runtime now constructs and owns all application services (FinRegistry, FinWorkService, ManualInbox, PlaybookService, ReporterService) - ExecutionRuntime remains constructed by bootstrap (due to circular dependency), then injected via SetExecutionRuntime() - Bootstrap simplified to construct ExecutionRuntime and FinHandler only, then retrieve other services from runtime - Container restructured to hold only Runtime and FinHandler, reducing surface exposure - HTTP transport updated to retrieve services from runtime via getters instead of container fields - All tests pass; no functionality broken This achieves the core goal of Phase 5a: runtime owns all services, bootstrap handles only ExecutionRuntime construction and FinHandler composition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove FinHandler construction from bootstrap - Keep bootstrap focused on execution runtime injection only - Compose FinHandler inside HTTP transport from runtime services - Preserve clean runtime/transport separation while keeping existing routes intact Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of mongo, use an sql database storage layer. This allows easy development with persistence without mongo. If we move to temporal, postgres is a required dependency for the temporal cluster. It makes more sense to use the same database technology. Also mongo isn't a good fit for atomic operations, locking, etc, if we use a queue like behavior backed by a db. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
hidde-jan
changed the title
This was referenced Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An experiment that can be used as input for the spike detailed in Maarten's vision document.
This is a very aggressive refactor with (mostly) backwards compatibility to see how soarca could be structured in preparation for the new workflow running model.
Some important things:
The real change of course is moving to a different state machine that can be persisted for workflow runs instead of in-memory call stacks.
Maybe this PR is best understood by lookin at the code instead of the diffs, since so much stuff has been renamed and moved.
This PR also does away with mongo, since we might introduce postgres as an operational dependency (or sqlite in development) if we use an external state machine management service.