Skip to content

feat!: run every building block runner type in one Go binary - #86

Open
grubmeshi wants to merge 15 commits into
mainfrom
feature/single-go-bin-runner
Open

grubmeshi wants to merge 15 commits into
mainfrom
feature/single-go-bin-runner

Conversation

@grubmeshi

@grubmeshi grubmeshi commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Written by AI agent

single-go-bin-runner

One Go binary replaces every building block runner: Terraform, manual, GitHub workflows, GitLab pipelines and Azure DevOps pipelines. Gradle, Kotlin and the JVM images are gone from this repo. This PR makes the next release v2.0.0.

The binary runs each run as a process from a run yml: it starts itself (process dispatcher), or the k8s dispatcher (-tags k8s) starts a Job on the runner type's image. A build tag per type (tf, manual, github, gitlab, azuredevops) links only that executor; no tag links all of them. The runner talks to meshStack only through the meshstack-cli client, so it also runs locally with a CLI profile.

The commits build on each other and read best one by one.

What changes for operators

  • One containers/Dockerfile builds every image. The five type images keep their names, and the new image building-block-runner runs every type. run-controller is the binary with -tags k8s.
  • A Job has backoffLimit: 0 (main had 1), so a run never runs twice. Exit code 3 reaches the run-controller through a podFailurePolicy.
  • k8s.jobTtl (default 60 minutes) keeps a finished Job. The run Secret and the per-definition ServiceAccount are owned by their Jobs and go with the last one.
  • Keycloak workload identity is dispatcher.keycloakWif for every dispatcher.
  • Metrics are named building_block_runner_*; the k8s build also serves the old run_controller_* names.
  • tf versions install under a lock, and a destroy deletes the workspace of the building block.
  • Asynchronous Terraform stays supported: the run is handed over to the pipeline its code starts, as on main.
  • The run JSON given to a pipeline or a pre-run script no longer holds the implementation, so the GitHub App key, trigger token, PAT and SSH key stay in the runner.
  • GitHub, GitLab and Azure DevOps triggers are retried only on 429 and 503, so a pipeline never starts twice.

Breaking changes

  • Every image reads /app/runner-config.yml or RUNNER_CONFIG_FILE, and no image contains a config. Spring's discovery of /app/config and the Spring env are gone.
  • Metrics and /healthz are served on RUNNER_METRICS_PORT=2112; the images used PORT=8080.
  • The run-controller role needs the verb patch on Jobs, Secrets and ServiceAccounts.
  • No image sets SSL_CERT_DIR any more; the entrypoint adds custom CA certificates to the system bundle.
  • The Go module path is github.com/meshcloud/building-block-runner/v2.
  • The tf image no longer preinstalls the AWS CLI: its first aws call installs awscli2 through nix profile add (a few seconds, output on stderr), and later calls use it directly. The image has no nix channel, so nix-env -i and nix-shell -p no longer work; nix profile add nixpkgs#<pkg> and nix shell nixpkgs#<pkg> use the nixpkgs pinned in containers/nix/flake.lock without downloading it, and dependabot keeps that pin current.

Tests

internal/testacc starts the binary built with -cover against a live meshStack: tf (also asynchronous), manual, GitHub, GitLab and Azure DevOps (against mocks of their APIs), SSH clone, Keycloak WIF and network faults. task test:acc:coverage merges unit and acceptance coverage.

Merge order

  1. feat(client): add the runner API and export the HTTP client as client/http meshstack-cli#33, then a CLI release. Move the go.mod pin from the fix: gradle rootProject name #33 head to that release.
  2. This PR, then release v2.0.0.
  3. meshcloud/meshfed-release#11468, which deploys v2.0.0.
  4. feat!: pick the runner through var.meshstack.runner_ref, and add meshstack/run-controller meshstack-hub#401 drops its e2e image pin once v2.0.0 exists.

Cross-repo PRs

🤖 Generated with Claude Code

@meshcloud-gh-actions

meshcloud-gh-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Coverage of the acceptance run against the meshStack backend, on 72758c9a218ac7a8013271895fc1b23080dbf454.

Scope Coverage
Unit tests 44.6%
Acceptance tests 80.6%
Combined 83.8%
Uncovered functions
v2/internal/config/internal/legacy/k8s.go:159: ExtraVolume.volume 0.0%
v2/internal/dispatcher/logctx.go:36: logHandler.WithGroup 0.0%
v2/internal/dispatcher/k8s/dispatcher.go:44: New 0.0%
v2/internal/dispatcher/k8s/workload_identity.go:85: fetchOidcIssuer 0.0%
v2/internal/executor/gitlab/trigger.go:131: identityVerificationRequired 0.0%
v2/internal/metrics/k8s.go:65: legacyGatherer.Gather 0.0%
v2/internal/metrics/k8s.go:102: infoLabel 0.0%
v2/internal/testacc/runnerproc/records.go:20: Record.String 0.0%
v2/internal/testacc/runnerproc/records.go:105: *Log.raw 0.0%

covdata func names a method without its receiver, so an entry can belong to an implementation nothing selects rather than to a function the tests never reached. Open the file and line before reading one as a coverage gap.

@meshcloud-gh-actions

meshcloud-gh-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Coverage of the terraform-provider-meshstack acceptance run against the meshStack backend, on 72758c9a218ac7a8013271895fc1b23080dbf454.

Scope Coverage
Unit tests 44.6%
Acceptance tests 45.8%
Combined 52.2%
Uncovered functions
v2/internal/config/internal/legacy/k8s.go:159: ExtraVolume.volume 0.0%
v2/internal/dispatcher/keycloakwif.go:65: *KeycloakWif.Registration 0.0%
v2/internal/dispatcher/keycloakwif.go:78: keycloakWifSubject 0.0%
v2/internal/dispatcher/keycloakwif.go:112: *KeycloakWif.createClient 0.0%
v2/internal/dispatcher/keycloakwif.go:162: hardcodedClaimMapper 0.0%
v2/internal/dispatcher/keycloakwif.go:179: *KeycloakWif.removeClientScopes 0.0%
v2/internal/dispatcher/keycloakwif.go:197: *KeycloakWif.deleteClient 0.0%
v2/internal/dispatcher/keycloakwif.go:253: *KeycloakWif.adminUrl 0.0%
v2/internal/dispatcher/keycloakwif.go:257: *KeycloakWif.adminRequest 0.0%
v2/internal/dispatcher/keycloakwif.go:272: *KeycloakWif.send 0.0%
v2/internal/dispatcher/logctx.go:36: logHandler.WithGroup 0.0%
v2/internal/dispatcher/k8s/dispatcher.go:44: New 0.0%
v2/internal/dispatcher/k8s/workload_identity.go:85: fetchOidcIssuer 0.0%
v2/internal/executor/client.go:18: NewRunClient 0.0%
v2/internal/executor/pipelineclient.go:11: NewPipelineClient 0.0%
v2/internal/executor/pipelineclient.go:23: IsPermanent 0.0%
v2/internal/executor/reporter.go:80: *Reporter.SetChangesDetected 0.0%
v2/internal/executor/reporter.go:197: *Reporter.requestAbort 0.0%
v2/internal/executor/azuredevops/client.go:28: newAzureDevOps 0.0%
v2/internal/executor/azuredevops/client.go:56: pipelineRun.webUrl 0.0%
v2/internal/executor/azuredevops/client.go:85: azureDevOps.trigger 0.0%
v2/internal/executor/azuredevops/client.go:90: azureDevOps.run 0.0%
v2/internal/executor/azuredevops/client.go:94: azureDevOps.timeline 0.0%
v2/internal/executor/azuredevops/client.go:101: azureDevOps.cancel 0.0%
v2/internal/executor/azuredevops/client.go:107: azureDevOps.do 0.0%
... and 188 more

covdata func names a method without its receiver, so an entry can belong to an implementation nothing selects rather than to a function the tests never reached. Open the file and line before reading one as a coverage gap.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failure reporting can be permanently lost, and debug HTTP logging can expose sensitive output values.

3 open findings
What changed in this PR

Consolidates the Go runners into one configurable binary supporting in-process and Kubernetes Job dispatch.

Changes:

  • Adds shared configuration, HTTP, crypto, and dispatcher packages.
  • Migrates Terraform and manual execution to the unified runner.
  • Updates containers, CI, tests, and documentation for both build modes.
File Description
cmd/​runner/​* Adds the unified runner entry point and build-tag dispatchers.
internal/​config/​* Adds unified and legacy-compatible configuration handling.
internal/​crypto/​* Adds run decryption and redaction.
internal/​http/​* Adds authenticated HTTP, retry, logging, and timeout handling.
internal/​http/​meshapi/​* Adds typed runner API DTOs and operations.
internal/​dispatcher/​* Adds claiming, registration, status reporting, and run-file execution.
internal/​dispatcher/​inprocess/​* Adds in-process Terraform/manual dispatch.
internal/​dispatcher/​k8s/​* Adds Kubernetes Job dispatch and informer recovery.
internal/​build/​build.go Centralizes build version metadata.
tf-block-runner/​tfrun/​* Migrates Terraform execution to shared APIs and configuration.
tf-block-runner/​Dockerfile Builds the unified runner for Terraform.
tf-block-runner/​README.md Documents unified runner usage.
tf-block-runner/​main.go Removes the former Terraform entry point.
tf-block-runner/​build/​build.go Removes runner-specific build metadata.
tf-block-runner/​resources/​test.pem Removes obsolete crypto fixture.
tf-block-runner/​.gitignore Removes package-local ignore rules.
manual-block-runner/​manual/​* Converts manual execution into an in-process executor.
manual-block-runner/​Dockerfile Builds the unified runner for manual runs.
manual-block-runner/​main.go Removes the former manual entry point.
run-controller/​controller/​* Removes the superseded controller implementation.
run-controller/​inprocess/​* Removes the former in-process dispatcher.
run-controller/​build/​* Removes controller-specific build and dispatcher types.
run-controller/​main.go Removes the former controller entry point.
run-controller/​Dockerfile Builds the Kubernetes-tagged unified runner.
run-controller/​k8s/​rbac.yaml Adds Job patch permission.
run-controller/​README.md Documents Kubernetes-tagged runner usage.
go-meshapi-client/​* Removes the superseded client and crypto implementation.
Taskfile.yml Runs builds and checks with both dispatcher modes.
.github/​workflows/​ci.yml Builds, lints, and tests both build-tag variants.
go.mod /​ go.sum Updates configuration and YAML dependencies.
README.md Documents the single-binary architecture.
meshstack-satellite.gradle Updates runner terminology.
.gitignore Preserves the centralized build package.
.agents/​skills/​backend-go/​* Updates repository Go development guidance.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/dispatcher/k8s/informer.go Resolved
Comment thread internal/http/logging.go Resolved
Comment thread internal/dispatcher/poller.go Resolved
@grubmeshi
grubmeshi force-pushed the feature/single-go-bin-runner branch from 1e5eca3 to ec806a3 Compare October 13, 2026 21:06
@grubmeshi grubmeshi changed the title refactor: one dispatcher for the Go runners, in process or as Kubernetes Jobs refactor: one Go runner binary with an in-process or Kubernetes dispatcher Oct 13, 2026
@grubmeshi
grubmeshi force-pushed the feature/single-go-bin-runner branch from ec806a3 to edf06e0 Compare October 14, 2026 06:42
@grubmeshi
grubmeshi removed this pull request from stack #88 October 14, 2026 07:02
@grubmeshi
grubmeshi force-pushed the feature/single-go-bin-runner branch 3 times, most recently from ab7e76e to c66d00e Compare October 14, 2026 11:54
@grubmeshi
grubmeshi marked this pull request as ready for review October 14, 2026 12:15
@grubmeshi
grubmeshi requested a balanced review from Copilot October 14, 2026 12:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Malformed ciphertext can panic the runner, and several authentication, Job-deletion, and health-server failure paths need correction.

4 open findings
3 resolved since last review

🧠 Review effort: Balanced

Comment thread internal/crypto/crypto.go Resolved
Comment thread internal/dispatcher/k8s/informer.go Resolved
Comment thread cmd/runner/main.go Resolved
Comment thread internal/config/api.go Resolved
@grubmeshi
grubmeshi force-pushed the feature/single-go-bin-runner branch from c66d00e to aa082e3 Compare October 14, 2026 12:46

@malhussan malhussan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked config and mostly higher level, great work!
Regarding legacy, is there a plan when we will phase it out?

Comment thread internal/crypto/crypto_test.go Resolved
grubmeshi and others added 15 commits October 30, 2026 08:56
- ignore the direnv files
- take over the golangci-lint linter set of meshstack-cli; tf-block-runner,
  go-meshapi-client, run-controller and manual-block-runner stay excluded
  until they are reworked or deleted
- update golangci-lint to v2.15.0 and golang.org/x/crypto to v0.58.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- internal/crypto holds the runner's private keys, inline or from files,
  and decrypts with any of them; its test keeps a value the former client
  encrypted
- internal/config loads one runner-config.yml in the new or a legacy
  layout, resolves Spring placeholders and RUNNER_* env overrides and
  validates per mode; config.Api answers the meshstack-cli settings, and
  internal/config/tf holds the tf settings
- crypto.DecryptRun decrypts the sensitive inputs and the implementation
  secret of a meshstack-cli run

go.mod pins meshstack-cli at the head of meshstack-cli#33 until it merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- internal/executor runs the run in RUNNER_RUN_YML with the executor of
  its type and reports through a Reporter's steps; exit code 0 means it
  delivered a final or handed-over status, 3 that it delivered FAILED;
  closing stdin stops the run with the reason written before it
- internal/poller claims and decrypts runs up to maxConcurrentJobs and
  reports FAILED for a run that did not deliver its status; the process
  dispatcher starts the binary again per run with an allowlisted env, a
  run dir of its own and stdin as lifeline, and kills the process group
  killGrace after it closed stdin
- one main in cmd/runner for every image; tf and manual register as
  executors, tfrun clones with the git CLI and SIGTERM fails a run with
  the signal as its reason
- an asynchronous Terraform run reports one "Prepare Run" step and is
  handed over to the pipeline its code starts, as the tf runner on main
  did; an async destroy applies first, so the destroy sees this run's
  inputs
- a run builds only the runner API of the meshStack client, and logs
  each finished tofu command with its duration and exit code
- dead tf-block-runner code goes, and go-meshapi-client is deleted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- internal/dispatcher/k8s starts a Job per run on the image of its type,
  with the run yml and run token in a Secret and an emptyDir as work dir;
  a podFailurePolicy maps exit code 3 to a Job condition and an informer
  keeps the capacity count across restarts
- a Job gets backoffLimit 0 and dispatcher.killGrace as termination grace
- k8s.jobTtl (default 60 minutes) keeps a finished Job, unless the job
  spec template sets ttlSecondsAfterFinished
- every Job becomes an owner of its run Secret and of the per-definition
  ServiceAccount through a strategic merge patch, so both go with their
  last Job; the runner needs patch on secrets and serviceaccounts
- creating the service account, Secret and Job retries transient
  Kubernetes API errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dispatcher.keycloakWif takes over the run-controller WIF: the poller
mints one token per TERRAFORM run from a Keycloak client of its own,
with sub bbd:<workspace>:<definition uuid>, and registers Keycloak as
the runner's issuer. A run whose token cannot be issued fails with a
fixed message and does not start.

- the process dispatcher writes the token below the runner dir, the k8s
  dispatcher puts it into the run's Secret
- the tf executor links it as .meshstack/wif-token and passes its path in
  meshstack_keycloak_wif_token_file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pace of the building block

- a flock per version under the install dir keeps two run processes from
  installing the same version at once
- the install goes to a temp dir that is renamed when complete, so an
  interrupted install no longer counts as complete
- the install logs through slog with the version and path
- a destroy deletes the workspace of the building block

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- /metrics and /healthz on RUNNER_METRICS_PORT, or PORT when unset; a
  port in use fails before the runner claims a run
- run_duration_seconds counts a run as failed when its exit code is not 0
- each finished run logs "run finished" with the labels of
  run_duration_seconds and its exit code
- a k8s build also exposes the run-controller metrics under their old
  names, except those that active_runs and max_concurrent_runs replace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The debug config points at http://localhost:8080, as the CLI accepts http
only for a loopback host, and each Job gets a socat sidecar that forwards
that port to meshfed-api on the host. The config moves to the current
layout, as only job spec templates can declare the sidecar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- a Reporter adds steps after it registered, and Step.SetStatus sets a
  step's status without failing the run
- the gitlab, azuredevops and github executors; a sync run reports the
  jobs or stages as steps and cancels the external run when it stops;
  GitHub authenticates as the GitHub App and finds the run by time when a
  dispatch has no run details; GitLab's MESHSTACK_RUN leaves out the
  trigger token
- GitLab gets an input that is no string as JSON text, which it parses
  into the type of a number, boolean or array spec:inputs entry
- executor.NewPipelineClient, which follows no redirects and retries a
  trigger only on 429 and 503

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt:runner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- containers/Dockerfile builds every image from the Go binary, with the
  target runner or tf and the build tag of the image, including the
  all-types building-block-runner image without a build tag
- dev/ holds the runner configs of task start:runner and
  task debug:minikube
- Gradle and the Kotlin runners are removed
- the k8s Job no longer sets RUN_JSON_FILE_PATH and RUNNER_API_URL, and
  the configs drop SPRING_PROFILES_ACTIVE
- no image sets SSL_CERT_DIR, which the manual image did: git ignores
  it, and the entrypoint adds custom CA certificates to the system
  bundle that git and Go read
- the images build with VCS stamping, so the runner reports the module
  version Go derives from the tags and the git revision

BREAKING CHANGE: the github, gitlab and azure-devops images run the Go
binary, which reads runner-config.yml from /app/runner-config.yml or the
path in RUNNER_CONFIG_FILE. Spring's discovery of /app/config and the
Spring environment no longer apply. Every image serves metrics on
RUNNER_METRICS_PORT=2112; tf and the other runner images used PORT=8080.
No image contains a runner-config.yml any more, so a deployment must
mount /app/runner-config.yml or set RUNNER_CONFIG_FILE. The Go module
path is github.com/meshcloud/building-block-runner/v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…estacc rules

The name matches the development skill of meshstack-cli. .claude/settings.json
enables the modern-go-guidelines plugin, which the skill builds on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- each TestAcc<Case> deploys the folder testdata/<case> with the meshStack
  provider, starts the runner binary built with coverage and asserts its
  JSON log and the run in meshStack
- it replaces the root testacc/ package
- tf runs: inputs, secrets, timeout, abort, plan approval, stale plan,
  killed child, workspaces, Terraform 1.5.5, failing init, git ref as tag
  and commit, asynchronous Terraform; manual and noop runs
- GitHub, GitLab and Azure DevOps runs against mocks of their APIs
- SSH clone of the module and of a child module from an in-process git server
- Keycloak workload identity against the stack's Keycloak
- network faults with toxiproxy
- the harness, the runner process, the mocks, the SSH server, toxiproxy and
  Keycloak are subpackages, which depguard keeps out of non-test code
- task test:acc:coverage merges the unit and acceptance coverage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ges current

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants