From 31432c4db36125e416df06fe9a417dff61ba968b Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Fri, 5 Jun 2026 15:48:45 -0500 Subject: [PATCH 1/8] Add Bazel RBE integration guide (DI-574) New customer-facing page covering Orka as the macOS worker layer in a Bazel Remote Build Execution setup. Covers two paths: open-source buildfarm (recommended) and BuildBuddy (for teams already using it). Includes multi-worker deployment, GitHub Actions workflow automation, LaunchDaemon auto-start pattern, and persistent cache setup. Co-Authored-By: Claude Sonnet 4.6 --- docs.json | 1 + orka/orka-devops-integrations/bazel-rbe.mdx | 553 ++++++++++++++++++++ 2 files changed, 554 insertions(+) create mode 100644 orka/orka-devops-integrations/bazel-rbe.mdx diff --git a/docs.json b/docs.json index f8084865..34279516 100644 --- a/docs.json +++ b/docs.json @@ -172,6 +172,7 @@ "orka/orka-devops-integrations/buildkite", "orka/orka-devops-integrations/teamcity", "orka/orka-devops-integrations/packer", + "orka/orka-devops-integrations/bazel-rbe", "orka/orka-devops-integrations/claude-code" ] }, diff --git a/orka/orka-devops-integrations/bazel-rbe.mdx b/orka/orka-devops-integrations/bazel-rbe.mdx new file mode 100644 index 00000000..e7a67bbf --- /dev/null +++ b/orka/orka-devops-integrations/bazel-rbe.mdx @@ -0,0 +1,553 @@ +--- +title: "Bazel Remote Build Execution with Orka" +description: "Run macOS Bazel RBE workers on Orka VMs. Use open-source buildfarm or BuildBuddy as the RBE control plane — Orka provides the macOS worker fleet either way." +--- + +Bazel Remote Build Execution (RBE) splits a build into two distinct layers: the control plane, which schedules actions, manages the cache, and routes work to available workers; and the worker pool, which executes the actions on real hardware. For iOS and macOS builds, the worker pool has to be macOS — and that's where Orka fits. + +This page covers two paths: using open-source [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm) as the control plane, and using [BuildBuddy](https://www.buildbuddy.io/) as the control plane. Both use Orka VMs as the macOS worker fleet. + +## Architecture + +``` +[Developer / CI] + | + | gRPC (remote-apis protocol) + v +[RBE server — Linux VM] + bazel-buildfarm or BuildBuddy + Redis backplane (buildfarm only) + | + | dispatch to workers + v +[Orka macOS VMs — worker pool] + Xcode, Homebrew, Java, Bazelisk + bazel-buildfarm worker or BuildBuddy executor + | + v +[Remote cache — Linux VM or S3] + bazel-remote, buildfarm CAS, or BuildBuddy cache +``` + +The RBE server and cache typically run on a Linux VM co-located in your Orka cluster at MacStadium. The macOS workers are Orka VMs deployed from a versioned OCI image. + + +Bazel 9.0.0 is not compatible with bazel-buildfarm as of February 2026. Pin your workers and client to Bazel 8.x by running `echo "8.2.1" > .bazelversion` in your buildfarm directory before building. This does not affect BuildBuddy setups. + + +--- + +## Option 1: Open-source RBE with bazel-buildfarm + +### What you need + +- One Ubuntu VM (or bare-metal Linux) for the buildfarm server and Redis. This can be a co-located VM in your MacStadium environment. +- One or more Orka VMs as macOS workers. Scale the worker count to match your parallelism target. +- Your Bazel client (local dev machine or CI runner) with network access to the buildfarm server. + +### Set up the buildfarm server + +On your Ubuntu VM: + +**Install Java:** + +```bash +sudo apt install default-jdk +``` + +**Install Docker:** + +```bash +sudo apt-get update +sudo apt-get install ca-certificates curl +sudo install -m 0755 -d /etc/apt/keyrings +sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc +sudo chmod a+r /etc/apt/keyrings/docker.asc + +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update +sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin +``` + +**Install Bazel:** + +```bash +sudo apt install npm +sudo npm install -g @bazel/bazelisk +``` + +**Start Redis:** + +```bash +sudo docker run -d --rm --name buildfarm-redis -p 6379:6379 redis:7.2.4 +``` + +**Install redis-cli and configure Redis:** + +```bash +sudo apt install redis-tools +redis-cli config set stop-writes-on-bgsave-error no +``` + +**Clone and configure buildfarm:** + +```bash +sudo apt install gh +gh auth login +gh repo clone bazelbuild/bazel-buildfarm +cd bazel-buildfarm + +# Pin Bazel version (9.0.0 is not supported) +echo "8.2.1" > .bazelversion +``` + +Edit `examples/config.minimal.yml` and set the Redis URI to your VM's IP: + +```yaml +backplane: + redisUri: "redis://:6379" + queues: + - name: "cpu" + properties: + - name: "min-cores" + value: "*" + - name: "max-cores" + value: "*" +``` + +**Start the buildfarm server:** + +```bash +bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- \ + --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ + $PWD/examples/config.minimal.yml +``` + +The server listens on port `8980` by default. + +--- + +### Prepare your Orka worker image + +Create a base macOS VM image with the following installed. You'll deploy multiple workers from this image, so getting it right once avoids repeating the setup on each VM. + +**Install Xcode:** + +Download Xcode from [developer.apple.com](https://developer.apple.com/xcode/). Launch it, accept the license agreements, and complete the initial setup. Make sure the version matches your project's minimum Xcode requirement. + +**Install Homebrew:** + +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +``` + +Follow the post-install instructions to add Homebrew to your shell path. + +**Install dependencies:** + +```bash +brew install temurin # Java +brew install bazelisk # Bazel version manager +brew install gh # GitHub CLI +``` + +**Clone buildfarm and pin Bazel:** + +```bash +gh auth login +gh repo clone bazelbuild/bazel-buildfarm +cd bazel-buildfarm +echo "8.2.1" > .bazelversion +``` + +Once everything is installed, save this VM as a base image in Orka. You'll use this image for all your workers. + +--- + +### Deploy and configure workers + +For each worker, deploy an Orka VM from your base image and configure it to point at the buildfarm server. + +**Deploy a worker VM:** + +```bash +orka3 vm deploy --vm-config --name +``` + +**Edit the worker config on each VM:** + +On the worker VM, edit `bazel-buildfarm/examples/config.minimal.yml`: + +```yaml +backplane: + redisUri: "redis://:6379" + queues: + - name: "cpu" + properties: + - name: "min-cores" + value: "*" + - name: "max-cores" + value: "*" +worker: + publicName: ":8981" # Must be unique per worker + executionPolicies: + - name: test + executionWrapper: + path: "/Users/admin/buildfarm/bazel-buildfarm/macos-wrapper.sh" +``` + +The `publicName` field must be unique for each worker. Use each VM's IP address. Workers that share a `publicName` will conflict. + +**Start the worker:** + +```bash +cd ~/bazel-buildfarm +bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- \ + --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ + $PWD/examples/config.minimal.yml +``` + +Repeat for each worker VM. All workers point at the same Redis and buildfarm server; the server distributes work across the pool automatically. + +--- + +### Scale the worker pool + +Because all workers use the same base image and the same buildfarm server address, scaling is straightforward: deploy more Orka VMs from the base image, set a unique `publicName` on each, and start the worker process. + +A reasonable starting point for iOS simulator workloads is one worker VM per Orka node, with each VM sized to leave headroom for simulator processes. For compile-bound Bazel actions, higher VM density per node is fine since those workloads are CPU-bound rather than I/O-bound. + + +If you're running parallel simulator tests across multiple VMs on the same node and see I/O slowdowns, the bottleneck is disk contention from concurrent simulator writes — not virtualization overhead. Distribute across more nodes at lower VM-per-node density rather than reducing the total VM count. + + +--- + +### Connect your Bazel client + +Add the following to your project's `.bazelrc`: + +``` +build --remote_executor=grpc://:8980 +build --remote_default_exec_properties=execution-policy=test +``` + +Or pass the flags directly: + +```bash +bazel build \ + --remote_executor=grpc://:8980 \ + --remote_default_exec_properties=execution-policy=test \ + //your:target +``` + +To verify the cache is working, run the build once (cold), then run `bazel clean --expunge` and build again. The second build should show `remote cache hit` for most actions and complete significantly faster. + +Expected output on a warm cache: + +``` +INFO: Elapsed time: 6.595s, Critical Path: 0.14s +INFO: 65 processes: 28 remote cache hit, 37 internal. +INFO: Build completed successfully, 65 total actions +``` + +--- + +## Option 2: BuildBuddy with Orka + +If your team is already using BuildBuddy as your RBE control plane, Orka VMs can serve as the self-hosted Mac executors that BuildBuddy dispatches work to. This keeps Orka in the stack as the macOS worker layer — you get clean VM boots, OCI image versioning, and node failover even when BuildBuddy is handling scheduling and caching. + +For teams evaluating options, the OSS buildfarm path above is the recommended starting point. It gives you full control over the stack, no additional vendor dependency, and the same Orka worker benefits. + + +BuildBuddy's snapshot-based warm-runner technology (Firecracker) is Linux-only. On macOS, BuildBuddy falls back to runner recycling — keeping the executor process alive between invocations rather than snapshotting it. Orka VM snapshots work independently of this and are available regardless of which RBE control plane you use. + + +### What you need + +- An existing BuildBuddy deployment (cloud or self-hosted). +- One or more Orka VMs as macOS executors. +- Your Bazel client with network access to BuildBuddy's gRPC endpoint. + +### Prepare your Orka executor image + +Install the following on a base macOS VM: + +**Xcode:** Download from [developer.apple.com](https://developer.apple.com/xcode/). Accept the license and complete setup. + +**Homebrew:** + +```bash +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" +``` + +**Dependencies:** + +```bash +brew install bazelisk +``` + +**BuildBuddy executor binary:** + +Download the latest `buildbuddy-executor` binary for macOS/ARM from [BuildBuddy's GitHub releases](https://github.com/buildbuddy-io/buildbuddy/releases). Move it to a stable path: + +```bash +chmod +x buildbuddy-executor +sudo mv buildbuddy-executor /usr/local/bin/buildbuddy-executor +``` + +**Create the executor config file** at `/etc/buildbuddy/executor.yaml`: + +```yaml +executor: + app_target: "grpcs://remote.buildbuddy.io:443" + api_key: "" + local_cache_size_bytes: 10000000000 # 10 GB local disk cache +``` + +For a self-hosted BuildBuddy deployment, replace `remote.buildbuddy.io` with your BuildBuddy server address. + +**Run the executor:** + +```bash +buildbuddy-executor --config_file /etc/buildbuddy/executor.yaml +``` + +To keep the executor running across VM restarts, configure it as a LaunchDaemon. Create `/Library/LaunchDaemons/io.buildbuddy.executor.plist`: + +```xml + + + + + Label + io.buildbuddy.executor + ProgramArguments + + /usr/local/bin/buildbuddy-executor + --config_file + /etc/buildbuddy/executor.yaml + + RunAtLoad + + KeepAlive + + StandardOutPath + /var/log/buildbuddy-executor.log + StandardErrorPath + /var/log/buildbuddy-executor.log + + +``` + +Load it: + +```bash +sudo launchctl load /Library/LaunchDaemons/io.buildbuddy.executor.plist +``` + +Save this VM as a base image and deploy additional executor VMs from it to scale the pool. + +--- + +### Connect your Bazel client + +Add the following to your `.bazelrc`: + +``` +build --remote_executor=grpcs://remote.buildbuddy.io:443 +build --remote_cache=grpcs://remote.buildbuddy.io:443 +build --remote_header=x-buildbuddy-api-key= +``` + +BuildBuddy's web UI shows build results, cache hit rates, and action timing at [app.buildbuddy.io](https://app.buildbuddy.io). + +--- + +## Automate worker lifecycle with GitHub Actions + +The manual steps above work for static worker pools that stay running. For ephemeral workers that spin up per build and tear down when it's done, you can drive the whole thing from a GitHub Actions workflow. This pattern works for both the buildfarm and BuildBuddy setups — the difference is just what's installed on the worker image. + +### Set up worker auto-start in the base image + +The key to making ephemeral workers viable is having the worker process start automatically when the VM boots, so the GitHub Actions workflow doesn't need to SSH into each VM to configure it. Do this once when building your base image, then save it. + +**Create a startup script** at `/Users/admin/start-buildfarm-worker.sh`: + +```bash +#!/bin/bash +cd /Users/admin/bazel-buildfarm +exec bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- \ + --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ + $PWD/examples/config.minimal.yml +``` + +```bash +chmod +x /Users/admin/start-buildfarm-worker.sh +``` + +**Create a LaunchDaemon** at `/Library/LaunchDaemons/io.macstadium.buildfarm-worker.plist`: + +```xml + + + + + Label + io.macstadium.buildfarm-worker + ProgramArguments + + /bin/bash + /Users/admin/start-buildfarm-worker.sh + + RunAtLoad + + KeepAlive + + WorkingDirectory + /Users/admin/bazel-buildfarm + StandardOutPath + /var/log/buildfarm-worker.log + StandardErrorPath + /var/log/buildfarm-worker.log + + +``` + +```bash +sudo launchctl load /Library/LaunchDaemons/io.macstadium.buildfarm-worker.plist +``` + +With this in place, every VM deployed from this image will register with your buildfarm server within about 30 seconds of boot. Save the image in Orka before continuing. + +For BuildBuddy setups, the LaunchDaemon above already covers auto-start — no separate script needed since the BuildBuddy LaunchDaemon is set up in the executor image prep steps above. + +--- + +### The workflow + +The workflow runs on a self-hosted runner that has access to your Orka cluster — the Linux VM running your buildfarm server is the natural home for it, since it's already in the same network as your Orka nodes. Store your Orka service account name and token as GitHub Actions secrets. + +```yaml +name: Bazel RBE + +on: + push: + branches: [main] + pull_request: + +env: + BUILDFARM_SERVER: "10.221.188.8" # IP of your persistent buildfarm server + WORKER_COUNT: "4" # How many Orka worker VMs to deploy + WORKER_VM_CONFIG: "bazel-worker" # Name of your saved Orka VM config + +jobs: + build: + runs-on: [self-hosted, linux] + + steps: + - uses: actions/checkout@v4 + + - name: Authenticate with Orka + id: auth + run: | + TOKEN=$(orka3 serviceaccount token "${{ secrets.ORKA_SA_NAME }}" \ + --endpoint "${{ vars.ORKA_ENDPOINT }}") + echo "token=$TOKEN" >> "$GITHUB_OUTPUT" + env: + ORKA_SA_TOKEN: ${{ secrets.ORKA_SA_TOKEN }} + + - name: Deploy worker VMs + id: workers + run: | + NAMES=() + for i in $(seq 1 "$WORKER_COUNT"); do + NAME="bazel-worker-${{ github.run_id }}-${i}" + orka3 vm deploy \ + --vm-config "$WORKER_VM_CONFIG" \ + --name "$NAME" \ + --endpoint "${{ vars.ORKA_ENDPOINT }}" \ + --token "${{ steps.auth.outputs.token }}" + NAMES+=("$NAME") + done + echo "names=${NAMES[*]}" >> "$GITHUB_OUTPUT" + + - name: Wait for workers to register + run: | + echo "Waiting for VMs to boot and register with buildfarm..." + sleep 45 + for NAME in ${{ steps.workers.outputs.names }}; do + orka3 vm list "$NAME" \ + --endpoint "${{ vars.ORKA_ENDPOINT }}" \ + --token "${{ steps.auth.outputs.token }}" + done + + - name: Run Bazel build + run: | + bazel build \ + --remote_executor=grpc://${{ env.BUILDFARM_SERVER }}:8980 \ + --remote_default_exec_properties=execution-policy=test \ + //... + + - name: Tear down worker VMs + if: always() + run: | + for NAME in ${{ steps.workers.outputs.names }}; do + orka3 vm delete "$NAME" \ + --endpoint "${{ vars.ORKA_ENDPOINT }}" \ + --token "${{ steps.auth.outputs.token }}" || true + done +``` + +A few things worth noting: + +- **Service account auth** — `orka3 serviceaccount token` is the right auth method for CI/CD. User tokens from `orka3 login` expire after an hour and will fail mid-build. See [Manage service accounts](/orka/orka-cluster-access/orka-cluster-manage-service-accounts) for setup. +- **Worker count** — `WORKER_COUNT` is the lever for parallelism. Start at 2–4 and increase if your build has enough parallelizable actions to justify it. Each VM adds to your cluster's VM slot consumption. +- **The 45-second wait** — covers VM boot plus the time for the LaunchDaemon to start the buildfarm worker process and register with the server. Adjust down if your cluster consistently boots faster; adjust up if you see "no workers available" errors on the first few actions. +- **`if: always()` on teardown** — ensures VMs are cleaned up even when the build fails. Without this, a failed build leaves orphaned VMs running until someone deletes them manually. +- **Unique VM names** — appending `github.run_id` to the VM name avoids naming collisions when multiple builds run in parallel. + +--- + +### Scale across multiple runs + +If your team runs many concurrent builds, the `WORKER_COUNT` per workflow multiplies by the number of parallel runs. Watch your cluster's available VM slots — the Orka web UI shows current VM counts per node. For sustained high-concurrency usage, a persistent warm pool (always-on workers) is more efficient than deploying and tearing down per-build, since it avoids the per-build boot latency and avoids VM slot churn. + +--- + +For both setups, a shared remote cache means build outputs survive VM restarts and are shared across all workers and clients. The simplest approach is running [bazel-remote](https://github.com/buchgr/bazel-remote) on your Linux VM: + +```bash +docker run -d \ + -u 1000:1000 \ + -v /path/to/cache:/data \ + -p 9090:8080 \ + --restart unless-stopped \ + buchgr/bazel-remote-cache \ + --max_size=50 +``` + +Point your Bazel client at it: + +``` +build --remote_cache=http://:9090 +``` + +For buildfarm setups, the buildfarm server also provides a Content-Addressable Store (CAS) — you don't need a separate cache server unless you want to share the cache with clients that bypass RBE. + +--- + +## Why Orka for the worker pool + +Bazel RBE handles scheduling, caching, and build orchestration. What it doesn't handle is provisioning a fleet of macOS workers that stays clean, versioned, and operationally tractable at scale. That's what Orka provides: + +- **Clean workers on demand.** Deploying a fresh Orka VM from a versioned OCI image takes seconds. Bare-metal re-imaging takes hours. +- **Image versioning.** Orka 3.5's Harbor OCI registry lets you treat your build environments like container images — versioned, signed, pullable by tag. Xcode drift across the fleet collapses cache hit rates; pinned OCI images prevent it. +- **Worker pool scaling.** Deploy additional worker VMs from the same base image without configuring physical hardware. +- **Declarative VM counts.** Orka's Kubernetes-native operator supports deployment-style VM pool management. Declare the pool size; the control plane keeps it warm. +- **Node failover.** If an Orka node goes down, VMs reschedule to healthy nodes. Bare-metal host failure is a manual remediation. + +For more on the Orka value proposition in Bazel RBE architectures, contact [support@macstadium.com](mailto:support@macstadium.com) or speak with your MacStadium solutions engineer. From e0712342097ea7a2e9dec187613076b1d53c649b Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Mon, 15 Jun 2026 13:12:42 -0500 Subject: [PATCH 2/8] update: strengthen Why Orka section with research findings - Expand bare-metal re-imaging explanation (DEP, Xcode, hours not seconds) - Rename Image versioning to Pinned build environments with Xcode drift framing - Add isolation bullet explaining cross-project contamination risk - Tighten worker pool scaling bullet - Source: competitive research from MPD-81 Co-Authored-By: Claude Sonnet 4.6 --- orka/orka-devops-integrations/bazel-rbe.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/orka/orka-devops-integrations/bazel-rbe.mdx b/orka/orka-devops-integrations/bazel-rbe.mdx index e7a67bbf..8a3c3318 100644 --- a/orka/orka-devops-integrations/bazel-rbe.mdx +++ b/orka/orka-devops-integrations/bazel-rbe.mdx @@ -542,12 +542,12 @@ For buildfarm setups, the buildfarm server also provides a Content-Addressable S ## Why Orka for the worker pool -Bazel RBE handles scheduling, caching, and build orchestration. What it doesn't handle is provisioning a fleet of macOS workers that stays clean, versioned, and operationally tractable at scale. That's what Orka provides: +Bazel RBE handles scheduling, caching, and build orchestration. What it doesn't provide is the macOS worker fleet itself. Running Bazel RBE on macOS requires clean, versioned, isolated Mac workers that don't turn into an ops problem at scale. That's what Orka handles. -- **Clean workers on demand.** Deploying a fresh Orka VM from a versioned OCI image takes seconds. Bare-metal re-imaging takes hours. -- **Image versioning.** Orka 3.5's Harbor OCI registry lets you treat your build environments like container images — versioned, signed, pullable by tag. Xcode drift across the fleet collapses cache hit rates; pinned OCI images prevent it. -- **Worker pool scaling.** Deploy additional worker VMs from the same base image without configuring physical hardware. -- **Declarative VM counts.** Orka's Kubernetes-native operator supports deployment-style VM pool management. Declare the pool size; the control plane keeps it warm. -- **Node failover.** If an Orka node goes down, VMs reschedule to healthy nodes. Bare-metal host failure is a manual remediation. +- **Clean workers on demand.** Deploying a fresh Orka VM from a versioned OCI image takes seconds. Bare-metal re-imaging takes hours: DEP enrollment, a full Xcode install, re-configuration. At CI cadence, that's not a viable clean-boot story. +- **Pinned build environments.** Bazel pins toolchain state into the action hash. If your worker's Xcode version or SDK drifts, cache hits collapse. Orka 3.5's Harbor OCI registry lets you version, sign, and pin your build images the same way you'd pin a container image. Bare metal has no equivalent primitive. +- **Isolation per project or team.** VMs are isolated by definition. If one build poisons a worker, it doesn't affect other workers. On bare metal, a bad build corrupts the host until someone re-images it. +- **Worker pool scaling without physical hardware.** Deploy more workers from the same base image. Orka's Kubernetes-native operator supports declarative pool sizing. +- **Node failover.** If an Orka node goes down, VMs reschedule to healthy nodes. A bare-metal host failure is a manual remediation. -For more on the Orka value proposition in Bazel RBE architectures, contact [support@macstadium.com](mailto:support@macstadium.com) or speak with your MacStadium solutions engineer. +For more on running Bazel RBE on Orka, contact [support@macstadium.com](mailto:support@macstadium.com) or speak with your MacStadium solutions engineer. From 7d6b22d9372e606b26fb8becf82101be62a8bd4e Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Thu, 18 Jun 2026 13:42:54 -0500 Subject: [PATCH 3/8] update: replace bazelisk run with JAR-based approach in bazel-rbe.mdx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Switch server and worker startup from `bazelisk run` (rebuilds from source on every restart) to pre-built JARs run with `java -jar`, matching the verified approach in the quickstart. Also replace config.minimal.yml references with flat YAML configs — the !include-based examples format does not work at runtime. Co-Authored-By: Claude Sonnet 4.6 --- orka/orka-devops-integrations/bazel-rbe.mdx | 126 ++++++++++++++------ 1 file changed, 91 insertions(+), 35 deletions(-) diff --git a/orka/orka-devops-integrations/bazel-rbe.mdx b/orka/orka-devops-integrations/bazel-rbe.mdx index 8a3c3318..8ef0fb96 100644 --- a/orka/orka-devops-integrations/bazel-rbe.mdx +++ b/orka/orka-devops-integrations/bazel-rbe.mdx @@ -92,7 +92,7 @@ sudo apt install redis-tools redis-cli config set stop-writes-on-bgsave-error no ``` -**Clone and configure buildfarm:** +**Clone and build buildfarm:** ```bash sudo apt install gh @@ -102,28 +102,44 @@ cd bazel-buildfarm # Pin Bazel version (9.0.0 is not supported) echo "8.2.1" > .bazelversion + +mkdir -p ~/buildfarm +bazel build //src/main/java/build/buildfarm:buildfarm-server_deploy.jar +cp bazel-bin/src/main/java/build/buildfarm/buildfarm-server_deploy.jar ~/buildfarm/ ``` -Edit `examples/config.minimal.yml` and set the Redis URI to your VM's IP: +**Create the server config** at `~/buildfarm/server-config.yml`: ```yaml +digestFunction: SHA256 +defaultActionTimeout: 600 +maximumActionTimeout: 3600 +server: + instanceType: SHARD + name: shard + port: 8980 backplane: - redisUri: "redis://:6379" + type: SHARD + redisUri: redis://:6379 queues: - - name: "cpu" - properties: - - name: "min-cores" - value: "*" - - name: "max-cores" - value: "*" + - name: cpu + allowUnmatched: true + properties: + - name: min-cores + value: '*' + - name: max-cores + value: '*' ``` + +The backplane type is `SHARD`, not `REDIS`. Use a flat YAML config — the `config.minimal.yml` in the examples directory uses a preprocessor format the server does not support at runtime. + + **Start the buildfarm server:** ```bash -bazelisk run //src/main/java/build/buildfarm:buildfarm-server -- \ - --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ - $PWD/examples/config.minimal.yml +cd ~/buildfarm +nohup java -jar buildfarm-server_deploy.jar server-config.yml > server.log 2>&1 & ``` The server listens on port `8980` by default. @@ -154,7 +170,7 @@ brew install bazelisk # Bazel version manager brew install gh # GitHub CLI ``` -**Clone buildfarm and pin Bazel:** +**Clone and build buildfarm:** ```bash gh auth login @@ -163,6 +179,27 @@ cd bazel-buildfarm echo "8.2.1" > .bazelversion ``` +Build the worker JAR. The `--copt` flags work around a macOS deployment target issue in abseil-cpp and protobuf: + +```bash +bazel build \ + //src/main/java/build/buildfarm:buildfarm-shard-worker_deploy.jar \ + --host_copt=-mmacosx-version-min=10.15 \ + --copt=-mmacosx-version-min=10.15 + +mkdir -p ~/buildfarm +cp bazel-bin/src/main/java/build/buildfarm/buildfarm-shard-worker_deploy.jar ~/buildfarm/ +``` + +This takes 3 to 5 minutes on first run. + +**Create the macOS execution wrapper:** + +```bash +printf '#!/bin/bash\nexec "$@"\n' > ~/buildfarm/macos-wrapper.sh +chmod +x ~/buildfarm/macos-wrapper.sh +``` + Once everything is installed, save this VM as a base image in Orka. You'll use this image for all your workers. --- @@ -177,26 +214,48 @@ For each worker, deploy an Orka VM from your base image and configure it to poin orka3 vm deploy --vm-config --name ``` -**Edit the worker config on each VM:** - -On the worker VM, edit `bazel-buildfarm/examples/config.minimal.yml`: +**Create the worker config** at `~/buildfarm/worker-config.yml`: ```yaml +digestFunction: SHA256 +defaultActionTimeout: 600 +maximumActionTimeout: 3600 backplane: - redisUri: "redis://:6379" + type: SHARD + redisUri: redis://:6379 queues: - - name: "cpu" - properties: - - name: "min-cores" - value: "*" - - name: "max-cores" - value: "*" + - name: cpu + allowUnmatched: true + properties: + - name: min-cores + value: '*' + - name: max-cores + value: '*' worker: - publicName: ":8981" # Must be unique per worker + port: 8981 + publicName: :8981 + root: /tmp/worker + dequeueMatchSettings: + allowUnmatched: true + storages: + - type: FILESYSTEM + path: /tmp/worker/cache + maxSizeBytes: 2147483648 + fileDirectoriesIndexInMemory: false + skipLoad: false + hexBucketLevels: 0 + execRootCopyFallback: false + executeStageWidth: 1 + inputFetchStageWidth: 1 + inputFetchDeadline: 60 + reportResultStageWidth: 1 + linkExecFileSystem: false + linkInputDirectories: false executionPolicies: - - name: test - executionWrapper: - path: "/Users/admin/buildfarm/bazel-buildfarm/macos-wrapper.sh" + - name: test + executionWrapper: + path: /Users/admin/buildfarm/macos-wrapper.sh + arguments: ``` The `publicName` field must be unique for each worker. Use each VM's IP address. Workers that share a `publicName` will conflict. @@ -204,10 +263,9 @@ The `publicName` field must be unique for each worker. Use each VM's IP address. **Start the worker:** ```bash -cd ~/bazel-buildfarm -bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- \ - --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ - $PWD/examples/config.minimal.yml +mkdir -p /tmp/worker/cache +export JAVA_HOME=$(/usr/libexec/java_home -v 21) +nohup $JAVA_HOME/bin/java -jar ~/buildfarm/buildfarm-shard-worker_deploy.jar ~/buildfarm/worker-config.yml > ~/buildfarm/worker.log 2>&1 & ``` Repeat for each worker VM. All workers point at the same Redis and buildfarm server; the server distributes work across the pool automatically. @@ -379,10 +437,8 @@ The key to making ephemeral workers viable is having the worker process start au ```bash #!/bin/bash -cd /Users/admin/bazel-buildfarm -exec bazelisk run //src/main/java/build/buildfarm:buildfarm-shard-worker -- \ - --jvm_flag=-Djava.util.logging.config.file=$PWD/examples/logging.properties \ - $PWD/examples/config.minimal.yml +export JAVA_HOME=$(/usr/libexec/java_home -v 21) +exec $JAVA_HOME/bin/java -jar /Users/admin/buildfarm/buildfarm-shard-worker_deploy.jar /Users/admin/buildfarm/worker-config.yml ``` ```bash From bb858303ba0ff6686008b482283764a714e7d6f4 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Thu, 2 Jul 2026 15:43:48 -0500 Subject: [PATCH 4/8] Add Bazel RBE quickstart and fix start-worker.sh publicName bug Adds the single-VM quickstart guide (bazel-rbe-quickstart.mdx) verified end-to-end against a live Orka cluster. Also fixes the start-worker.sh startup script template in bazel-rbe.mdx: the script was setting publicName to `:8981` (no hostname), causing every findMissingBlobs call to fail with "Invalid DNS name: :8981". Fixed to use MY_IP=127.0.0.1 and rewrite publicName dynamically on each boot. LaunchDaemon plist updated to match actual script path and add UserName/WorkingDirectory. Closes DI-574 Co-Authored-By: Claude Sonnet 4.6 --- .../bazel-rbe-quickstart.mdx | 267 ++++++++++++++++++ orka/orka-devops-integrations/bazel-rbe.mdx | 29 +- 2 files changed, 287 insertions(+), 9 deletions(-) create mode 100644 orka/orka-devops-integrations/bazel-rbe-quickstart.mdx diff --git a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx new file mode 100644 index 00000000..a3a0d707 --- /dev/null +++ b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx @@ -0,0 +1,267 @@ +--- +title: "Bazel RBE + Orka: Getting Started" +description: "How to set up a Bazel Remote Build Execution proof of concept with Orka-hosted macOS workers: server setup, worker image preparation, and running your first remote build." +--- + +This guide walks through a working PoC setup: a remote execution server, an Orka-hosted macOS worker registered against it, and a Bazel client that offloads actions to that worker. By the end, you can run a build, confirm actions executed remotely, and verify cache hits on a clean rebuild. + +The RBE server is [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source remote execution implementation maintained by the Bazel team. Everything in this guide runs on a single Orka VM: Redis, the buildfarm server, and the worker all communicate over localhost, which removes the cross-node networking complexity that a production setup would need to address. + + +**Client compatibility.** Bazel 8.x and 9.x both work as clients against buildfarm. The `.bazelversion` file in the buildfarm source repo reflects what Bazel version is used to *build buildfarm itself* — that's separate from which version your project uses. The guide uses Bazel 8.2.1 for the client. + + +## What you'll need + +- An Orka 3.x cluster with at least one arm64 node +- A macOS base image in your Orka cluster with Remote Login (SSH) enabled +- Bazelisk installed on the machine where you'll run Bazel: `brew install bazelisk` + +No Linux host, no Docker. The macOS VM handles everything. + +## How it fits together + +Three processes run on a single Orka VM: + +1. **Redis** stores the Content Addressable Storage (CAS) and Action Cache (AC). +2. **buildfarm server** accepts remote execution requests from Bazel clients and schedules actions. +3. **buildfarm shard worker** executes build actions and reports results back to the server. + +The Bazel client connects to the server from outside the VM over an SSH tunnel. + +## Step 1: Deploy the VM + +```bash +orka3 vm deploy buildfarm --image +``` + +Note the SSH port from the output. All subsequent steps SSH into this VM. + +## Step 2: Install dependencies + +```bash +# Homebrew (non-interactive) +NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +eval "$(/opt/homebrew/bin/brew shellenv)" + +# Java 21 — buildfarm compiles to class version 65 (Java 21), so 17 is not sufficient +brew install --cask temurin@21 + +# Bazelisk (used to build buildfarm from source) and Redis +brew install bazelisk redis +``` + +Start Redis bound to localhost: + +```bash +redis-server /opt/homebrew/etc/redis.conf --daemonize yes +redis-cli ping # should return PONG +``` + +## Step 3: Build buildfarm from source + +buildfarm does not publish pre-built JARs. You build both the server and worker JARs from source using Bazelisk. Use the system git — the Homebrew-installed version has a libcurl mismatch on Sonoma. + +```bash +mkdir -p ~/buildfarm && cd ~/buildfarm +/usr/bin/git clone --depth=1 https://github.com/bazelbuild/bazel-buildfarm.git +cd bazel-buildfarm +``` + +Build both JARs. The `--host_copt` and `--copt` flags are required to work around a macOS deployment target issue in abseil-cpp and protobuf: + +```bash +bazel build \ + //src/main/java/build/buildfarm:buildfarm-server_deploy.jar \ + //src/main/java/build/buildfarm:buildfarm-shard-worker_deploy.jar \ + --host_copt=-mmacosx-version-min=10.15 \ + --copt=-mmacosx-version-min=10.15 +``` + +This takes 3 to 5 minutes on first run. Copy the JARs out of the build cache: + +```bash +cp bazel-bin/src/main/java/build/buildfarm/buildfarm-server_deploy.jar ~/buildfarm/ +cp bazel-bin/src/main/java/build/buildfarm/buildfarm-shard-worker_deploy.jar ~/buildfarm/ +``` + +## Step 4: Configure and start the server + +Create `~/buildfarm/server-config.yml`: + +```yaml +digestFunction: SHA256 +defaultActionTimeout: 600 +maximumActionTimeout: 3600 +server: + instanceType: SHARD + name: shard + port: 8980 +backplane: + type: SHARD + redisUri: redis://127.0.0.1:6379 + queues: + - name: cpu + allowUnmatched: true + properties: + - name: min-cores + value: '*' + - name: max-cores + value: '*' +``` + + +The backplane type is `SHARD`, not `REDIS`. The config format uses a flat YAML file — the `!include`-based `config.minimal.yml` in the buildfarm examples directory is a dev preprocessor format the server does not support at runtime. + + +Start the server: + +```bash +export JAVA_HOME=$(/usr/libexec/java_home -v 21) +cd ~/buildfarm +nohup $JAVA_HOME/bin/java -jar buildfarm-server_deploy.jar server-config.yml > server.log 2>&1 & +``` + +Confirm it initialized: + +```bash +grep "initialized" ~/buildfarm/server.log +# Jun 04, 2026 ... buildfarm-server-127.0.0.1:8980-... initialized +``` + +## Step 5: Configure and start the worker + +Create the macOS execution wrapper. buildfarm requires a wrapper script around action invocations; on macOS, a passthrough is sufficient: + +```bash +printf '#!/bin/bash\nexec "$@"\n' > ~/buildfarm/macos-wrapper.sh +chmod +x ~/buildfarm/macos-wrapper.sh +``` + +Create `~/buildfarm/worker-config.yml`: + +```yaml +digestFunction: SHA256 +defaultActionTimeout: 600 +maximumActionTimeout: 3600 +backplane: + type: SHARD + redisUri: redis://127.0.0.1:6379 + queues: + - name: cpu + allowUnmatched: true + properties: + - name: min-cores + value: '*' + - name: max-cores + value: '*' +worker: + port: 8981 + publicName: 127.0.0.1:8981 + root: /tmp/worker + dequeueMatchSettings: + allowUnmatched: true + storages: + - type: FILESYSTEM + path: /tmp/worker/cache + maxSizeBytes: 2147483648 + fileDirectoriesIndexInMemory: false + skipLoad: false + hexBucketLevels: 0 + execRootCopyFallback: false + executeStageWidth: 1 + inputFetchStageWidth: 1 + inputFetchDeadline: 60 + reportResultStageWidth: 1 + linkExecFileSystem: false + linkInputDirectories: false + executionPolicies: + - name: test + executionWrapper: + path: /Users/admin/buildfarm/macos-wrapper.sh + arguments: +``` + +Start the worker: + +```bash +mkdir -p /tmp/worker/cache +nohup $JAVA_HOME/bin/java -jar ~/buildfarm/buildfarm-shard-worker_deploy.jar ~/buildfarm/worker-config.yml > ~/buildfarm/worker.log 2>&1 & +``` + +Confirm it registered: + +```bash +grep "initialized" ~/buildfarm/worker.log +# Jun 04, 2026 ... buildfarm-worker-127.0.0.1:8981-... initialized +``` + +Warnings about missing `process-wrapper`, `linux-sandbox`, and cgroups are expected on macOS and do not affect functionality. A Prometheus port conflict on 9090 (already used by the server) is also harmless. + +## Step 6: Connect your Bazel client + +Port 8980 is not forwarded by Orka's default port mapping. Open an SSH tunnel from the machine where you'll run Bazel: + +```bash +ssh -N -L 8980:127.0.0.1:8980 -p admin@ +``` + +Leave the tunnel running. In a separate terminal, add to your project's `.bazelrc`: + +``` +build --remote_executor=grpc://localhost:8980 +build --jobs=10 +``` + +Pin your Bazel version in `.bazelversion`: + +``` +8.2.1 +``` + +## Step 7: Run a build and verify + +```bash +bazel build --verbose_failures //:your_target +``` + +In the output, the `processes:` line shows how many actions ran remotely: + +``` +INFO: 18 processes: 10 internal, 8 remote. +``` + +To confirm cache hits, run a full clean and rebuild: + +```bash +bazel clean --expunge +bazel build //:your_target +``` + +The second build should complete in a few seconds with all actions restored from the remote cache: + +``` +INFO: 18 processes: 8 remote cache hit, 10 internal. +``` + +You'll also see a deprecation warning about the buildfarm server's supported API version — this is harmless for a PoC and does not affect correctness. + +## Where to go from here + +**Automate startup.** Add launchd plists to the VM for Redis, the server, and the worker so all three start on boot. Save the configured VM as an Orka image and it becomes your baseline. + +**Scale workers.** Once the server image is stable, the worker process is the unit you want to multiply. Deploy additional VMs from the same image on the same node (they share the `192.168.64.x` subnet and can reach the server). On M4 Pro hardware, each node supports two concurrent VMs. + +**Cross-node networking.** Orka VMs on different physical nodes cannot reach each other's internal IPs — each host runs its own NAT subnet. For a multi-node worker pool, you'll need either a VPN across your cluster, or a dedicated node that runs the server and all workers together. This is the main architectural decision to resolve before moving to production. + +**Production cache backend.** Redis on localhost is sufficient for a PoC. For production, use a Redis cluster or managed equivalent sized for your artifact volume. + +**Warm pool.** Use an Orka VM config to keep a declared number of worker VMs running. The Orka operator maintains the pool size without manual dispatch. + +## Related + +- [Orka + Bazel RBE: Architecture and positioning](/orka/orka-devops-integrations/bazel-rbe) +- [OCI images overview](/orka/oci-images/oci-images-overview) +- [Image caching](/orka/managing-vms/image-caching) +- [VM configs](/orka/managing-vms/vm-configs) diff --git a/orka/orka-devops-integrations/bazel-rbe.mdx b/orka/orka-devops-integrations/bazel-rbe.mdx index 8ef0fb96..43d424a2 100644 --- a/orka/orka-devops-integrations/bazel-rbe.mdx +++ b/orka/orka-devops-integrations/bazel-rbe.mdx @@ -433,18 +433,27 @@ The manual steps above work for static worker pools that stay running. For ephem The key to making ephemeral workers viable is having the worker process start automatically when the VM boots, so the GitHub Actions workflow doesn't need to SSH into each VM to configure it. Do this once when building your base image, then save it. -**Create a startup script** at `/Users/admin/start-buildfarm-worker.sh`: +**Create a startup script** at `/Users/admin/buildfarm/start-worker.sh`: ```bash #!/bin/bash -export JAVA_HOME=$(/usr/libexec/java_home -v 21) -exec $JAVA_HOME/bin/java -jar /Users/admin/buildfarm/buildfarm-shard-worker_deploy.jar /Users/admin/buildfarm/worker-config.yml +MY_IP=127.0.0.1 +sed -i "" "s|publicName:.*|publicName: ${MY_IP}:8981|" /Users/admin/buildfarm/worker-config.yml +until nc -z 127.0.0.1 8980 2>/dev/null; do sleep 2; done +mkdir -p /tmp/worker/cache /Users/admin/buildfarm/tmp +exec /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java \ + -Djava.net.preferIPv4Stack=true \ + -Djava.io.tmpdir=/Users/admin/buildfarm/tmp \ + -jar /Users/admin/buildfarm/buildfarm-shard-worker_deploy.jar \ + /Users/admin/buildfarm/worker-config.yml ``` ```bash -chmod +x /Users/admin/start-buildfarm-worker.sh +chmod +x /Users/admin/buildfarm/start-worker.sh ``` +The script rewrites `publicName` on each boot to match `MY_IP`. This matters: if `publicName` is set to just `:8981` (no hostname), the server will fail every `findMissingBlobs` call with `Invalid DNS name: :8981`. For a single-node setup where the server and worker share a VM, `127.0.0.1` is correct. For a multi-node setup where workers are on separate VMs, set `MY_IP` to the IP the server can reach the worker on. + **Create a LaunchDaemon** at `/Library/LaunchDaemons/io.macstadium.buildfarm-worker.plist`: ```xml @@ -454,21 +463,23 @@ chmod +x /Users/admin/start-buildfarm-worker.sh Label io.macstadium.buildfarm-worker + UserName + admin + WorkingDirectory + /Users/admin/buildfarm ProgramArguments /bin/bash - /Users/admin/start-buildfarm-worker.sh + /Users/admin/buildfarm/start-worker.sh RunAtLoad KeepAlive - WorkingDirectory - /Users/admin/bazel-buildfarm StandardOutPath - /var/log/buildfarm-worker.log + /Users/admin/buildfarm/worker.log StandardErrorPath - /var/log/buildfarm-worker.log + /Users/admin/buildfarm/worker.log ``` From c7d444cb7dddb2d76b8f2b7dd4f164a402555893 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Thu, 2 Jul 2026 16:04:09 -0500 Subject: [PATCH 5/8] Add bazel-rbe-quickstart to nav Co-Authored-By: Claude Sonnet 4.6 --- docs.json | 1 + 1 file changed, 1 insertion(+) diff --git a/docs.json b/docs.json index 34279516..ae52bfe1 100644 --- a/docs.json +++ b/docs.json @@ -173,6 +173,7 @@ "orka/orka-devops-integrations/teamcity", "orka/orka-devops-integrations/packer", "orka/orka-devops-integrations/bazel-rbe", + "orka/orka-devops-integrations/bazel-rbe-quickstart", "orka/orka-devops-integrations/claude-code" ] }, From 6b7b6f14da59609ded8c4adcaf338d27167af4e1 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Wed, 8 Jul 2026 11:07:12 -0500 Subject: [PATCH 6/8] Fix GHA workflow auth/flags, Harbor claim, Bazel note, em dashes - Fix GitHub Actions workflow: orka3 user set-token for auth, --config flag (not --vm-config), remove non-existent --endpoint/--token flags - Correct Bazel version note: .bazelversion pin is for building buildfarm from source, not for the client (8.x and 9.x both work as clients) - Replace Harbor OCI registry reference with accurate language: Orka supports any OCI-compatible registry (ghcr.io, ECR, etc.) - Remove all em dashes from both pages Co-Authored-By: Claude Sonnet 4.6 --- .../bazel-rbe-quickstart.mdx | 12 ++-- orka/orka-devops-integrations/bazel-rbe.mdx | 66 ++++++++----------- 2 files changed, 33 insertions(+), 45 deletions(-) diff --git a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx index a3a0d707..635f7f06 100644 --- a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx +++ b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx @@ -8,7 +8,7 @@ This guide walks through a working PoC setup: a remote execution server, an Orka The RBE server is [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm), an open-source remote execution implementation maintained by the Bazel team. Everything in this guide runs on a single Orka VM: Redis, the buildfarm server, and the worker all communicate over localhost, which removes the cross-node networking complexity that a production setup would need to address. -**Client compatibility.** Bazel 8.x and 9.x both work as clients against buildfarm. The `.bazelversion` file in the buildfarm source repo reflects what Bazel version is used to *build buildfarm itself* — that's separate from which version your project uses. The guide uses Bazel 8.2.1 for the client. +**Client compatibility.** Bazel 8.x and 9.x both work as clients against buildfarm. The `.bazelversion` file in the buildfarm source repo controls what Bazel version is used to *build buildfarm itself*, which is separate from which version your project uses. The guide uses Bazel 8.2.1 for the client. ## What you'll need @@ -45,7 +45,7 @@ NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Ho eval "$(/opt/homebrew/bin/brew shellenv)" -# Java 21 — buildfarm compiles to class version 65 (Java 21), so 17 is not sufficient +# Java 21 required: buildfarm compiles to class version 65 (Java 21), so 17 is not sufficient brew install --cask temurin@21 # Bazelisk (used to build buildfarm from source) and Redis @@ -61,7 +61,7 @@ redis-cli ping # should return PONG ## Step 3: Build buildfarm from source -buildfarm does not publish pre-built JARs. You build both the server and worker JARs from source using Bazelisk. Use the system git — the Homebrew-installed version has a libcurl mismatch on Sonoma. +buildfarm does not publish pre-built JARs. You build both the server and worker JARs from source using Bazelisk. Use the system git; the Homebrew-installed version has a libcurl mismatch on Sonoma. ```bash mkdir -p ~/buildfarm && cd ~/buildfarm @@ -112,7 +112,7 @@ backplane: ``` -The backplane type is `SHARD`, not `REDIS`. The config format uses a flat YAML file — the `!include`-based `config.minimal.yml` in the buildfarm examples directory is a dev preprocessor format the server does not support at runtime. +The backplane type is `SHARD`, not `REDIS`. The config format uses a flat YAML file. The `!include`-based `config.minimal.yml` in the buildfarm examples directory is a dev preprocessor format the server does not support at runtime. Start the server: @@ -245,7 +245,7 @@ The second build should complete in a few seconds with all actions restored from INFO: 18 processes: 8 remote cache hit, 10 internal. ``` -You'll also see a deprecation warning about the buildfarm server's supported API version — this is harmless for a PoC and does not affect correctness. +You'll also see a deprecation warning about the buildfarm server's supported API version. This is harmless for a PoC and does not affect correctness. ## Where to go from here @@ -253,7 +253,7 @@ You'll also see a deprecation warning about the buildfarm server's supported API **Scale workers.** Once the server image is stable, the worker process is the unit you want to multiply. Deploy additional VMs from the same image on the same node (they share the `192.168.64.x` subnet and can reach the server). On M4 Pro hardware, each node supports two concurrent VMs. -**Cross-node networking.** Orka VMs on different physical nodes cannot reach each other's internal IPs — each host runs its own NAT subnet. For a multi-node worker pool, you'll need either a VPN across your cluster, or a dedicated node that runs the server and all workers together. This is the main architectural decision to resolve before moving to production. +**Cross-node networking.** Orka VMs on different physical nodes cannot reach each other's internal IPs: each host runs its own NAT subnet. For a multi-node worker pool, you'll need either a VPN across your cluster or a dedicated node that runs the server and all workers together. This is the main architectural decision to resolve before moving to production. **Production cache backend.** Redis on localhost is sufficient for a PoC. For production, use a Redis cluster or managed equivalent sized for your artifact volume. diff --git a/orka/orka-devops-integrations/bazel-rbe.mdx b/orka/orka-devops-integrations/bazel-rbe.mdx index 43d424a2..528f4717 100644 --- a/orka/orka-devops-integrations/bazel-rbe.mdx +++ b/orka/orka-devops-integrations/bazel-rbe.mdx @@ -1,9 +1,9 @@ --- title: "Bazel Remote Build Execution with Orka" -description: "Run macOS Bazel RBE workers on Orka VMs. Use open-source buildfarm or BuildBuddy as the RBE control plane — Orka provides the macOS worker fleet either way." +description: "Run macOS Bazel RBE workers on Orka VMs. Use open-source buildfarm or BuildBuddy as the RBE control plane. Orka provides the macOS worker fleet either way." --- -Bazel Remote Build Execution (RBE) splits a build into two distinct layers: the control plane, which schedules actions, manages the cache, and routes work to available workers; and the worker pool, which executes the actions on real hardware. For iOS and macOS builds, the worker pool has to be macOS — and that's where Orka fits. +Bazel Remote Build Execution (RBE) splits a build into two distinct layers: the control plane, which schedules actions, manages the cache, and routes work to available workers; and the worker pool, which executes the actions on real hardware. For iOS and macOS builds, the worker pool has to be macOS. That's where Orka fits. This page covers two paths: using open-source [bazel-buildfarm](https://github.com/bazelbuild/bazel-buildfarm) as the control plane, and using [BuildBuddy](https://www.buildbuddy.io/) as the control plane. Both use Orka VMs as the macOS worker fleet. @@ -14,25 +14,25 @@ This page covers two paths: using open-source [bazel-buildfarm](https://github.c | | gRPC (remote-apis protocol) v -[RBE server — Linux VM] +[RBE server (Linux VM)] bazel-buildfarm or BuildBuddy Redis backplane (buildfarm only) | | dispatch to workers v -[Orka macOS VMs — worker pool] +[Orka macOS VMs (worker pool)] Xcode, Homebrew, Java, Bazelisk bazel-buildfarm worker or BuildBuddy executor | v -[Remote cache — Linux VM or S3] +[Remote cache (Linux VM or S3)] bazel-remote, buildfarm CAS, or BuildBuddy cache ``` The RBE server and cache typically run on a Linux VM co-located in your Orka cluster at MacStadium. The macOS workers are Orka VMs deployed from a versioned OCI image. -Bazel 9.0.0 is not compatible with bazel-buildfarm as of February 2026. Pin your workers and client to Bazel 8.x by running `echo "8.2.1" > .bazelversion` in your buildfarm directory before building. This does not affect BuildBuddy setups. +The `.bazelversion` file in the bazel-buildfarm source repo controls which Bazel version is used to **build buildfarm itself**. Pin it to 8.x before building the JARs. Your project's Bazel client version (8.x or 9.x) is unaffected. This does not apply to BuildBuddy setups. --- @@ -132,7 +132,7 @@ backplane: ``` -The backplane type is `SHARD`, not `REDIS`. Use a flat YAML config — the `config.minimal.yml` in the examples directory uses a preprocessor format the server does not support at runtime. +The backplane type is `SHARD`, not `REDIS`. Use a flat YAML config. The `config.minimal.yml` in the examples directory uses a preprocessor format the server does not support at runtime. **Start the buildfarm server:** @@ -279,7 +279,7 @@ Because all workers use the same base image and the same buildfarm server addres A reasonable starting point for iOS simulator workloads is one worker VM per Orka node, with each VM sized to leave headroom for simulator processes. For compile-bound Bazel actions, higher VM density per node is fine since those workloads are CPU-bound rather than I/O-bound. -If you're running parallel simulator tests across multiple VMs on the same node and see I/O slowdowns, the bottleneck is disk contention from concurrent simulator writes — not virtualization overhead. Distribute across more nodes at lower VM-per-node density rather than reducing the total VM count. +If you're running parallel simulator tests across multiple VMs on the same node and see I/O slowdowns, the bottleneck is disk contention from concurrent simulator writes, not virtualization overhead. Distribute across more nodes at lower VM-per-node density rather than reducing the total VM count. --- @@ -316,12 +316,12 @@ INFO: Build completed successfully, 65 total actions ## Option 2: BuildBuddy with Orka -If your team is already using BuildBuddy as your RBE control plane, Orka VMs can serve as the self-hosted Mac executors that BuildBuddy dispatches work to. This keeps Orka in the stack as the macOS worker layer — you get clean VM boots, OCI image versioning, and node failover even when BuildBuddy is handling scheduling and caching. +If your team is already using BuildBuddy as your RBE control plane, Orka VMs can serve as the self-hosted Mac executors that BuildBuddy dispatches work to. This keeps Orka in the stack as the macOS worker layer: clean VM boots, OCI image versioning, and node failover even when BuildBuddy is handling scheduling and caching. For teams evaluating options, the OSS buildfarm path above is the recommended starting point. It gives you full control over the stack, no additional vendor dependency, and the same Orka worker benefits. -BuildBuddy's snapshot-based warm-runner technology (Firecracker) is Linux-only. On macOS, BuildBuddy falls back to runner recycling — keeping the executor process alive between invocations rather than snapshotting it. Orka VM snapshots work independently of this and are available regardless of which RBE control plane you use. +BuildBuddy's snapshot-based warm-runner technology (Firecracker) is Linux-only. On macOS, BuildBuddy falls back to runner recycling, keeping the executor process alive between invocations rather than snapshotting it. Orka VM snapshots work independently of this and are available regardless of which RBE control plane you use. ### What you need @@ -427,7 +427,7 @@ BuildBuddy's web UI shows build results, cache hit rates, and action timing at [ ## Automate worker lifecycle with GitHub Actions -The manual steps above work for static worker pools that stay running. For ephemeral workers that spin up per build and tear down when it's done, you can drive the whole thing from a GitHub Actions workflow. This pattern works for both the buildfarm and BuildBuddy setups — the difference is just what's installed on the worker image. +The manual steps above work for static worker pools that stay running. For ephemeral workers that spin up per build and tear down when it's done, you can drive the whole thing from a GitHub Actions workflow. This pattern works for both the buildfarm and BuildBuddy setups; the difference is just what's installed on the worker image. ### Set up worker auto-start in the base image @@ -452,7 +452,7 @@ exec /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home/bin/java \ chmod +x /Users/admin/buildfarm/start-worker.sh ``` -The script rewrites `publicName` on each boot to match `MY_IP`. This matters: if `publicName` is set to just `:8981` (no hostname), the server will fail every `findMissingBlobs` call with `Invalid DNS name: :8981`. For a single-node setup where the server and worker share a VM, `127.0.0.1` is correct. For a multi-node setup where workers are on separate VMs, set `MY_IP` to the IP the server can reach the worker on. +The script rewrites `publicName` on each boot to match `MY_IP`. This matters: if `publicName` is set to just `:8981` (no hostname), the server will fail every `findMissingBlobs` call with `Invalid DNS name: :8981`. For a single-node setup where the server and worker share a VM, `127.0.0.1` is correct. For a multi-node setup where workers are on separate VMs, set `MY_IP` to the IP the server can reach that worker on. **Create a LaunchDaemon** at `/Library/LaunchDaemons/io.macstadium.buildfarm-worker.plist`: @@ -490,13 +490,13 @@ sudo launchctl load /Library/LaunchDaemons/io.macstadium.buildfarm-worker.plist With this in place, every VM deployed from this image will register with your buildfarm server within about 30 seconds of boot. Save the image in Orka before continuing. -For BuildBuddy setups, the LaunchDaemon above already covers auto-start — no separate script needed since the BuildBuddy LaunchDaemon is set up in the executor image prep steps above. +For BuildBuddy setups, the LaunchDaemon above already covers auto-start. No separate script is needed since the BuildBuddy LaunchDaemon is set up in the executor image prep steps above. --- ### The workflow -The workflow runs on a self-hosted runner that has access to your Orka cluster — the Linux VM running your buildfarm server is the natural home for it, since it's already in the same network as your Orka nodes. Store your Orka service account name and token as GitHub Actions secrets. +The workflow runs on a self-hosted runner that has access to your Orka cluster. The Linux VM running your buildfarm server is the natural home for it, since it's already in the same network as your Orka nodes. Store your Orka service account token as a GitHub Actions secret (`ORKA_SA_TOKEN`). The runner must have `orka3` installed and configured to point at your cluster (`orka3 config set --api-url `). ```yaml name: Bazel RBE @@ -519,13 +519,7 @@ jobs: - uses: actions/checkout@v4 - name: Authenticate with Orka - id: auth - run: | - TOKEN=$(orka3 serviceaccount token "${{ secrets.ORKA_SA_NAME }}" \ - --endpoint "${{ vars.ORKA_ENDPOINT }}") - echo "token=$TOKEN" >> "$GITHUB_OUTPUT" - env: - ORKA_SA_TOKEN: ${{ secrets.ORKA_SA_TOKEN }} + run: orka3 user set-token "${{ secrets.ORKA_SA_TOKEN }}" - name: Deploy worker VMs id: workers @@ -534,10 +528,8 @@ jobs: for i in $(seq 1 "$WORKER_COUNT"); do NAME="bazel-worker-${{ github.run_id }}-${i}" orka3 vm deploy \ - --vm-config "$WORKER_VM_CONFIG" \ - --name "$NAME" \ - --endpoint "${{ vars.ORKA_ENDPOINT }}" \ - --token "${{ steps.auth.outputs.token }}" + --config "$WORKER_VM_CONFIG" \ + --name "$NAME" NAMES+=("$NAME") done echo "names=${NAMES[*]}" >> "$GITHUB_OUTPUT" @@ -547,9 +539,7 @@ jobs: echo "Waiting for VMs to boot and register with buildfarm..." sleep 45 for NAME in ${{ steps.workers.outputs.names }}; do - orka3 vm list "$NAME" \ - --endpoint "${{ vars.ORKA_ENDPOINT }}" \ - --token "${{ steps.auth.outputs.token }}" + orka3 vm list "$NAME" done - name: Run Bazel build @@ -563,25 +553,23 @@ jobs: if: always() run: | for NAME in ${{ steps.workers.outputs.names }}; do - orka3 vm delete "$NAME" \ - --endpoint "${{ vars.ORKA_ENDPOINT }}" \ - --token "${{ steps.auth.outputs.token }}" || true + orka3 vm delete "$NAME" || true done ``` A few things worth noting: -- **Service account auth** — `orka3 serviceaccount token` is the right auth method for CI/CD. User tokens from `orka3 login` expire after an hour and will fail mid-build. See [Manage service accounts](/orka/orka-cluster-access/orka-cluster-manage-service-accounts) for setup. -- **Worker count** — `WORKER_COUNT` is the lever for parallelism. Start at 2–4 and increase if your build has enough parallelizable actions to justify it. Each VM adds to your cluster's VM slot consumption. -- **The 45-second wait** — covers VM boot plus the time for the LaunchDaemon to start the buildfarm worker process and register with the server. Adjust down if your cluster consistently boots faster; adjust up if you see "no workers available" errors on the first few actions. -- **`if: always()` on teardown** — ensures VMs are cleaned up even when the build fails. Without this, a failed build leaves orphaned VMs running until someone deletes them manually. -- **Unique VM names** — appending `github.run_id` to the VM name avoids naming collisions when multiple builds run in parallel. +- **Service account auth:** Store the SA token as a long-lived secret (generated once with `orka3 sa token --no-expiration`). In CI, authenticate with `orka3 user set-token`. User tokens from `orka3 login` expire after an hour and will fail mid-build. See [Manage service accounts](/orka/orka-cluster-access/orka-cluster-manage-service-accounts) for setup. +- **Worker count:** `WORKER_COUNT` is the lever for parallelism. Start at 2-4 and increase if your build has enough parallelizable actions to justify it. Each VM adds to your cluster's VM slot consumption. +- **The 45-second wait:** Covers VM boot plus the time for the LaunchDaemon to start the buildfarm worker process and register with the server. Adjust down if your cluster consistently boots faster; adjust up if you see "no workers available" errors on the first few actions. +- **`if: always()` on teardown:** Ensures VMs are cleaned up even when the build fails. Without this, a failed build leaves orphaned VMs running until someone deletes them manually. +- **Unique VM names:** Appending `github.run_id` to the VM name avoids naming collisions when multiple builds run in parallel. --- ### Scale across multiple runs -If your team runs many concurrent builds, the `WORKER_COUNT` per workflow multiplies by the number of parallel runs. Watch your cluster's available VM slots — the Orka web UI shows current VM counts per node. For sustained high-concurrency usage, a persistent warm pool (always-on workers) is more efficient than deploying and tearing down per-build, since it avoids the per-build boot latency and avoids VM slot churn. +If your team runs many concurrent builds, the `WORKER_COUNT` per workflow multiplies by the number of parallel runs. Watch your cluster's available VM slots in the Orka web UI. For sustained high-concurrency usage, a persistent warm pool (always-on workers) is more efficient than deploying and tearing down per-build, since it avoids per-build boot latency and VM slot churn. --- @@ -603,7 +591,7 @@ Point your Bazel client at it: build --remote_cache=http://:9090 ``` -For buildfarm setups, the buildfarm server also provides a Content-Addressable Store (CAS) — you don't need a separate cache server unless you want to share the cache with clients that bypass RBE. +For buildfarm setups, the buildfarm server also provides a Content-Addressable Store (CAS). You don't need a separate cache server unless you want to share the cache with clients that bypass RBE. --- @@ -612,7 +600,7 @@ For buildfarm setups, the buildfarm server also provides a Content-Addressable S Bazel RBE handles scheduling, caching, and build orchestration. What it doesn't provide is the macOS worker fleet itself. Running Bazel RBE on macOS requires clean, versioned, isolated Mac workers that don't turn into an ops problem at scale. That's what Orka handles. - **Clean workers on demand.** Deploying a fresh Orka VM from a versioned OCI image takes seconds. Bare-metal re-imaging takes hours: DEP enrollment, a full Xcode install, re-configuration. At CI cadence, that's not a viable clean-boot story. -- **Pinned build environments.** Bazel pins toolchain state into the action hash. If your worker's Xcode version or SDK drifts, cache hits collapse. Orka 3.5's Harbor OCI registry lets you version, sign, and pin your build images the same way you'd pin a container image. Bare metal has no equivalent primitive. +- **Pinned build environments.** Bazel pins toolchain state into the action hash. If your worker's Xcode version or SDK drifts, cache hits collapse. Orka's OCI image support lets you version and pin your build environments using any OCI-compatible registry (ghcr.io, ECR, etc.), the same way you'd pin a container image. Bare metal has no equivalent primitive. - **Isolation per project or team.** VMs are isolated by definition. If one build poisons a worker, it doesn't affect other workers. On bare metal, a bad build corrupts the host until someone re-images it. - **Worker pool scaling without physical hardware.** Deploy more workers from the same base image. Orka's Kubernetes-native operator supports declarative pool sizing. - **Node failover.** If an Orka node goes down, VMs reschedule to healthy nodes. A bare-metal host failure is a manual remediation. From c22f458ddbd86a17a8167e4e0849f3f6078b2283 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Wed, 8 Jul 2026 11:17:06 -0500 Subject: [PATCH 7/8] Fix cross-node networking claim: bridge networking supports cross-node Orka 3.5+ bridge networking gives VMs a routable IP on the host network, so they can reach each other across nodes directly. The previous note implied cross-node connectivity was always blocked (NAT-only assumption). Updated to cover both bridge and NAT mode and point to support for cluster network config confirmation. Co-Authored-By: Claude Sonnet 4.6 --- orka/orka-devops-integrations/bazel-rbe-quickstart.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx index 635f7f06..0873dda9 100644 --- a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx +++ b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx @@ -253,7 +253,7 @@ You'll also see a deprecation warning about the buildfarm server's supported API **Scale workers.** Once the server image is stable, the worker process is the unit you want to multiply. Deploy additional VMs from the same image on the same node (they share the `192.168.64.x` subnet and can reach the server). On M4 Pro hardware, each node supports two concurrent VMs. -**Cross-node networking.** Orka VMs on different physical nodes cannot reach each other's internal IPs: each host runs its own NAT subnet. For a multi-node worker pool, you'll need either a VPN across your cluster or a dedicated node that runs the server and all workers together. This is the main architectural decision to resolve before moving to production. +**Cross-node networking.** With bridge networking enabled (Orka 3.5+), VMs get a routable IP on the host network and can reach each other across nodes directly -- straightforward for a multi-node buildfarm setup. If your cluster is running in NAT mode, VMs on different nodes cannot reach each other's internal IPs; in that case you'll need either a VPN across your cluster or a dedicated node that runs the server and all workers together. Contact MacStadium support to confirm your cluster's network configuration before planning a multi-node setup. **Production cache backend.** Redis on localhost is sufficient for a PoC. For production, use a Redis cluster or managed equivalent sized for your artifact volume. From 8daa926e46d2e109ba7631261edea8e5333d1325 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Thu, 9 Jul 2026 09:13:29 -0500 Subject: [PATCH 8/8] Fix broken internal links in bazel-rbe-quickstart /orka/managing-vms/ directory does not exist. Correct paths: - image-caching -> /orka/orka-resources/image-caching - vm-configs -> /orka/orka3-cli-reference/vm-deployment-and-configuration Co-Authored-By: Claude Sonnet 4.6 --- orka/orka-devops-integrations/bazel-rbe-quickstart.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx index 0873dda9..9a488e58 100644 --- a/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx +++ b/orka/orka-devops-integrations/bazel-rbe-quickstart.mdx @@ -263,5 +263,5 @@ You'll also see a deprecation warning about the buildfarm server's supported API - [Orka + Bazel RBE: Architecture and positioning](/orka/orka-devops-integrations/bazel-rbe) - [OCI images overview](/orka/oci-images/oci-images-overview) -- [Image caching](/orka/managing-vms/image-caching) -- [VM configs](/orka/managing-vms/vm-configs) +- [Image caching](/orka/orka-resources/image-caching) +- [VM configs](/orka/orka3-cli-reference/vm-deployment-and-configuration)