Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This README is the router. The contract details live in focused docs so humans a
| Find the workflow a user should trigger | [workflow-entrypoints.md](workflow-entrypoints.md) |
| Change reusable build, infra, deploy, or release workflow behavior | [reusable-workflows.md](reusable-workflows.md) |
| Change saved plan, apply-from-plan, artifact naming, or plan metadata behavior | [artifacts-and-plans.md](artifacts-and-plans.md) |
| Change directory discovery, service/container matrices, or Terragrunt graph waves | [discovery-and-matrices.md](discovery-and-matrices.md) |
| Change runtime manifests, action discovery, or Terragrunt graph waves | [discovery-and-matrices.md](discovery-and-matrices.md) |
| Change `.github/actions/**`, OIDC role ARN construction, release tagging, or action tests | [repo-local-actions.md](repo-local-actions.md) |
| Change destroy behavior, post-destroy cleanup, or tagged-resource sweeps | [destroy.md](destroy.md) |
| Review any workflow or CI contract change | [feasibility-checks.md](feasibility-checks.md) |
Expand Down
35 changes: 23 additions & 12 deletions .github/docs/discovery-and-matrices.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,34 @@
# Discovery And Matrices

Use this when changing directory discovery, runtime matrices, service/container naming, or Terragrunt graph waves.
Use this when changing runtime manifests, repo-local action discovery, or Terragrunt graph waves.

## Directory Discovery

`shared_directories_get.yml` derives directory-based matrices used by wrapper workflows and PR action-test discovery.
`shared_directories_get.yml` derives repo-local action matrices used by PR action-test discovery.

`service_dirs` and `container_dirs` are intentionally different:
Lambda discovery is manifest-based:

- `service_dirs` contains deployable ECS service image directories only.
- `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`.
- ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need shared sidecar images as well as service images.
- Workflows that only need app service names or task/service stack derivation should use `service_dirs`.
- `lambdas/deploy.yml` is the source of truth for Lambda build and deploy records.
- `shared_build.yml` derives unique Lambda source records from the manifest when it runs.
- `shared_deploy.yml` derives every Lambda deploy record from the manifest when it runs.
- wrapper workflows do not pass Lambda matrices; changing the Lambda deployment set is a `lambdas/deploy.yml` change.
- `stack` values are repo-relative Terragrunt stack path templates such as `infra/live/{environment}/aws/lambda_api`.
- `source_dir` values are repo-relative source paths; the artifact filename is computed from `basename(source_dir)`.

ECS discovery is manifest-based:

- `containers/deploy.yml` is the source of truth for ECS image build and service deploy records.
- `shared_build.yml` derives unique ECS image records from the manifest when it runs.
- `shared_deploy.yml` derives every ECS service deploy record from the manifest when it runs.
- wrapper workflows do not pass ECS or task matrices; changing the ECS deployment set is a `containers/deploy.yml` change.
- `task_stack` and `service_stack` values are repo-relative Terragrunt stack path templates such as `infra/live/{environment}/aws/task_api`.
- `image` is the ECR tag prefix and maps to the default Docker service source directory `containers/<image>`.
- `support_images` lists shared images such as `debug` and `otel_collector` that are built with ECS images because task definitions require them.

Top-level runtime discovery rules:

- top-level Lambda directories under `lambdas/` are deployable functions, excluding generated build output
- top-level deployable ECS service directories under `containers/` are exposed through `service_dirs`
- the broader `container_dirs` matrix includes deployable service directories plus shared sidecar image targets
- Lambda deployability is declared in `lambdas/deploy.yml`; top-level Lambda directories are not deploy targets unless the manifest references them
- ECS deployability is declared in `containers/deploy.yml`; top-level container directories are not deploy targets unless the manifest references them

## Module Discovery

Expand Down Expand Up @@ -50,7 +61,7 @@ Each wave contains only modules whose direct dependencies were satisfied by earl

## Runtime Coverage Checks

- If Lambda directories are auto-detected, confirm matching live Terragrunt stacks still exist.
- If ECS directories are auto-detected, confirm matching `task_*` and `service_*` live Terragrunt stacks still exist.
- If Lambda manifest entries change, confirm each `stack` path exists for every deployed environment and each `source_dir` still builds.
- If ECS manifest entries change, confirm each `task_stack` and `service_stack` path exists for every deployed environment and each `image` source still builds.
- For `*_code` wrappers, confirm dispatch inputs cover every runtime being deployed.
- If ECS deploys are included, confirm `ecs_version` is exposed or intentionally derived.
4 changes: 2 additions & 2 deletions .github/docs/feasibility-checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ Run these checks on every CI, workflow, or deploy-contract change.

## Runtime Coverage

- If Lambda directories are auto-detected, confirm matching live Terragrunt stacks still exist.
- If ECS directories are auto-detected, confirm matching `task_*` and `service_*` live Terragrunt stacks still exist.
- If Lambda manifest entries change, confirm each `source_dir` exists, each computed zip basename is unique, and each `stack` path exists for every deployed environment.
- If ECS manifest entries change, confirm each `image` source exists and each `task_stack` / `service_stack` path exists for every deployed environment.
- For `*_code` wrappers, confirm dispatch inputs cover every runtime being deployed.
- If ECS deploys are included, confirm `ecs_version` is exposed or intentionally derived.

Expand Down
18 changes: 12 additions & 6 deletions .github/docs/reusable-workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ Use this when editing shared workflows under `.github/workflows/shared_*.yml` or
- Its `check` job normally runs `.github/actions/get-changes` using the PR base SHA for a PR-style `base...HEAD` diff.
- Manual `workflow_dispatch` runs force every change flag on and rerun the full validation surface without a PR diff.
- When `.github/actions/**` changed, it reuses `shared_directories_get.yml` to discover action directories with `Dockerfile`s and runs a Docker unit-test matrix after GitHub formatting.
- Lambda naming checks only run when Lambda sources changed.
- ECS task/service pair checks run when container sources or Terragrunt live-stack directories changed.
- Lambda naming and ECS task/service pair checks are explicit prerequisites for the corresponding build jobs.
- Lambda manifest validation only runs when Lambda sources changed, and checks both the deploy manifest and matching live stack paths.
- ECS manifest validation runs when container sources or Terragrunt live-stack directories changed, and checks both the deploy manifest and task/service stack pairs.
- Lambda and ECS manifest validation are explicit prerequisites for the corresponding build jobs.
- Terragrunt installation uses `jdx/mise-action@v4`.
- TFLint setup uses the Node 24 `terraform-linters/setup-tflint@v6` line.

Expand All @@ -40,10 +40,15 @@ The local version action can also be tested outside GitHub Actions by running th

`shared_build.yml` builds and publishes frontend, Lambda, and ECS artifacts.

`shared_build_get.yml` resolves artifact locations and derives matrices used by downstream deploy wrappers.
- Lambda builds are derived internally from `lambdas/deploy.yml`; callers do not pass a Lambda matrix.
- ECS image builds are derived internally from `containers/deploy.yml`; callers do not pass an ECS/container matrix.

`shared_build_get.yml` resolves artifact locations used by downstream deploy wrappers.

- Its multi-step `images` and `lambdas` jobs configure AWS credentials once.
- Repeated `just` calls reuse that ambient session against the same account.
- Prod deploy resolution checks the computed Lambda zip keys from `lambdas/deploy.yml` exist in the shared code bucket.
- Prod deploy resolution checks the computed ECS image tags from `containers/deploy.yml` exist in ECR.

```mermaid
flowchart LR
Expand Down Expand Up @@ -84,8 +89,9 @@ Current infra selection comes from the Terragrunt dependency graph and derived w
`shared_deploy.yml` rolls out feature code.

- Publishes Lambda versions.
- Optionally invokes the `migrations` Lambda when it is part of the Lambda deploy matrix.
- Optionally runs reconciliation Lambdas.
- Derives Lambda deploy records internally from `lambdas/deploy.yml`; callers do not pass a Lambda matrix.
- Optionally invokes Lambdas whose deploy manifest entry sets `after_deploy: invoke`.
- Derives ECS deploy records internally from `containers/deploy.yml`; callers do not pass an ECS task matrix.
- Registers ECS task revisions.
- Updates ECS services.
- Optionally deploys frontend assets.
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/destroy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ permissions:
env:
TF_VAR_lambda_version: this
TF_VAR_image_uri: destroy-placeholder
TF_VAR_aws_otel_collector_image_uri: destroy-placeholder
TF_VAR_debug_image_uri: destroy-placeholder
TF_VAR_otel_collector_uri: destroy-placeholder
TF_VAR_debug_uri: destroy-placeholder
AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role
AWS_REGION: ${{ vars.AWS_REGION }}

Expand Down
28 changes: 3 additions & 25 deletions .github/workflows/dev_code_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,46 +9,24 @@ permissions:
contents: write

jobs:
setup:
name: Discover
uses: ./.github/workflows/shared_directories_get.yml

build:
name: Build
uses: ./.github/workflows/shared_build.yml
needs:
- setup
with:
environment: dev
lambda_version: ${{ github.sha }}
frontend_version: ${{ github.sha }}
ecs_version: ${{ github.sha }}
lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }}
ecs_matrix: ${{ needs.setup.outputs.container_dirs }}


get_build:
name: Resolve
needs: build
uses: ./.github/workflows/shared_build_get.yml
with:
environment: dev
lambda_version: ${{ needs.build.outputs.lambda_version }}
frontend_version: ${{ needs.build.outputs.frontend_version }}
ecs_version: ${{ needs.build.outputs.ecs_version }}

deploy:
name: Deploy
uses: ./.github/workflows/shared_deploy.yml
needs:
- setup
- build
- get_build
with:
environment: dev
lambda_version: ${{ needs.build.outputs.lambda_version }}
frontend_version: ${{ needs.build.outputs.frontend_version }}
code_bucket: ${{ needs.get_build.outputs.code_bucket }}
lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }}
task_matrix: ${{ needs.get_build.outputs.ecs_task_matrix }}
ecs_image_uris: ${{ needs.get_build.outputs.ecs_image_uris }}
ecs_version: ${{ needs.build.outputs.ecs_version }}
code_bucket: ${{ needs.build.outputs.code_bucket }}
repository_url: ${{ needs.build.outputs.repository_url }}
8 changes: 2 additions & 6 deletions .github/workflows/prod_code_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,6 @@ jobs:
environment: prod
lambda_version: ${{ needs.get_build.outputs.lambda_version }}
frontend_version: ${{ needs.get_build.outputs.frontend_version }}
ecs_version: ${{ needs.get_build.outputs.ecs_version }}
code_bucket: ${{ needs.get_build.outputs.code_bucket }}
lambda_matrix: ${{ needs.get_build.outputs.lambda_version_files }}
task_matrix: ${{ needs.get_build.outputs.ecs_task_matrix }}
ecs_image_uris: ${{ needs.get_build.outputs.ecs_image_uris }}
# we can also scope the deployment here if needed, as below
# lambda_matrix: '["lambda_api"]'
# task_matrix: '["worker"]'
repository_url: ${{ needs.get_build.outputs.repository_url }}
98 changes: 73 additions & 25 deletions .github/workflows/pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,47 @@ jobs:
needs:
- check
- format-github
if: ${{ needs.check.outputs.lambdas == 'true' || needs.check.outputs.containers == 'true' || needs.check.outputs.actions == 'true' }}
if: ${{ needs.check.outputs.actions == 'true' }}
uses: ./.github/workflows/shared_directories_get.yml

lambda-build-matrix:
name: Lambda Build Records
needs:
- check
- check-lambda-manifest
if: ${{ needs.check.outputs.lambdas == 'true' }}
runs-on: ubuntu-latest
outputs:
lambda_build_matrix: ${{ steps.lambda_build_matrix.outputs.just_outputs }}
steps:
- uses: actions/checkout@v6

- name: Get Lambda Build Matrix
id: lambda_build_matrix
uses: ./.github/actions/just
with:
justfile_path: justfile.ci
just_action: lambda-get-build-matrix

ecs-build-matrix:
name: ECS Build Records
needs:
- check
- check-ecs-manifest
if: ${{ needs.check.outputs.containers == 'true' }}
runs-on: ubuntu-latest
outputs:
ecs_build_matrix: ${{ steps.ecs_build_matrix.outputs.just_outputs }}
steps:
- uses: actions/checkout@v6

- name: Get ECS Build Matrix
id: ecs_build_matrix
uses: ./.github/actions/just
with:
justfile_path: justfile.ci
just_action: ecs-get-build-matrix

action-docker-unit-tests:
name: GH Action Docker Unit Tests
needs:
Expand Down Expand Up @@ -229,30 +267,39 @@ jobs:
justfile_path: justfile.ci
just_action: tf-lint-check

check-lambda-naming:
check-lambda-manifest:
needs: check
runs-on: ubuntu-latest
if: ${{ needs.check.outputs.lambdas == 'true' }}
name: Lambda Pairs
name: Lambda Manifest
steps:
- uses: actions/checkout@v6
- name: Fail if any lambda directory uses hyphens
run: |
bad_dirs=$(find lambdas -mindepth 1 -maxdepth 1 -type d -name '*-*')
if [ -n "$bad_dirs" ]; then
echo "::error::❌ Lambda directories must use underscores, not hyphens: $bad_dirs"
exit 1
fi
echo "✅ All lambda directories use underscores."
- name: Validate Lambda deploy manifest
uses: ./.github/actions/just
with:
justfile_path: justfile.ci
just_action: lambda-get-deploy-matrix

check-ecs-module-pairs:
- name: Validate Lambda stack paths
uses: ./.github/actions/just
with:
justfile_path: justfile.ci
just_action: lambda-check-deploy-stacks

check-ecs-manifest:
needs: check
runs-on: ubuntu-latest
if: ${{ needs.check.outputs.containers == 'true' || needs.check.outputs.terragrunt == 'true' }}
name: ECS Pairs
name: ECS Manifest
steps:
- uses: actions/checkout@v6

- name: Validate ECS deploy manifest
uses: ./.github/actions/just
with:
justfile_path: justfile.ci
just_action: ecs-get-deploy-matrix

- name: Fail if task_/service_ pairs are incomplete
shell: bash
run: |
Expand Down Expand Up @@ -296,47 +343,48 @@ jobs:
echo "✅ All ECS task_/service_ pairs are present."

build-lambdas:
name: Build Lambdas
name: "${{ matrix.value.artifact_name }}"
if: ${{ needs.check.outputs.lambdas == 'true' }}
needs:
- check
- check-lambda-naming
- setup
- check-lambda-manifest
- lambda-build-matrix
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
value: ${{ fromJson(needs.setup.outputs.lambda_dirs) }}
value: ${{ fromJson(needs.lambda-build-matrix.outputs.lambda_build_matrix) }}
steps:
- uses: actions/checkout@v6

- name: "Build ${{ matrix.value }} Lambda"
- name: "Build ${{ matrix.value.artifact_name }} Lambda"
uses: ./.github/actions/just
env:
LAMBDA_NAME: ${{ matrix.value }}
LAMBDA_SOURCE_DIR: ${{ matrix.value.source_dir }}
LAMBDA_ARTIFACT_NAME: ${{ matrix.value.artifact_name }}
with:
justfile_path: justfile.deploy
just_action: lambda-build

build-containers:
name: Build Containers
name: "${{ matrix.value.image }}"
if: ${{ needs.check.outputs.containers == 'true' }}
needs:
- check
- check-ecs-module-pairs
- setup
- check-ecs-manifest
- ecs-build-matrix
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
value: ${{ fromJson(needs.setup.outputs.container_dirs) }}
value: ${{ fromJson(needs.ecs-build-matrix.outputs.ecs_build_matrix) }}
steps:
- uses: actions/checkout@v6

- name: "Build ${{ matrix.value }} ECS image"
- name: "Build ${{ matrix.value.image }} ECS image"
uses: ./.github/actions/just
env:
CONTAINER_NAME: ${{ matrix.value }}
CONTAINER_NAME: ${{ matrix.value.image }}
with:
justfile_path: justfile.deploy
just_action: docker-build
Expand Down
12 changes: 0 additions & 12 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,21 +100,11 @@ jobs:
echo "$COMMITS" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT

get-apps:
name: Discover
needs:
- get-next-tag
- create-tag
if: ${{ needs.get-next-tag.outputs.create-new-release == 'true' }}
uses: ./.github/workflows/shared_directories_get.yml


build:
name: Build
needs:
- create-tag
- get-next-tag
- get-apps
- code
if: ${{ needs.get-next-tag.outputs.create-new-release == 'true' }}
uses: ./.github/workflows/shared_build.yml
Expand All @@ -126,8 +116,6 @@ jobs:
lambda_version: ${{ needs.get-next-tag.outputs.tag }}
frontend_version: ${{ needs.get-next-tag.outputs.tag }}
ecs_version: ${{ needs.get-next-tag.outputs.tag }}
lambda_matrix: ${{ needs.get-apps.outputs.lambda_dirs }}
ecs_matrix: ${{ needs.get-apps.outputs.container_dirs }}

code:
name: Artifacts
Expand Down
Loading