diff --git a/.github/docs/README.md b/.github/docs/README.md index 171bc33d..3758a0ed 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -11,7 +11,7 @@ This README is the router. The contract details live in focused docs so humans a | Find the workflow a user should trigger | [workflow-entrypoints.md](workflow-entrypoints.md) | | Change reusable build, infra, deploy, or release workflow behavior | [reusable-workflows.md](reusable-workflows.md) | | Change saved plan, apply-from-plan, artifact naming, or plan metadata behavior | [artifacts-and-plans.md](artifacts-and-plans.md) | -| Change directory discovery, service/container matrices, or Terragrunt graph waves | [discovery-and-matrices.md](discovery-and-matrices.md) | +| Change runtime manifests, action discovery, or Terragrunt graph waves | [discovery-and-matrices.md](discovery-and-matrices.md) | | Change `.github/actions/**`, OIDC role ARN construction, release tagging, or action tests | [repo-local-actions.md](repo-local-actions.md) | | Change destroy behavior, post-destroy cleanup, or tagged-resource sweeps | [destroy.md](destroy.md) | | Review any workflow or CI contract change | [feasibility-checks.md](feasibility-checks.md) | diff --git a/.github/docs/discovery-and-matrices.md b/.github/docs/discovery-and-matrices.md index 4abf421b..53da01e6 100644 --- a/.github/docs/discovery-and-matrices.md +++ b/.github/docs/discovery-and-matrices.md @@ -1,23 +1,34 @@ # Discovery And Matrices -Use this when changing directory discovery, runtime matrices, service/container naming, or Terragrunt graph waves. +Use this when changing runtime manifests, repo-local action discovery, or Terragrunt graph waves. ## Directory Discovery -`shared_directories_get.yml` derives directory-based matrices used by wrapper workflows and PR action-test discovery. +`shared_directories_get.yml` derives repo-local action matrices used by PR action-test discovery. -`service_dirs` and `container_dirs` are intentionally different: +Lambda discovery is manifest-based: -- `service_dirs` contains deployable ECS service image directories only. -- `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. -- ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need shared sidecar images as well as service images. -- Workflows that only need app service names or task/service stack derivation should use `service_dirs`. +- `lambdas/deploy.yml` is the source of truth for Lambda build and deploy records. +- `shared_build.yml` derives unique Lambda source records from the manifest when it runs. +- `shared_deploy.yml` derives every Lambda deploy record from the manifest when it runs. +- wrapper workflows do not pass Lambda matrices; changing the Lambda deployment set is a `lambdas/deploy.yml` change. +- `stack` values are repo-relative Terragrunt stack path templates such as `infra/live/{environment}/aws/lambda_api`. +- `source_dir` values are repo-relative source paths; the artifact filename is computed from `basename(source_dir)`. + +ECS discovery is manifest-based: + +- `containers/deploy.yml` is the source of truth for ECS image build and service deploy records. +- `shared_build.yml` derives unique ECS image records from the manifest when it runs. +- `shared_deploy.yml` derives every ECS service deploy record from the manifest when it runs. +- wrapper workflows do not pass ECS or task matrices; changing the ECS deployment set is a `containers/deploy.yml` change. +- `task_stack` and `service_stack` values are repo-relative Terragrunt stack path templates such as `infra/live/{environment}/aws/task_api`. +- `image` is the ECR tag prefix and maps to the default Docker service source directory `containers/`. +- `support_images` lists shared images such as `debug` and `otel_collector` that are built with ECS images because task definitions require them. Top-level runtime discovery rules: -- top-level Lambda directories under `lambdas/` are deployable functions, excluding generated build output -- top-level deployable ECS service directories under `containers/` are exposed through `service_dirs` -- the broader `container_dirs` matrix includes deployable service directories plus shared sidecar image targets +- Lambda deployability is declared in `lambdas/deploy.yml`; top-level Lambda directories are not deploy targets unless the manifest references them +- ECS deployability is declared in `containers/deploy.yml`; top-level container directories are not deploy targets unless the manifest references them ## Module Discovery @@ -50,7 +61,7 @@ Each wave contains only modules whose direct dependencies were satisfied by earl ## Runtime Coverage Checks -- If Lambda directories are auto-detected, confirm matching live Terragrunt stacks still exist. -- If ECS directories are auto-detected, confirm matching `task_*` and `service_*` live Terragrunt stacks still exist. +- If Lambda manifest entries change, confirm each `stack` path exists for every deployed environment and each `source_dir` still builds. +- If ECS manifest entries change, confirm each `task_stack` and `service_stack` path exists for every deployed environment and each `image` source still builds. - For `*_code` wrappers, confirm dispatch inputs cover every runtime being deployed. - If ECS deploys are included, confirm `ecs_version` is exposed or intentionally derived. diff --git a/.github/docs/feasibility-checks.md b/.github/docs/feasibility-checks.md index 8e7583c9..502bf00a 100644 --- a/.github/docs/feasibility-checks.md +++ b/.github/docs/feasibility-checks.md @@ -32,8 +32,8 @@ Run these checks on every CI, workflow, or deploy-contract change. ## Runtime Coverage -- If Lambda directories are auto-detected, confirm matching live Terragrunt stacks still exist. -- If ECS directories are auto-detected, confirm matching `task_*` and `service_*` live Terragrunt stacks still exist. +- If Lambda manifest entries change, confirm each `source_dir` exists, each computed zip basename is unique, and each `stack` path exists for every deployed environment. +- If ECS manifest entries change, confirm each `image` source exists and each `task_stack` / `service_stack` path exists for every deployed environment. - For `*_code` wrappers, confirm dispatch inputs cover every runtime being deployed. - If ECS deploys are included, confirm `ecs_version` is exposed or intentionally derived. diff --git a/.github/docs/reusable-workflows.md b/.github/docs/reusable-workflows.md index 2f522c27..5da0d5f2 100644 --- a/.github/docs/reusable-workflows.md +++ b/.github/docs/reusable-workflows.md @@ -21,9 +21,9 @@ Use this when editing shared workflows under `.github/workflows/shared_*.yml` or - Its `check` job normally runs `.github/actions/get-changes` using the PR base SHA for a PR-style `base...HEAD` diff. - Manual `workflow_dispatch` runs force every change flag on and rerun the full validation surface without a PR diff. - When `.github/actions/**` changed, it reuses `shared_directories_get.yml` to discover action directories with `Dockerfile`s and runs a Docker unit-test matrix after GitHub formatting. -- Lambda naming checks only run when Lambda sources changed. -- ECS task/service pair checks run when container sources or Terragrunt live-stack directories changed. -- Lambda naming and ECS task/service pair checks are explicit prerequisites for the corresponding build jobs. +- Lambda manifest validation only runs when Lambda sources changed, and checks both the deploy manifest and matching live stack paths. +- ECS manifest validation runs when container sources or Terragrunt live-stack directories changed, and checks both the deploy manifest and task/service stack pairs. +- Lambda and ECS manifest validation are explicit prerequisites for the corresponding build jobs. - Terragrunt installation uses `jdx/mise-action@v4`. - TFLint setup uses the Node 24 `terraform-linters/setup-tflint@v6` line. @@ -40,10 +40,15 @@ The local version action can also be tested outside GitHub Actions by running th `shared_build.yml` builds and publishes frontend, Lambda, and ECS artifacts. -`shared_build_get.yml` resolves artifact locations and derives matrices used by downstream deploy wrappers. +- Lambda builds are derived internally from `lambdas/deploy.yml`; callers do not pass a Lambda matrix. +- ECS image builds are derived internally from `containers/deploy.yml`; callers do not pass an ECS/container matrix. + +`shared_build_get.yml` resolves artifact locations used by downstream deploy wrappers. - Its multi-step `images` and `lambdas` jobs configure AWS credentials once. - Repeated `just` calls reuse that ambient session against the same account. +- Prod deploy resolution checks the computed Lambda zip keys from `lambdas/deploy.yml` exist in the shared code bucket. +- Prod deploy resolution checks the computed ECS image tags from `containers/deploy.yml` exist in ECR. ```mermaid flowchart LR @@ -84,8 +89,9 @@ Current infra selection comes from the Terragrunt dependency graph and derived w `shared_deploy.yml` rolls out feature code. - Publishes Lambda versions. -- Optionally invokes the `migrations` Lambda when it is part of the Lambda deploy matrix. -- Optionally runs reconciliation Lambdas. +- Derives Lambda deploy records internally from `lambdas/deploy.yml`; callers do not pass a Lambda matrix. +- Optionally invokes Lambdas whose deploy manifest entry sets `after_deploy: invoke`. +- Derives ECS deploy records internally from `containers/deploy.yml`; callers do not pass an ECS task matrix. - Registers ECS task revisions. - Updates ECS services. - Optionally deploys frontend assets. diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 36d197fc..f1b99a9a 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -28,8 +28,8 @@ permissions: env: TF_VAR_lambda_version: this TF_VAR_image_uri: destroy-placeholder - TF_VAR_aws_otel_collector_image_uri: destroy-placeholder - TF_VAR_debug_image_uri: destroy-placeholder + TF_VAR_otel_collector_uri: destroy-placeholder + TF_VAR_debug_uri: destroy-placeholder AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} diff --git a/.github/workflows/dev_code_deploy.yml b/.github/workflows/dev_code_deploy.yml index 63405573..ef06189e 100644 --- a/.github/workflows/dev_code_deploy.yml +++ b/.github/workflows/dev_code_deploy.yml @@ -9,46 +9,24 @@ permissions: contents: write jobs: - setup: - name: Discover - uses: ./.github/workflows/shared_directories_get.yml - build: name: Build uses: ./.github/workflows/shared_build.yml - needs: - - setup with: environment: dev lambda_version: ${{ github.sha }} frontend_version: ${{ github.sha }} ecs_version: ${{ github.sha }} - lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }} - ecs_matrix: ${{ needs.setup.outputs.container_dirs }} - - - get_build: - name: Resolve - needs: build - uses: ./.github/workflows/shared_build_get.yml - with: - environment: dev - lambda_version: ${{ needs.build.outputs.lambda_version }} - frontend_version: ${{ needs.build.outputs.frontend_version }} - ecs_version: ${{ needs.build.outputs.ecs_version }} deploy: name: Deploy uses: ./.github/workflows/shared_deploy.yml needs: - - setup - build - - get_build with: environment: dev lambda_version: ${{ needs.build.outputs.lambda_version }} frontend_version: ${{ needs.build.outputs.frontend_version }} - code_bucket: ${{ needs.get_build.outputs.code_bucket }} - lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }} - task_matrix: ${{ needs.get_build.outputs.ecs_task_matrix }} - ecs_image_uris: ${{ needs.get_build.outputs.ecs_image_uris }} + ecs_version: ${{ needs.build.outputs.ecs_version }} + code_bucket: ${{ needs.build.outputs.code_bucket }} + repository_url: ${{ needs.build.outputs.repository_url }} diff --git a/.github/workflows/prod_code_deploy.yml b/.github/workflows/prod_code_deploy.yml index f05e7c26..27fdebc7 100644 --- a/.github/workflows/prod_code_deploy.yml +++ b/.github/workflows/prod_code_deploy.yml @@ -37,10 +37,6 @@ jobs: environment: prod lambda_version: ${{ needs.get_build.outputs.lambda_version }} frontend_version: ${{ needs.get_build.outputs.frontend_version }} + ecs_version: ${{ needs.get_build.outputs.ecs_version }} code_bucket: ${{ needs.get_build.outputs.code_bucket }} - lambda_matrix: ${{ needs.get_build.outputs.lambda_version_files }} - task_matrix: ${{ needs.get_build.outputs.ecs_task_matrix }} - ecs_image_uris: ${{ needs.get_build.outputs.ecs_image_uris }} - # we can also scope the deployment here if needed, as below - # lambda_matrix: '["lambda_api"]' - # task_matrix: '["worker"]' + repository_url: ${{ needs.get_build.outputs.repository_url }} diff --git a/.github/workflows/pull_request.yml b/.github/workflows/pull_request.yml index dc779009..1712a283 100644 --- a/.github/workflows/pull_request.yml +++ b/.github/workflows/pull_request.yml @@ -160,9 +160,47 @@ jobs: needs: - check - format-github - if: ${{ needs.check.outputs.lambdas == 'true' || needs.check.outputs.containers == 'true' || needs.check.outputs.actions == 'true' }} + if: ${{ needs.check.outputs.actions == 'true' }} uses: ./.github/workflows/shared_directories_get.yml + lambda-build-matrix: + name: Lambda Build Records + needs: + - check + - check-lambda-manifest + if: ${{ needs.check.outputs.lambdas == 'true' }} + runs-on: ubuntu-latest + outputs: + lambda_build_matrix: ${{ steps.lambda_build_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + + - name: Get Lambda Build Matrix + id: lambda_build_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-get-build-matrix + + ecs-build-matrix: + name: ECS Build Records + needs: + - check + - check-ecs-manifest + if: ${{ needs.check.outputs.containers == 'true' }} + runs-on: ubuntu-latest + outputs: + ecs_build_matrix: ${{ steps.ecs_build_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + + - name: Get ECS Build Matrix + id: ecs_build_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: ecs-get-build-matrix + action-docker-unit-tests: name: GH Action Docker Unit Tests needs: @@ -229,30 +267,39 @@ jobs: justfile_path: justfile.ci just_action: tf-lint-check - check-lambda-naming: + check-lambda-manifest: needs: check runs-on: ubuntu-latest if: ${{ needs.check.outputs.lambdas == 'true' }} - name: Lambda Pairs + name: Lambda Manifest steps: - uses: actions/checkout@v6 - - name: Fail if any lambda directory uses hyphens - run: | - bad_dirs=$(find lambdas -mindepth 1 -maxdepth 1 -type d -name '*-*') - if [ -n "$bad_dirs" ]; then - echo "::error::❌ Lambda directories must use underscores, not hyphens: $bad_dirs" - exit 1 - fi - echo "✅ All lambda directories use underscores." + - name: Validate Lambda deploy manifest + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-get-deploy-matrix - check-ecs-module-pairs: + - name: Validate Lambda stack paths + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-check-deploy-stacks + + check-ecs-manifest: needs: check runs-on: ubuntu-latest if: ${{ needs.check.outputs.containers == 'true' || needs.check.outputs.terragrunt == 'true' }} - name: ECS Pairs + name: ECS Manifest steps: - uses: actions/checkout@v6 + - name: Validate ECS deploy manifest + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: ecs-get-deploy-matrix + - name: Fail if task_/service_ pairs are incomplete shell: bash run: | @@ -296,47 +343,48 @@ jobs: echo "✅ All ECS task_/service_ pairs are present." build-lambdas: - name: Build Lambdas + name: "${{ matrix.value.artifact_name }}" if: ${{ needs.check.outputs.lambdas == 'true' }} needs: - check - - check-lambda-naming - - setup + - check-lambda-manifest + - lambda-build-matrix runs-on: ubuntu-latest strategy: fail-fast: false matrix: - value: ${{ fromJson(needs.setup.outputs.lambda_dirs) }} + value: ${{ fromJson(needs.lambda-build-matrix.outputs.lambda_build_matrix) }} steps: - uses: actions/checkout@v6 - - name: "Build ${{ matrix.value }} Lambda" + - name: "Build ${{ matrix.value.artifact_name }} Lambda" uses: ./.github/actions/just env: - LAMBDA_NAME: ${{ matrix.value }} + LAMBDA_SOURCE_DIR: ${{ matrix.value.source_dir }} + LAMBDA_ARTIFACT_NAME: ${{ matrix.value.artifact_name }} with: justfile_path: justfile.deploy just_action: lambda-build build-containers: - name: Build Containers + name: "${{ matrix.value.image }}" if: ${{ needs.check.outputs.containers == 'true' }} needs: - check - - check-ecs-module-pairs - - setup + - check-ecs-manifest + - ecs-build-matrix runs-on: ubuntu-latest strategy: fail-fast: false matrix: - value: ${{ fromJson(needs.setup.outputs.container_dirs) }} + value: ${{ fromJson(needs.ecs-build-matrix.outputs.ecs_build_matrix) }} steps: - uses: actions/checkout@v6 - - name: "Build ${{ matrix.value }} ECS image" + - name: "Build ${{ matrix.value.image }} ECS image" uses: ./.github/actions/just env: - CONTAINER_NAME: ${{ matrix.value }} + CONTAINER_NAME: ${{ matrix.value.image }} with: justfile_path: justfile.deploy just_action: docker-build diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a6c20770..fdf3868c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -100,21 +100,11 @@ jobs: echo "$COMMITS" >> $GITHUB_OUTPUT echo "EOF" >> $GITHUB_OUTPUT - get-apps: - name: Discover - needs: - - get-next-tag - - create-tag - if: ${{ needs.get-next-tag.outputs.create-new-release == 'true' }} - uses: ./.github/workflows/shared_directories_get.yml - - build: name: Build needs: - create-tag - get-next-tag - - get-apps - code if: ${{ needs.get-next-tag.outputs.create-new-release == 'true' }} uses: ./.github/workflows/shared_build.yml @@ -126,8 +116,6 @@ jobs: lambda_version: ${{ needs.get-next-tag.outputs.tag }} frontend_version: ${{ needs.get-next-tag.outputs.tag }} ecs_version: ${{ needs.get-next-tag.outputs.tag }} - lambda_matrix: ${{ needs.get-apps.outputs.lambda_dirs }} - ecs_matrix: ${{ needs.get-apps.outputs.container_dirs }} code: name: Artifacts diff --git a/.github/workflows/shared_build.yml b/.github/workflows/shared_build.yml index 5b5d70b0..7da4ca41 100644 --- a/.github/workflows/shared_build.yml +++ b/.github/workflows/shared_build.yml @@ -15,14 +15,6 @@ on: ecs_version: required: true type: string - lambda_matrix: - required: false - type: string - default: "[]" - ecs_matrix: - required: false - type: string - default: "[]" outputs: code_bucket: description: "Bucket containing build artifacts" @@ -39,12 +31,6 @@ on: repository_url: description: "ECR repository url" value: ${{ jobs.ecr.outputs.repository_url }} - ecs_image_uris: - description: "List of full ECS image URIs built by this workflow" - value: ${{ jobs.containers.outputs.ecs_image_uris }} - lambda_s3_keys: - description: "List of lambda S3 object keys built by this workflow" - value: ${{ jobs.lambdas.outputs.lambda_s3_keys }} concurrency: # only run one instance of workflow at any one time group: build-${{ inputs.environment }} @@ -60,6 +46,38 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} jobs: + lambda-matrix: + runs-on: ubuntu-latest + outputs: + lambda_build_matrix: ${{ steps.lambda_build_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.lambda_version }} + + - name: Get Lambda Build Matrix + id: lambda_build_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-get-build-matrix + + ecs-matrix: + runs-on: ubuntu-latest + outputs: + ecs_build_matrix: ${{ steps.ecs_build_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.ecs_version }} + + - name: Get ECS Build Matrix + id: ecs_build_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: ecs-get-build-matrix + bucket: runs-on: ubuntu-latest outputs: @@ -133,14 +151,15 @@ jobs: echo "repository_url=$(echo $TG_OUTPUTS | jq -r '.repository_url.value')" >> $GITHUB_OUTPUT containers: - needs: ecr + name: "${{ matrix.value.image }}" + needs: + - ecr + - ecs-matrix runs-on: ubuntu-latest - outputs: - ecs_image_uris: ${{ steps.image_uris.outputs.ecs_image_uris }} strategy: fail-fast: true matrix: - value: ${{ fromJson(inputs.ecs_matrix) }} + value: ${{ fromJson(needs.ecs-matrix.outputs.ecs_build_matrix) }} steps: - uses: actions/checkout@v6 @@ -149,39 +168,25 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: "Build ${{ matrix.value }} ECS image" + - name: "Build ${{ matrix.value.image }} ECS image" uses: ./.github/actions/just env: - CONTAINER_NAME: ${{ matrix.value }} - IMAGE_URI: ${{ needs.ecr.outputs.repository_url }}:${{ matrix.value }}-${{ inputs.ecs_version }} + CONTAINER_NAME: ${{ matrix.value.image }} + IMAGE_URI: ${{ needs.ecr.outputs.repository_url }}:${{ matrix.value.image }}-${{ inputs.ecs_version }} with: justfile_path: justfile.deploy just_action: docker-build docker-push - - name: Build ECS image URI list - if: ${{ matrix.value == fromJson(inputs.ecs_matrix)[0] }} - id: image_uris - shell: bash - env: - REPOSITORY_URL: ${{ needs.ecr.outputs.repository_url }} - ECS_VERSION: ${{ inputs.ecs_version }} - ECS_MATRIX: ${{ inputs.ecs_matrix }} - run: | - echo "ecs_image_uris=$(jq -cn \ - --arg repo "$REPOSITORY_URL" \ - --arg version "$ECS_VERSION" \ - --argjson images "$ECS_MATRIX" \ - '$images | map("\($repo):\(.)-\($version)")')" >> "$GITHUB_OUTPUT" - lambdas: - needs: bucket + name: "${{ matrix.value.artifact_name }}" + needs: + - bucket + - lambda-matrix runs-on: ubuntu-latest - outputs: - lambda_s3_keys: ${{ steps.lambda_s3_keys.outputs.lambda_s3_keys }} strategy: fail-fast: true matrix: - value: ${{ fromJson(inputs.lambda_matrix) }} + value: ${{ fromJson(needs.lambda-matrix.outputs.lambda_build_matrix) }} steps: - uses: actions/checkout@v6 @@ -190,25 +195,13 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: "Upload ${{ matrix.value }} Lambda" + - name: "Upload ${{ matrix.value.artifact_name }} Lambda" uses: ./.github/actions/just env: - LAMBDA_NAME: ${{ matrix.value }} + LAMBDA_SOURCE_DIR: ${{ matrix.value.source_dir }} + LAMBDA_ARTIFACT_NAME: ${{ matrix.value.artifact_name }} BUCKET_NAME: ${{ needs.bucket.outputs.code_bucket_name }} VERSION: ${{ inputs.lambda_version }} with: justfile_path: justfile.deploy just_action: lambda-build lambda-upload - - - name: Build lambda S3 key list - if: ${{ matrix.value == fromJson(inputs.lambda_matrix)[0] }} - id: lambda_s3_keys - shell: bash - env: - LAMBDA_VERSION: ${{ inputs.lambda_version }} - LAMBDA_MATRIX: ${{ inputs.lambda_matrix }} - run: | - echo "lambda_s3_keys=$(jq -cn \ - --arg version "$LAMBDA_VERSION" \ - --argjson lambdas "$LAMBDA_MATRIX" \ - '$lambdas | map("lambdas/\($version)/\(.).zip")')" >> "$GITHUB_OUTPUT" diff --git a/.github/workflows/shared_build_get.yml b/.github/workflows/shared_build_get.yml index 2cc733cd..6c86e7ec 100644 --- a/.github/workflows/shared_build_get.yml +++ b/.github/workflows/shared_build_get.yml @@ -31,21 +31,6 @@ on: repository_url: description: "Resolved ECR repository URL" value: ${{ jobs.ecr.outputs.repository_url }} - ecs_image_uris: - description: "List of full ECS image URIs" - value: ${{ jobs.images.outputs.ecs_image_uris }} - ecs_task_matrix: - description: "List of ECS service names for the version" - value: ${{ jobs.images.outputs.ecs_task_matrix }} - ecs_service_matrix: - description: "List of ECS service stack names for the version" - value: ${{ jobs.images.outputs.ecs_service_matrix }} - lambda_version_files: - description: "List of lambda names" - value: ${{ jobs.lambdas.outputs.lambda_version_files }} - lambda_s3_keys: - description: "List of lambda S3 object keys" - value: ${{ jobs.lambdas.outputs.lambda_s3_keys }} concurrency: # only run one instance of workflow at any one time group: ${{ github.workflow }}-${{ inputs.environment }} @@ -67,7 +52,7 @@ jobs: steps: - uses: actions/checkout@v6 with: - ref: ${{ inputs.lambda_version }} + ref: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} - uses: aws-actions/configure-aws-credentials@v6 with: @@ -141,10 +126,6 @@ jobs: images: needs: ecr runs-on: ubuntu-latest - outputs: - ecs_image_uris: ${{ steps.image_uris.outputs.ecs_image_uris }} - ecs_task_matrix: ${{ steps.task_matrix.outputs.just_outputs }} - ecs_service_matrix: ${{ steps.service_matrix.outputs.ecs_service_matrix }} steps: - uses: actions/checkout@v6 with: @@ -155,48 +136,14 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Get ECR version images - id: get_version_images + - name: Check ECS images exist uses: ./.github/actions/just env: REPOSITORY_URL: ${{ needs.ecr.outputs.repository_url }} VERSION: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} with: justfile_path: justfile.ci - just_action: get-ecr-version-images - - - name: Build full image URIs - id: image_uris - shell: bash - env: - REPOSITORY_URL: ${{ needs.ecr.outputs.repository_url }} - VERSION: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} - IMAGE_NAMES: ${{ steps.get_version_images.outputs.just_outputs }} - run: | - echo "ecs_image_uris=$(jq -cn \ - --arg repo "$REPOSITORY_URL" \ - --arg version "$VERSION" \ - --argjson images "$IMAGE_NAMES" \ - '$images | map("\($repo):\(.)-\($version)")')" >> "$GITHUB_OUTPUT" - - - name: Build ECS task matrix - id: task_matrix - uses: ./.github/actions/just - env: - REPOSITORY_URL: ${{ needs.ecr.outputs.repository_url }} - VERSION: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} - with: - justfile_path: justfile.ci - just_action: get-ecr-version-tasks - - - name: Build ECS service matrix - id: service_matrix - env: - ECS_TASK_MATRIX: ${{ steps.task_matrix.outputs.just_outputs }} - run: | - echo "ecs_service_matrix=$(jq -cn \ - --argjson tasks "$ECS_TASK_MATRIX" \ - '$tasks | map("service_" + .)')" >> "$GITHUB_OUTPUT" + just_action: ecs-check-deploy-images frontend: needs: bucket @@ -223,9 +170,6 @@ jobs: lambdas: needs: bucket runs-on: ubuntu-latest - outputs: - lambda_version_files: ${{ steps.get_build_files.outputs.just_outputs }} - lambda_s3_keys: ${{ steps.get_build_file_keys.outputs.just_outputs }} steps: - uses: actions/checkout@v6 @@ -246,22 +190,11 @@ jobs: justfile_path: justfile.ci just_action: lambda-check-version - - name: Get lambda names - id: get_build_files - uses: ./.github/actions/just - env: - BUCKET_NAME: ${{ needs.bucket.outputs.code_bucket_name }} - VERSION: ${{ inputs.lambda_version }} - with: - justfile_path: justfile.ci - just_action: get-version-files - - - name: Get lambda S3 keys - id: get_build_file_keys + - name: Check Lambda deploy artifacts exist uses: ./.github/actions/just env: BUCKET_NAME: ${{ needs.bucket.outputs.code_bucket_name }} VERSION: ${{ inputs.lambda_version }} with: justfile_path: justfile.ci - just_action: get-version-file-keys + just_action: lambda-check-deploy-artifacts diff --git a/.github/workflows/shared_deploy.yml b/.github/workflows/shared_deploy.yml index b7f6f8a9..f36fdb3d 100644 --- a/.github/workflows/shared_deploy.yml +++ b/.github/workflows/shared_deploy.yml @@ -16,23 +16,20 @@ on: required: false type: string default: "" + ecs_version: + description: "Valid ECS version" + required: false + type: string + default: "" code_bucket: description: "Bucket containing lambda and frontend zips" required: true type: string - ecs_image_uris: - description: "List of full ECS image URIs" - required: false - type: string - default: "[]" - lambda_matrix: + repository_url: + description: "ECR repository URL hosting the ECS images" required: false type: string - default: "[]" - task_matrix: - required: false - type: string - default: "[]" + default: "" lambda_keep: description: "Number of lambda versions to keep" default: '5' @@ -52,12 +49,54 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} jobs: + lambda-matrix: + runs-on: ubuntu-latest + outputs: + lambda_deploy_matrix: ${{ steps.lambda_deploy_matrix.outputs.just_outputs }} + lambda_invoke_matrix: ${{ steps.lambda_invoke_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.lambda_version }} + + - name: Get Lambda Deploy Matrix + id: lambda_deploy_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-get-deploy-matrix + + - name: Get Lambda Invoke Matrix + id: lambda_invoke_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: lambda-get-invoke-matrix + + ecs-matrix: + runs-on: ubuntu-latest + outputs: + ecs_deploy_matrix: ${{ steps.ecs_deploy_matrix.outputs.just_outputs }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} + + - name: Get ECS Deploy Matrix + id: ecs_deploy_matrix + uses: ./.github/actions/just + with: + justfile_path: justfile.ci + just_action: ecs-get-deploy-matrix + lambdas: + name: "${{ matrix.value.artifact_name }}" + needs: lambda-matrix runs-on: ubuntu-latest strategy: fail-fast: true matrix: - value: ${{ fromJson(inputs.lambda_matrix) }} + value: ${{ fromJson(needs.lambda-matrix.outputs.lambda_deploy_matrix) }} steps: - uses: actions/checkout@v6 @@ -66,20 +105,27 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Set AppSpec paths - id: appspec + - name: Set Lambda deploy paths + id: lambda_paths shell: bash + env: + ENVIRONMENT: ${{ inputs.environment }} + STACK_TEMPLATE: ${{ matrix.value.stack }} + ARTIFACT_FILE: ${{ matrix.value.artifact_file }} run: | - lambda_zip_key="lambdas/${{ inputs.lambda_version }}/${{ matrix.value }}.zip" - lambda_appspec_key="lambdas/${{ inputs.lambda_version }}/${{ matrix.value }}-appspec.zip" + stack_path="${STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" + stack_name="${stack_path##*/}" + lambda_zip_key="lambdas/${{ inputs.lambda_version }}/$ARTIFACT_FILE" + lambda_appspec_key="lambdas/${{ inputs.lambda_version }}/$stack_name-appspec.zip" + echo "stack_path=$stack_path" >> $GITHUB_OUTPUT echo "lambda_zip_key=$lambda_zip_key" >> $GITHUB_OUTPUT echo "lambda_appspec_key=$lambda_appspec_key" >> $GITHUB_OUTPUT - - name: Get ${{ matrix.value }} infra + - name: Get ${{ matrix.value.stack }} infra uses: ./.github/actions/terragrunt id: get-infra with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} + tg_directory: ${{ steps.lambda_paths.outputs.stack_path }} tg_action: init - name: Get infra detail @@ -96,7 +142,7 @@ jobs: env: BUCKET_NAME: ${{ inputs.code_bucket }} FUNCTION_NAME: ${{ steps.get_infra_detail.outputs.lambda_function_name }} - LAMBDA_ZIP_KEY: ${{ steps.appspec.outputs.lambda_zip_key }} + LAMBDA_ZIP_KEY: ${{ steps.lambda_paths.outputs.lambda_zip_key }} with: justfile_path: justfile.deploy just_action: lambda-create-version @@ -121,86 +167,53 @@ jobs: CURRENT_VERSION: ${{ steps.get-version.outputs.just_outputs }} NEW_VERSION: ${{ steps.publish.outputs.just_outputs }} APP_SPEC_FILE: ${{ github.workspace }}/config/deploy/appspec-lambda.rendered.yml - APP_SPEC_KEY: ${{ steps.appspec.outputs.lambda_appspec_key }} + APP_SPEC_KEY: ${{ steps.lambda_paths.outputs.lambda_appspec_key }} with: justfile_path: justfile.deploy just_action: lambda-upload-bundle lambda-set-code-deploy-alarms lambda-deploy lambda-prune - run-migrations: + invoke-lambdas: + name: "${{ matrix.value.artifact_name }}" runs-on: ubuntu-latest - needs: lambdas + needs: + - lambdas + - lambda-matrix + strategy: + fail-fast: false + matrix: + value: ${{ fromJson(needs.lambda-matrix.outputs.lambda_invoke_matrix) }} steps: - uses: actions/checkout@v6 - uses: aws-actions/configure-aws-credentials@v6 - if: ${{ contains(fromJson(inputs.lambda_matrix), 'migrations') }} with: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Skip when migrations is not in the lambda matrix - if: ${{ !contains(fromJson(inputs.lambda_matrix), 'migrations') }} - run: echo "No migrations Lambda in this deploy matrix." - - - name: Get migrations infra - if: ${{ contains(fromJson(inputs.lambda_matrix), 'migrations') }} - uses: ./.github/actions/terragrunt - id: get-infra - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/migrations - tg_action: init - - - name: Get migrations function name - if: ${{ contains(fromJson(inputs.lambda_matrix), 'migrations') }} - id: get_infra_detail + - name: Set Lambda stack path + id: lambda_paths + shell: bash env: - TG_OUTPUTS: ${{ steps.get-infra.outputs.tg_outputs }} + ENVIRONMENT: ${{ inputs.environment }} + STACK_TEMPLATE: ${{ matrix.value.stack }} run: | - echo "lambda_function_name=$(echo "$TG_OUTPUTS" | jq -r '.lambda_function_name.value')" >> "$GITHUB_OUTPUT" - - - name: Run database migrations - if: ${{ contains(fromJson(inputs.lambda_matrix), 'migrations') }} - uses: ./.github/actions/just - env: - LAMBDA_NAME: ${{ steps.get_infra_detail.outputs.lambda_function_name }} - with: - justfile_path: justfile.deploy - just_action: lambda-invoke - - run-rds-reader-tagger: - runs-on: ubuntu-latest - needs: lambdas - steps: - - uses: actions/checkout@v6 + echo "stack_path=${STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" >> "$GITHUB_OUTPUT" - - uses: aws-actions/configure-aws-credentials@v6 - if: ${{ contains(fromJson(inputs.lambda_matrix), 'rds_reader_tagger') }} - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Skip when rds_reader_tagger is not in the lambda matrix - if: ${{ !contains(fromJson(inputs.lambda_matrix), 'rds_reader_tagger') }} - run: echo "No rds_reader_tagger Lambda in this deploy matrix." - - - name: Get rds_reader_tagger infra - if: ${{ contains(fromJson(inputs.lambda_matrix), 'rds_reader_tagger') }} + - name: Get Lambda infra uses: ./.github/actions/terragrunt id: get-infra with: - tg_directory: infra/live/${{ inputs.environment }}/aws/rds_reader_tagger + tg_directory: ${{ steps.lambda_paths.outputs.stack_path }} tg_action: init - - name: Get rds_reader_tagger function name - if: ${{ contains(fromJson(inputs.lambda_matrix), 'rds_reader_tagger') }} + - name: Get Lambda function name id: get_infra_detail env: TG_OUTPUTS: ${{ steps.get-infra.outputs.tg_outputs }} run: | echo "lambda_function_name=$(echo "$TG_OUTPUTS" | jq -r '.lambda_function_name.value')" >> "$GITHUB_OUTPUT" - - name: Run reader tag reconciliation - if: ${{ contains(fromJson(inputs.lambda_matrix), 'rds_reader_tagger') }} + - name: Invoke Lambda uses: ./.github/actions/just env: LAMBDA_NAME: ${{ steps.get_infra_detail.outputs.lambda_function_name }} @@ -249,11 +262,13 @@ jobs: just_action: frontend-deploy frontend-invalidate tasks: + name: "${{ matrix.value.image }}" runs-on: ubuntu-latest + needs: ecs-matrix strategy: fail-fast: false matrix: - value: ${{ fromJson(inputs.task_matrix) }} + value: ${{ fromJson(needs.ecs-matrix.outputs.ecs_deploy_matrix) }} steps: - uses: actions/checkout@v6 @@ -263,40 +278,53 @@ jobs: aws-region: ${{ env.AWS_REGION }} - name: Resolve image URIs - id: image_uris + id: task_inputs + shell: bash env: - ECS_IMAGE_URIS: ${{ inputs.ecs_image_uris }} - TASK_NAME: ${{ matrix.value }} - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: ecs-task-get-image-uris + REPOSITORY_URL: ${{ inputs.repository_url }} + ECS_VERSION: ${{ inputs.ecs_version != '' && inputs.ecs_version || inputs.lambda_version }} + ECS_IMAGE_NAME: ${{ matrix.value.image }} + run: | + if [ -z "$REPOSITORY_URL" ]; then + echo "::error::repository_url input is required to resolve ECS image URIs" + exit 1 + fi + echo "service_image_uri=${REPOSITORY_URL}:${ECS_IMAGE_NAME}-${ECS_VERSION}" >> "$GITHUB_OUTPUT" + echo "debug_uri=${REPOSITORY_URL}:debug-${ECS_VERSION}" >> "$GITHUB_OUTPUT" + echo "otel_collector_uri=${REPOSITORY_URL}:otel_collector-${ECS_VERSION}" >> "$GITHUB_OUTPUT" - - name: Set image outputs - id: task_inputs + - name: Set ECS stack paths + id: ecs_paths + shell: bash env: - IMAGE_URIS_JSON: ${{ steps.image_uris.outputs.just_outputs }} + ENVIRONMENT: ${{ inputs.environment }} + TASK_STACK_TEMPLATE: ${{ matrix.value.task_stack }} + SERVICE_STACK_TEMPLATE: ${{ matrix.value.service_stack }} run: | - echo "service_image_uri=$(echo "$IMAGE_URIS_JSON" | jq -r '.service_image_uri')" >> "$GITHUB_OUTPUT" - echo "debug_image_uri=$(echo "$IMAGE_URIS_JSON" | jq -r '.debug_image_uri')" >> "$GITHUB_OUTPUT" - echo "otel_image_uri=$(echo "$IMAGE_URIS_JSON" | jq -r '.otel_image_uri')" >> "$GITHUB_OUTPUT" + task_stack_path="${TASK_STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" + service_stack_path="${SERVICE_STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" + echo "task_stack_path=$task_stack_path" >> "$GITHUB_OUTPUT" + echo "service_stack_path=$service_stack_path" >> "$GITHUB_OUTPUT" - - name: Deploy ${{ matrix.value }} ECS task + - name: Deploy ${{ matrix.value.task_stack }} ECS task uses: ./.github/actions/terragrunt env: TF_VAR_image_uri: ${{ steps.task_inputs.outputs.service_image_uri }} - TF_VAR_debug_image_uri: ${{ steps.task_inputs.outputs.debug_image_uri }} - TF_VAR_aws_otel_collector_image_uri: ${{ steps.task_inputs.outputs.otel_image_uri }} + TF_VAR_debug_uri: ${{ steps.task_inputs.outputs.debug_uri }} + TF_VAR_otel_collector_uri: ${{ steps.task_inputs.outputs.otel_collector_uri }} with: - tg_directory: infra/live/${{ inputs.environment }}/aws/task_${{ matrix.value }} + tg_directory: ${{ steps.ecs_paths.outputs.task_stack_path }} ecs: + name: "${{ matrix.value.image }}" runs-on: ubuntu-latest - needs: tasks + needs: + - tasks + - ecs-matrix strategy: fail-fast: false matrix: - value: ${{ fromJson(inputs.task_matrix) }} + value: ${{ fromJson(needs.ecs-matrix.outputs.ecs_deploy_matrix) }} steps: - uses: actions/checkout@v6 @@ -305,14 +333,29 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Get ${{ matrix.value }} task infra + - name: Set ECS stack paths + id: ecs_paths + shell: bash + env: + ENVIRONMENT: ${{ inputs.environment }} + TASK_STACK_TEMPLATE: ${{ matrix.value.task_stack }} + SERVICE_STACK_TEMPLATE: ${{ matrix.value.service_stack }} + run: | + task_stack_path="${TASK_STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" + service_stack_path="${SERVICE_STACK_TEMPLATE//\{environment\}/$ENVIRONMENT}" + service_stack_name="${service_stack_path##*/}" + echo "task_stack_path=$task_stack_path" >> "$GITHUB_OUTPUT" + echo "service_stack_path=$service_stack_path" >> "$GITHUB_OUTPUT" + echo "service_stack_name=$service_stack_name" >> "$GITHUB_OUTPUT" + + - name: Get ${{ matrix.value.task_stack }} task infra uses: ./.github/actions/terragrunt id: get-task-infra with: - tg_directory: infra/live/${{ inputs.environment }}/aws/task_${{ matrix.value }} + tg_directory: ${{ steps.ecs_paths.outputs.task_stack_path }} tg_action: init - - name: Get ${{ matrix.value }} task outputs + - name: Get ${{ matrix.value.task_stack }} task outputs id: get-task-outputs env: TG_OUTPUTS: ${{ steps.get-task-infra.outputs.tg_outputs }} @@ -320,14 +363,14 @@ jobs: echo "task_definition_arn=$(echo "$TG_OUTPUTS" | jq -r '.task_definition_arn.value')" >> "$GITHUB_OUTPUT" echo "container_name=$(echo "$TG_OUTPUTS" | jq -r '.service_name.value')" >> "$GITHUB_OUTPUT" - - name: Get ${{ matrix.value }} service infra + - name: Get ${{ matrix.value.service_stack }} service infra uses: ./.github/actions/terragrunt id: get-service-infra with: - tg_directory: infra/live/${{ inputs.environment }}/aws/service_${{ matrix.value }} + tg_directory: ${{ steps.ecs_paths.outputs.service_stack_path }} tg_action: init - - name: Get ${{ matrix.value }} service outputs + - name: Get ${{ matrix.value.service_stack }} service outputs id: get-service-outputs env: SERVICE_OUTPUTS: ${{ steps.get-service-infra.outputs.tg_outputs }} @@ -337,7 +380,7 @@ jobs: echo "container_port=$(echo "$SERVICE_OUTPUTS" | jq -r '.container_port.value')" >> "$GITHUB_OUTPUT" echo "codedeploy_app_name=$(echo "$SERVICE_OUTPUTS" | jq -r '.codedeploy_app_name.value')" >> "$GITHUB_OUTPUT" echo "codedeploy_group_name=$(echo "$SERVICE_OUTPUTS" | jq -r '.codedeploy_deployment_group_name.value')" >> "$GITHUB_OUTPUT" - echo "app_spec_key=ecs/${{ inputs.environment }}/${{ matrix.value }}-$(echo "${{ steps.get-task-outputs.outputs.task_definition_arn }}" | awk -F: '{print $NF}').yml" >> "$GITHUB_OUTPUT" + echo "app_spec_key=ecs/${{ inputs.environment }}/${{ steps.ecs_paths.outputs.service_stack_name }}-$(echo "${{ steps.get-task-outputs.outputs.task_definition_arn }}" | awk -F: '{print $NF}').yml" >> "$GITHUB_OUTPUT" - name: Work out ECS deployment mode id: deploy_mode diff --git a/.github/workflows/shared_directories_get.yml b/.github/workflows/shared_directories_get.yml index 33860b38..8b12851f 100644 --- a/.github/workflows/shared_directories_get.yml +++ b/.github/workflows/shared_directories_get.yml @@ -3,21 +3,6 @@ name: Shared Directories Get on: workflow_call: outputs: - lambda_dirs: - description: "List of lambda directory names" - value: ${{ jobs.directories.outputs.lambda_dirs }} - container_dirs: - description: "List of container directory names" - value: ${{ jobs.directories.outputs.container_dirs }} - service_dirs: - description: "List of ECS service directory names" - value: ${{ jobs.directories.outputs.service_dirs }} - task_dirs: - description: "List of ECS task directory names" - value: ${{ jobs.directories.outputs.task_dirs }} - ecs_service_dirs: - description: "List of ECS service stack directory names" - value: ${{ jobs.directories.outputs.ecs_service_dirs }} action_dirs: description: "List of repo-local GitHub action directory names with Dockerfiles" value: ${{ jobs.directories.outputs.action_dirs }} @@ -25,50 +10,10 @@ jobs: directories: runs-on: ubuntu-latest outputs: - lambda_dirs: ${{ steps.lambda_dirs.outputs.just_outputs }} - container_dirs: ${{ steps.container_dirs.outputs.just_outputs }} - service_dirs: ${{ steps.service_dirs.outputs.just_outputs }} - task_dirs: ${{ steps.task_dirs.outputs.just_outputs }} - ecs_service_dirs: ${{ steps.ecs_service_dirs.outputs.just_outputs }} action_dirs: ${{ steps.action_dirs.outputs.just_outputs }} steps: - uses: actions/checkout@v6 - - name: Get Lambda Directories - id: lambda_dirs - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: lambda-get-directories - - - name: Get Service Directories - id: service_dirs - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: service-get-directories - - - name: Get Container Directories - id: container_dirs - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: container-get-directories - - - name: Get Task Directories - id: task_dirs - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: task-get-directories - - - name: Get ECS Service Directories - id: ecs_service_dirs - uses: ./.github/actions/just - with: - justfile_path: justfile.ci - just_action: ecs-service-get-directories - - name: Get Action Directories id: action_dirs uses: ./.github/actions/just diff --git a/containers/README.md b/containers/README.md index 479f957a..2a71232d 100644 --- a/containers/README.md +++ b/containers/README.md @@ -4,11 +4,10 @@ Container source directories for this boilerplate. ## Structure +- `deploy.yml` is the ECS image build/service deploy manifest - each deployable service lives in its own top-level directory such as `api/` or `worker/` - `lib/` contains ECS-only helper code used by deployable services and is intentionally not treated as a deployable image target -- a deployable ECS runtime also needs the corresponding live Terragrunt stacks -- task stacks use `infra/live//aws/task_/terragrunt.hcl` -- service stacks use `infra/live//aws/service_/terragrunt.hcl` when applicable +- a deployable ECS runtime also needs the live Terragrunt task and service stacks declared by its manifest entry ## Common Shape @@ -19,11 +18,15 @@ Container source directories for this boilerplate. ## Build Behavior -- ECS directory discovery auto-detects deployable top-level directories under `containers/` -- ECS image discovery only includes deployable service directories +- ECS discovery reads `containers/deploy.yml` +- `task_stack` and `service_stack` are repo-relative Terragrunt stack path templates and must use `{environment}` for the environment segment, for example `infra/live/{environment}/aws/task_api` +- `image` is the ECR image tag prefix and maps to the default source directory `containers/` +- build workflows deduplicate by `image`; deploy workflows keep every manifest entry so the same image can roll out to multiple ECS services +- wrapper workflows do not pass ECS or task matrices; update this manifest to add, remove, or remap deployed ECS services +- `support_images` lists shared images such as `debug` and `otel_collector` that are built alongside service images because task definitions require them - container images copy only the files referenced by the Dockerfile for the selected service shape, including shared helpers from `lib/` and `containers/lib/` - markdown files in `containers/` are documentation only and are not included in container image artifacts -- detection alone is not enough: the runtime still needs matching Terragrunt task and service stacks +- manifest detection alone is not enough: the runtime still needs the declared Terragrunt task and service stacks - local Docker services should be added explicitly to `docker-compose.local.yml` and `Dockerfile.local` - the local Dockerfile can mirror the production parameterized pattern by passing a `SERVICE` build arg for each target - Compose owns service-specific local commands and env overrides diff --git a/containers/deploy.yml b/containers/deploy.yml new file mode 100644 index 00000000..c167825b --- /dev/null +++ b/containers/deploy.yml @@ -0,0 +1,14 @@ +version: 1 + +support_images: + - debug + - otel_collector + +services: + - task_stack: infra/live/{environment}/aws/task_api + service_stack: infra/live/{environment}/aws/service_api + image: api + + - task_stack: infra/live/{environment}/aws/task_worker + service_stack: infra/live/{environment}/aws/service_worker + image: worker diff --git a/infra/README.md b/infra/README.md index 484dba2a..06d054af 100644 --- a/infra/README.md +++ b/infra/README.md @@ -151,7 +151,7 @@ Current stack examples include: ECS API service shape exposed on the shared API Gateway at `/ecs` using `vpc_link` and `blue_green`, backed by a dedicated listener on the shared ALB. Through the frontend distribution it is reached at `/api/ecs/*`, while the Lambda API is reached at `/api/*`. The ECS task wrappers share common app-level tracing code from `containers/lib`, so enabling `xray_enabled` produces app spans as well as sidecar export wiring. -That `containers/lib` directory is helper code only and is not treated as a deployable ECS image target by the CI directory-discovery recipes. +That `containers/lib` directory is helper code only and is not treated as a deployable ECS image target by the manifest-driven build helpers. ## Dependency Notes @@ -269,3 +269,6 @@ In CI workflows, be careful whether a matrix is carrying: - or concrete stack names like `task_worker` / `service_worker` That distinction has caused several workflow bugs already. + +Lambda deploy records are derived internally from `lambdas/deploy.yml`. Wrapper workflows should not pass Lambda matrices; update the manifest instead. +ECS deploy records are derived internally from `containers/deploy.yml`. Wrapper workflows should not pass ECS or task matrices; update the manifest instead. diff --git a/infra/modules/aws/_shared/task/README.md b/infra/modules/aws/_shared/task/README.md index e50ee564..a32b1ed5 100644 --- a/infra/modules/aws/_shared/task/README.md +++ b/infra/modules/aws/_shared/task/README.md @@ -14,8 +14,8 @@ Shared ECS task-definition module. - `image_uri` - `ecr_repository_name` -- `debug_image_uri` -- `aws_otel_collector_image_uri` +- `debug_uri` +- `otel_collector_uri` - `local_tunnel` - `xray_enabled` - `command` diff --git a/infra/modules/aws/_shared/task/locals.tf b/infra/modules/aws/_shared/task/locals.tf index b22e71e1..47e21a2c 100644 --- a/infra/modules/aws/_shared/task/locals.tf +++ b/infra/modules/aws/_shared/task/locals.tf @@ -1,11 +1,11 @@ locals { - cloudwatch_log_name = "/ecs/${var.service_name}" - cloudwatch_otel_log_name = "/ecs/${var.service_name}/otel" - image_uri = var.image_uri - aws_otel_collector_image_uri = var.aws_otel_collector_image_uri - debug_image_uri = var.debug_image_uri - ecr_repository_arn = "arn:aws:ecr:${var.aws_region}:${data.aws_caller_identity.current.account_id}:repository/${var.ecr_repository_name}" - root_path_prefix = var.root_path != "" ? "/${var.root_path}" : "" + cloudwatch_log_name = "/ecs/${var.service_name}" + cloudwatch_otel_log_name = "/ecs/${var.service_name}/otel" + image_uri = var.image_uri + otel_collector_uri = var.otel_collector_uri + debug_uri = var.debug_uri + ecr_repository_arn = "arn:aws:ecr:${var.aws_region}:${data.aws_caller_identity.current.account_id}:repository/${var.ecr_repository_name}" + root_path_prefix = var.root_path != "" ? "/${var.root_path}" : "" shared_environment = [ { @@ -105,7 +105,7 @@ locals { otel-collector = { name = "${var.service_name}-otel-collector" - image = local.aws_otel_collector_image_uri + image = local.otel_collector_uri portMappings = [ { @@ -134,7 +134,7 @@ locals { debug-container = { name = "${var.service_name}-debug" - image = local.debug_image_uri + image = local.debug_uri command = ["sleep", "infinity"] diff --git a/infra/modules/aws/_shared/task/variables.tf b/infra/modules/aws/_shared/task/variables.tf index 8149b287..86237939 100644 --- a/infra/modules/aws/_shared/task/variables.tf +++ b/infra/modules/aws/_shared/task/variables.tf @@ -34,7 +34,7 @@ variable "image_uri" { type = string } -variable "aws_otel_collector_image_uri" { +variable "otel_collector_uri" { type = string } @@ -44,7 +44,7 @@ variable "otel_sampling_percentage" { default = 10.0 } -variable "debug_image_uri" { +variable "debug_uri" { type = string } diff --git a/infra/modules/aws/task_api/main.tf b/infra/modules/aws/task_api/main.tf index fd03b27c..df581474 100644 --- a/infra/modules/aws/task_api/main.tf +++ b/infra/modules/aws/task_api/main.tf @@ -8,10 +8,10 @@ module "task_api" { cpu = var.cpu memory = var.memory - image_uri = var.image_uri - debug_image_uri = var.debug_image_uri - aws_otel_collector_image_uri = var.aws_otel_collector_image_uri - otel_sampling_percentage = var.otel_sampling_percentage + image_uri = var.image_uri + debug_uri = var.debug_uri + otel_collector_uri = var.otel_collector_uri + otel_sampling_percentage = var.otel_sampling_percentage local_tunnel = var.local_tunnel xray_enabled = var.xray_enabled diff --git a/infra/modules/aws/task_api/variables.tf b/infra/modules/aws/task_api/variables.tf index c4bd09a5..2aa07a73 100644 --- a/infra/modules/aws/task_api/variables.tf +++ b/infra/modules/aws/task_api/variables.tf @@ -39,7 +39,7 @@ variable "image_uri" { type = string } -variable "aws_otel_collector_image_uri" { +variable "otel_collector_uri" { type = string } @@ -49,7 +49,7 @@ variable "otel_sampling_percentage" { default = 10.0 } -variable "debug_image_uri" { +variable "debug_uri" { type = string } diff --git a/infra/modules/aws/task_worker/main.tf b/infra/modules/aws/task_worker/main.tf index 04f3073b..563fdce6 100644 --- a/infra/modules/aws/task_worker/main.tf +++ b/infra/modules/aws/task_worker/main.tf @@ -13,10 +13,10 @@ module "task_worker" { cpu = var.cpu memory = var.memory - image_uri = var.image_uri - debug_image_uri = var.debug_image_uri - aws_otel_collector_image_uri = var.aws_otel_collector_image_uri - otel_sampling_percentage = var.otel_sampling_percentage + image_uri = var.image_uri + debug_uri = var.debug_uri + otel_collector_uri = var.otel_collector_uri + otel_sampling_percentage = var.otel_sampling_percentage local_tunnel = var.local_tunnel xray_enabled = var.xray_enabled diff --git a/infra/modules/aws/task_worker/variables.tf b/infra/modules/aws/task_worker/variables.tf index a629b8b1..88ef4cb4 100644 --- a/infra/modules/aws/task_worker/variables.tf +++ b/infra/modules/aws/task_worker/variables.tf @@ -39,7 +39,7 @@ variable "image_uri" { type = string } -variable "aws_otel_collector_image_uri" { +variable "otel_collector_uri" { type = string } @@ -49,7 +49,7 @@ variable "otel_sampling_percentage" { default = 10.0 } -variable "debug_image_uri" { +variable "debug_uri" { type = string } diff --git a/justfile b/justfile index cb317ca4..4c532598 100644 --- a/justfile +++ b/justfile @@ -73,7 +73,6 @@ FRONTEND_DIR := "frontend" CONTAINERS_DIR := "containers" APPSPEC_DIR := "appspec" INFRA_PLAN_DIR := "terragrunt_plan" -EXTRA_CONTAINER_DIRECTORIES := "[\"debug\",\"otel_collector\"]" NON_SERVICE_CONTAINER_DIRECTORIES := "[\"lib\",\"_shared\"]" @@ -154,8 +153,8 @@ tg-all env op: cd {{justfile_directory()}}/infra/live/{{env}} export TF_VAR_lambda_version="this" export TF_VAR_image_uri="plan-placeholder" - export TF_VAR_aws_otel_collector_image_uri="plan-placeholder" - export TF_VAR_debug_image_uri="plan-placeholder" + export TF_VAR_otel_collector_uri="plan-placeholder" + export TF_VAR_debug_uri="plan-placeholder" terragrunt run-all {{op}} diff --git a/justfile.ci b/justfile.ci index 573afe7b..8de99ad3 100644 --- a/justfile.ci +++ b/justfile.ci @@ -8,8 +8,6 @@ FRONTEND_DIR := `just --justfile justfile --evaluate FRONTEND_DIR` APPSPEC_DIR := `just --justfile justfile --evaluate APPSPEC_DIR` CONTAINERS_DIR := `just --justfile justfile --evaluate CONTAINERS_DIR` INFRA_PLAN_DIR := `just --justfile justfile --evaluate INFRA_PLAN_DIR` -EXTRA_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate EXTRA_CONTAINER_DIRECTORIES` -NON_SERVICE_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate NON_SERVICE_CONTAINER_DIRECTORIES` # Convert raw Terragrunt graph output from TG_GRAPH_OUTPUT into compact JSON. @@ -222,37 +220,165 @@ lambda-check-version: fi -# List Lambda artifact basenames for a published version. -get-version-files: +# Render Lambda deploy manifest records as compact JSON. +lambda-get-deploy-matrix: #!/usr/bin/env bash set -euo pipefail - if [[ -z "$BUCKET_NAME" ]]; then - echo "❌ BUCKET_NAME environment variable is not set." - exit 1 - fi + cd "{{PROJECT_DIR}}" - if [[ -z "$VERSION" ]]; then - echo "❌ VERSION environment variable is not set." - exit 1 - fi + ruby -ryaml -rjson -rpathname -e ' + manifest_path = ARGV.fetch(0) + project_dir = Pathname(ARGV.fetch(1)).realpath + manifest = YAML.safe_load(File.read(manifest_path), aliases: false) + + unless manifest.is_a?(Hash) && manifest["version"] == 1 + abort("Error: #{manifest_path} must declare version: 1") + end + + lambdas = manifest["lambdas"] || [] + unless lambdas.is_a?(Array) + abort("Error: #{manifest_path} lambdas must be a list when present") + end + + artifacts = {} + stacks = {} + records = lambdas.each_with_index.map do |record, index| + unless record.is_a?(Hash) + abort("Error: lambdas[#{index}] must be a mapping") + end + + stack = record["stack"] + source_dir = record["source_dir"] + after_deploy = record["after_deploy"] + + unless stack.is_a?(String) && stack.start_with?("infra/live/{environment}/aws/") + abort("Error: lambdas[#{index}].stack must start with infra/live/{environment}/aws/") + end + + if stacks.key?(stack) + abort("Error: lambdas[#{index}].stack duplicates lambdas[#{stacks[stack]}].stack: #{stack}") + end + stacks[stack] = index + + unless source_dir.is_a?(String) && !source_dir.empty? && !Pathname(source_dir).absolute? && !source_dir.split("/").include?("..") + abort("Error: lambdas[#{index}].source_dir must be a repo-relative path") + end + + source_path = project_dir.join(source_dir) + unless source_path.directory? + abort("Error: lambdas[#{index}].source_dir does not exist: #{source_dir}") + end + + if after_deploy && after_deploy != "invoke" + abort("Error: lambdas[#{index}].after_deploy only supports invoke") + end + + artifact_name = File.basename(source_dir) + artifact_file = "#{artifact_name}.zip" + prior_source_dir = artifacts[artifact_file] + if prior_source_dir && prior_source_dir != source_dir + abort("Error: Lambda artifact collision: #{prior_source_dir} and #{source_dir} both produce #{artifact_file}") + end + artifacts[artifact_file] = source_dir + + output = { + "stack" => stack, + "source_dir" => source_dir, + "artifact_name" => artifact_name, + "artifact_file" => artifact_file + } + output["after_deploy"] = after_deploy if after_deploy + output + end + + puts JSON.generate(records) + ' "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/deploy.yml" "{{PROJECT_DIR}}" + + +# Filter the Lambda deploy manifest to entries that invoke after deploy. +lambda-get-invoke-matrix: + #!/usr/bin/env bash + set -euo pipefail - FULL_BUCKET_PATH="s3://$BUCKET_NAME/lambdas/$VERSION/" + deploy_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" lambda-get-deploy-matrix)" - aws s3api head-bucket --bucket "$BUCKET_NAME" >/dev/null - aws s3 ls "$FULL_BUCKET_PATH" >/dev/null + jq -cn --argjson lambdas "$deploy_matrix" ' + [$lambdas[] | select(.after_deploy == "invoke")] + ' - aws s3 ls "$FULL_BUCKET_PATH" --recursive \ - | awk '{print $4}' \ - | xargs -n1 basename \ - | sed 's/\.[^.]*$//' \ - | grep -v 'appspec' \ - | jq -R . \ - | jq -s -c . +# Discover unique Lambda source directories to build from the deploy manifest. +lambda-get-build-matrix: + #!/usr/bin/env bash + set -euo pipefail + + deploy_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" lambda-get-deploy-matrix)" + + jq -cn --argjson lambdas "$deploy_matrix" ' + reduce $lambdas[] as $lambda + ([]; + if any(.[]; .source_dir == $lambda.source_dir) then + . + else + . + [{ + source_dir: $lambda.source_dir, + artifact_name: $lambda.artifact_name, + artifact_file: $lambda.artifact_file + }] + end + ) + ' + + +# Check every Lambda stack in the deploy manifest exists in runtime environments. +lambda-check-deploy-stacks: + #!/usr/bin/env bash + set -euo pipefail + + deploy_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" lambda-get-deploy-matrix)" + + ruby -rjson -rpathname -e ' + project_dir = Pathname(ARGV.fetch(0)).realpath + records = JSON.parse(STDIN.read) + + if records.empty? + puts "Lambda manifest declares no entries; skipping stack-path check." + exit 0 + end + + stack_templates = records.map { |record| record.fetch("stack") } + env_dirs = project_dir.glob("infra/live/*/aws").select(&:directory?) + missing = [] + checked = [] + + env_dirs.each do |aws_dir| + env_name = aws_dir.parent.basename.to_s + stack_paths = stack_templates.map { |template| template.gsub("{environment}", env_name) } + existing = stack_paths.select { |path| project_dir.join(path).directory? } + next if existing.empty? + + checked << env_name + stack_paths.each do |path| + missing << path unless project_dir.join(path).directory? + end + end -# List Lambda zip object keys for a published version. -get-version-file-keys: + if checked.empty? + abort("Error: no live environment contains any Lambda stack from lambdas/deploy.yml") + end + + unless missing.empty? + missing.each { |path| warn("::error::Missing Lambda stack path: #{path}") } + exit 1 + end + + puts "Lambda manifest stacks are present in: #{checked.sort.join(", ")}" + ' "{{PROJECT_DIR}}" <<< "$deploy_matrix" + + +# Check every Lambda artifact from the deploy manifest exists for the requested version. +lambda-check-deploy-artifacts: #!/usr/bin/env bash set -euo pipefail @@ -266,17 +392,16 @@ get-version-file-keys: exit 1 fi - FULL_BUCKET_PATH="s3://$BUCKET_NAME/lambdas/$VERSION/" - - aws s3api head-bucket --bucket "$BUCKET_NAME" >/dev/null - aws s3 ls "$FULL_BUCKET_PATH" >/dev/null + deploy_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" lambda-get-deploy-matrix)" - aws s3 ls "$FULL_BUCKET_PATH" --recursive \ - | awk '{print $4}' \ - | grep '\.zip$' \ - | grep -v 'appspec' \ - | jq -R . \ - | jq -s -c . + jq -r '.[].artifact_file' <<< "$deploy_matrix" \ + | sort -u \ + | while read -r artifact_file; do + key="lambdas/$VERSION/$artifact_file" + aws s3 ls "s3://$BUCKET_NAME/$key" >/dev/null \ + && echo "✅ $key found" \ + || (echo "❌ $key not found in s3://$BUCKET_NAME" && exit 1) + done # Return the Lambda artifact directory name from the repo-root justfile. @@ -299,132 +424,124 @@ code-bucket-get-appspec-artifact-dir: @just --justfile "{{PROJECT_DIR}}/justfile" code-bucket-get-appspec-artifact-dir -# List ECR image tag prefixes for a published version. -get-ecr-version-images: +# Render ECS service deploy manifest records as compact JSON. +ecs-get-deploy-matrix: #!/usr/bin/env bash set -euo pipefail - if [[ -z "${REPOSITORY_URL:-}" ]]; then - echo "❌ REPOSITORY_URL environment variable is not set." - exit 1 - fi - - if [[ -z "${VERSION:-}" ]]; then - echo "❌ VERSION environment variable is not set." - exit 1 - fi - - repository_name="${REPOSITORY_URL#*/}" - - aws ecr describe-images \ - --repository-name "$repository_name" \ - --query 'imageDetails[].imageTags[]' \ - --output text \ - | tr '\t' '\n' \ - | grep -- "-$VERSION\$" \ - | sed "s/-$VERSION$//" \ - | jq -R . \ - | jq -s -c . - + cd "{{PROJECT_DIR}}" -# Derive the ECS task matrix from published ECR image tags. -get-ecr-version-tasks: + ruby -ryaml -rjson -rpathname -e ' + manifest_path = ARGV.fetch(0) + project_dir = Pathname(ARGV.fetch(1)).realpath + manifest = YAML.safe_load(File.read(manifest_path), aliases: false) + + unless manifest.is_a?(Hash) && manifest["version"] == 1 + abort("Error: #{manifest_path} must declare version: 1") + end + + support_images = manifest["support_images"] || [] + unless support_images.is_a?(Array) && support_images.all? { |image| image.is_a?(String) && image.match?(/\A[a-z0-9_][a-z0-9_-]*\z/) } + abort("Error: #{manifest_path} support_images must be a list of simple image names") + end + + services = manifest["services"] || [] + unless services.is_a?(Array) + abort("Error: #{manifest_path} services must be a list when present") + end + + stacks = {} + records = services.each_with_index.map do |record, index| + unless record.is_a?(Hash) + abort("Error: services[#{index}] must be a mapping") + end + + task_stack = record["task_stack"] + service_stack = record["service_stack"] + image = record["image"] + + unless task_stack.is_a?(String) && task_stack.start_with?("infra/live/{environment}/aws/") + abort("Error: services[#{index}].task_stack must start with infra/live/{environment}/aws/") + end + + unless service_stack.is_a?(String) && service_stack.start_with?("infra/live/{environment}/aws/") + abort("Error: services[#{index}].service_stack must start with infra/live/{environment}/aws/") + end + + [["task_stack", task_stack], ["service_stack", service_stack]].each do |field, value| + if stacks.key?(value) + prior = stacks[value] + abort("Error: services[#{index}].#{field} duplicates services[#{prior[:index]}].#{prior[:field]}: #{value}") + end + stacks[value] = { index: index, field: field } + end + + unless image.is_a?(String) && image.match?(/\A[a-z0-9_][a-z0-9_-]*\z/) + abort("Error: services[#{index}].image must be a simple image name") + end + + source_path = project_dir.join("{{CONTAINERS_DIR}}", image) + unless source_path.directory? + abort("Error: services[#{index}].image source directory does not exist: {{CONTAINERS_DIR}}/#{image}") + end + + { + "task_stack" => task_stack, + "service_stack" => service_stack, + "image" => image + } + end + + puts JSON.generate(records) + ' "{{PROJECT_DIR}}/{{CONTAINERS_DIR}}/deploy.yml" "{{PROJECT_DIR}}" + + +# Discover unique ECS image records to build from the deploy manifest. +ecs-get-build-matrix: #!/usr/bin/env bash set -euo pipefail - image_names="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" get-ecr-version-images)" + deploy_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" ecs-get-deploy-matrix)" jq -cn \ - --argjson images "$image_names" \ + --argjson services "$deploy_matrix" \ + --argjson manifest "$(ruby -ryaml -rjson -e 'puts JSON.generate(YAML.safe_load(File.read(ARGV.fetch(0)), aliases: false))' "{{PROJECT_DIR}}/{{CONTAINERS_DIR}}/deploy.yml")" \ ' - $images - | map(select(. != "bootstrap" and . != "debug" and . != "otel_collector")) + ( + ($services | map(.image)) + ($manifest.support_images // []) + | unique + | map({image: .}) + ) ' -# Discover deployable Lambda directory names. -lambda-get-directories: - #!/usr/bin/env bash - set -euo pipefail - find "{{LAMBDA_DIR}}" -mindepth 1 -maxdepth 1 -type d \ - | xargs -I{} basename "{}" \ - | grep -v '^build$' \ - | grep -v '^lib$' \ - | grep -v '^_shared$' \ - | tr '-' '_' \ - | jq -R . \ - | jq -s -c . - - -# Discover deployable ECS service/container directory names. -service-get-directories: - #!/usr/bin/env bash - set -euo pipefail - - find "{{CONTAINERS_DIR}}" -mindepth 1 -maxdepth 1 -type d \ - | xargs -I{} basename "{}" \ - | jq -R . \ - | jq -sc --argjson reserved '{{NON_SERVICE_CONTAINER_DIRECTORIES}}' 'map(select(. as $name | ($reserved | index($name) | not)))' \ - | jq -r '.[]' \ - | tr '-' '_' \ - | jq -R . \ - | jq -s -c . - - -# Derive Terragrunt `task_*` stack names from service directories. -task-get-directories: - #!/usr/bin/env bash - set -euo pipefail - - found_dirs="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" service-get-directories)" - - jq -cn \ - --argjson found "$found_dirs" \ - '$found | map("task_" + .)' - - -# Resolve per-task ECS image URIs from the published image set. -ecs-task-get-image-uris: +# Check every ECS image from the deploy manifest exists for the requested version. +ecs-check-deploy-images: #!/usr/bin/env bash set -euo pipefail - if [[ -z "${ECS_IMAGE_URIS:-}" ]]; then - echo "❌ ECS_IMAGE_URIS environment variable is not set." + if [[ -z "${REPOSITORY_URL:-}" ]]; then + echo "❌ REPOSITORY_URL environment variable is not set." exit 1 fi - if [[ -z "${TASK_NAME:-}" ]]; then - echo "❌ TASK_NAME environment variable is not set." + if [[ -z "${VERSION:-}" ]]; then + echo "❌ VERSION environment variable is not set." exit 1 fi - service_name="${TASK_NAME#task_}" - - jq -cn \ - --argjson image_uris "$ECS_IMAGE_URIS" \ - --arg service_name "$service_name" \ - ' - { - service_image_uri: ($image_uris | map(select(test(":" + $service_name + "-")))[0] // ""), - debug_image_uri: ($image_uris | map(select(test(":debug-")))[0] // ""), - otel_image_uri: ($image_uris | map(select(test(":otel_collector-")))[0] // "") - } - | if .service_image_uri == "" then error("Missing ECS image URI for " + $service_name) else . end - | if .debug_image_uri == "" then error("Missing debug image URI") else . end - | if .otel_image_uri == "" then error("Missing otel_collector image URI") else . end - ' - - -# Derive Terragrunt `service_*` stack names from service directories. -ecs-service-get-directories: - #!/usr/bin/env bash - set -euo pipefail - - found_dirs="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" service-get-directories)" - - jq -cn \ - --argjson found "$found_dirs" \ - '$found | map("service_" + .)' + repository_name="${REPOSITORY_URL#*/}" + build_matrix="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" ecs-get-build-matrix)" + + jq -r '.[].image' <<< "$build_matrix" \ + | while read -r image; do + tag="$image-$VERSION" + aws ecr describe-images \ + --repository-name "$repository_name" \ + --image-ids "imageTag=$tag" >/dev/null \ + && echo "✅ $tag found" \ + || (echo "❌ $tag not found in $REPOSITORY_URL" && exit 1) + done # Discover repo-local Docker-based GitHub action directories. @@ -443,19 +560,6 @@ action-get-directories: | jq -s -c . -# Discover container build targets including shared CI extras. -container-get-directories: - #!/usr/bin/env bash - set -euo pipefail - - found_dirs="$(just --justfile "{{PROJECT_DIR}}/justfile.ci" service-get-directories)" - - jq -cn \ - --argjson found "$found_dirs" \ - --argjson extra '{{EXTRA_CONTAINER_DIRECTORIES}}' \ - '$found + $extra | unique' - - # Check that the requested frontend artifact version exists in the code bucket. frontend-check-version: #!/usr/bin/env bash diff --git a/justfile.deploy b/justfile.deploy index 2a80548f..e8e6e1f5 100644 --- a/justfile.deploy +++ b/justfile.deploy @@ -98,11 +98,20 @@ lambda-build: #!/usr/bin/env bash set -euo pipefail - if [[ -z "$LAMBDA_NAME" ]]; then - echo "❌ LAMBDA_NAME environment variable is not set." + if [[ -z "${LAMBDA_SOURCE_DIR:-}" && -z "${LAMBDA_NAME:-}" ]]; then + echo "❌ LAMBDA_SOURCE_DIR or LAMBDA_NAME environment variable is required." exit 1 fi + lambda_source_dir="${LAMBDA_SOURCE_DIR:-{{LAMBDA_DIR}}/$LAMBDA_NAME}" + lambda_artifact_name="${LAMBDA_ARTIFACT_NAME:-$(basename "$lambda_source_dir")}" + + if [[ "$lambda_source_dir" = /* ]]; then + lambda_source_path="$lambda_source_dir" + else + lambda_source_path="{{PROJECT_DIR}}/$lambda_source_dir" + fi + python3 -m venv .venv source .venv/bin/activate @@ -111,21 +120,21 @@ lambda-build: echo "🔄 Cleaning previous builds..." rm -rf $LAMBDA_BUILD_DIR - echo "📦 Building $LAMBDA_NAME Lambda..." - pip install --target "$LAMBDA_BUILD_DIR/$LAMBDA_NAME" -r "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$LAMBDA_NAME/requirements.txt" - cp "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$LAMBDA_NAME"/*.py "$LAMBDA_BUILD_DIR/$LAMBDA_NAME/" - cp "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/lib/lambda_shared.py" "$LAMBDA_BUILD_DIR/$LAMBDA_NAME/" - cp "{{PROJECT_DIR}}/lib/db_shared.py" "$LAMBDA_BUILD_DIR/$LAMBDA_NAME/" - cp "{{PROJECT_DIR}}/lib/runtime_logging.py" "$LAMBDA_BUILD_DIR/$LAMBDA_NAME/" - if [[ -d "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$LAMBDA_NAME/database_models" ]]; then - cp -R "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$LAMBDA_NAME/database_models" "$LAMBDA_BUILD_DIR/$LAMBDA_NAME/" + echo "📦 Building $lambda_artifact_name Lambda from $lambda_source_dir..." + pip install --target "$LAMBDA_BUILD_DIR/$lambda_artifact_name" -r "$lambda_source_path/requirements.txt" + cp "$lambda_source_path"/*.py "$LAMBDA_BUILD_DIR/$lambda_artifact_name/" + cp "{{PROJECT_DIR}}/{{LAMBDA_DIR}}/lib/lambda_shared.py" "$LAMBDA_BUILD_DIR/$lambda_artifact_name/" + cp "{{PROJECT_DIR}}/lib/db_shared.py" "$LAMBDA_BUILD_DIR/$lambda_artifact_name/" + cp "{{PROJECT_DIR}}/lib/runtime_logging.py" "$LAMBDA_BUILD_DIR/$lambda_artifact_name/" + if [[ -d "$lambda_source_path/database_models" ]]; then + cp -R "$lambda_source_path/database_models" "$LAMBDA_BUILD_DIR/$lambda_artifact_name/" fi - find "$LAMBDA_BUILD_DIR/$LAMBDA_NAME" -type f -name '*.md' -delete + find "$LAMBDA_BUILD_DIR/$lambda_artifact_name" -type f -name '*.md' -delete ( - cd "$LAMBDA_BUILD_DIR/$LAMBDA_NAME" - zip -r "../../$LAMBDA_NAME.zip" . > /dev/null + cd "$LAMBDA_BUILD_DIR/$lambda_artifact_name" + zip -r "../../$lambda_artifact_name.zip" . > /dev/null ) - echo "✅ Done: lambdas/$LAMBDA_NAME.zip" + echo "✅ Done: lambdas/$lambda_artifact_name.zip" # Upload a packaged Lambda zip to the shared code bucket. @@ -133,8 +142,8 @@ lambda-upload: #!/usr/bin/env bash set -euo pipefail - if [[ -z "$LAMBDA_NAME" ]]; then - echo "❌ LAMBDA_NAME environment variable is not set." + if [[ -z "${LAMBDA_SOURCE_DIR:-}" && -z "${LAMBDA_NAME:-}" ]]; then + echo "❌ LAMBDA_SOURCE_DIR or LAMBDA_NAME environment variable is required." exit 1 fi @@ -148,10 +157,13 @@ lambda-upload: exit 1 fi - LAMBDA_ZIP="{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$LAMBDA_NAME.zip" - echo "📤 Uploading $LAMBDA_ZIP to s3://$BUCKET_NAME/lambdas/$VERSION/$LAMBDA_NAME.zip" + lambda_source_dir="${LAMBDA_SOURCE_DIR:-{{LAMBDA_DIR}}/$LAMBDA_NAME}" + lambda_artifact_name="${LAMBDA_ARTIFACT_NAME:-$(basename "$lambda_source_dir")}" + + LAMBDA_ZIP="{{PROJECT_DIR}}/{{LAMBDA_DIR}}/$lambda_artifact_name.zip" + echo "📤 Uploading $LAMBDA_ZIP to s3://$BUCKET_NAME/lambdas/$VERSION/$lambda_artifact_name.zip" - aws s3 cp "$LAMBDA_ZIP" "s3://$BUCKET_NAME/lambdas/$VERSION/$LAMBDA_NAME.zip" \ + aws s3 cp "$LAMBDA_ZIP" "s3://$BUCKET_NAME/lambdas/$VERSION/$lambda_artifact_name.zip" \ --storage-class STANDARD diff --git a/lambdas/README.md b/lambdas/README.md index c8eed717..ee0d227e 100644 --- a/lambdas/README.md +++ b/lambdas/README.md @@ -4,10 +4,11 @@ Lambda source directories for this boilerplate. ## Structure -- each top-level directory under `lambdas/` is treated as a deployable Lambda +- `deploy.yml` is the Lambda build/deploy manifest +- each entry in `deploy.yml` maps a Lambda source directory to a live Terragrunt stack path template - the generated `lambdas/build` directory is build output only and is intentionally excluded from Lambda discovery - `lambdas/lib/` contains Lambda-only helper modules and is intentionally excluded from Lambda discovery -- a deployable Lambda also needs a corresponding live Terragrunt stack under `infra/live//aws//terragrunt.hcl` +- a deployable Lambda also needs the live Terragrunt stack declared by its manifest `stack` value ## Common Shape @@ -18,12 +19,17 @@ Lambda source directories for this boilerplate. ## Build Behavior -- Lambda directory discovery auto-detects top-level directories under `lambdas/` for build and deploy workflows -- discovery excludes `lambdas/build` and `lambdas/lib` +- Lambda discovery reads `lambdas/deploy.yml` +- `stack` is a repo-relative Terragrunt stack path template and must use `{environment}` for the environment segment, for example `infra/live/{environment}/aws/lambda_api` +- `source_dir` is the repo-relative source directory to package, for example `lambdas/lambda_api` +- the zip artifact name is computed from `basename(source_dir)`, so `lambdas/lambda_api` publishes `lambdas//lambda_api.zip` +- build workflows deduplicate by `source_dir`; deploy workflows keep every manifest entry so the same source can roll out to multiple Lambda stacks +- wrapper workflows do not pass Lambda matrices; update this manifest to add, remove, or remap deployed Lambdas +- `after_deploy: invoke` can be set on a manifest entry when the deployed Lambda should be invoked after CodeDeploy completes - the Lambda build flow installs `requirements.txt` into a per-Lambda build directory - it copies Python source files, shared helpers from `lib/` and `lambdas/lib/`, and supported package directories into the zip artifact - markdown files in Lambda source trees are documentation only and are pruned before the zip artifact is created -- detection alone is not enough: the runtime still needs the matching Terragrunt stack to participate in infra apply and code rollout correctly +- manifest detection alone is not enough: the runtime still needs the declared Terragrunt stack to participate in infra apply and code rollout correctly ## Boilerplate Patterns diff --git a/lambdas/deploy.yml b/lambdas/deploy.yml new file mode 100644 index 00000000..6ccbbfe1 --- /dev/null +++ b/lambdas/deploy.yml @@ -0,0 +1,16 @@ +version: 1 + +lambdas: + - stack: infra/live/{environment}/aws/lambda_api + source_dir: lambdas/lambda_api + + - stack: infra/live/{environment}/aws/lambda_worker + source_dir: lambdas/lambda_worker + + - stack: infra/live/{environment}/aws/migrations + source_dir: lambdas/migrations + after_deploy: invoke + + - stack: infra/live/{environment}/aws/rds_reader_tagger + source_dir: lambdas/rds_reader_tagger + after_deploy: invoke