ci(dagster): deploy code location as PEX instead of an image - #863
Merged
Conversation
Both Dagster+ CD workflows built and pushed a container image to ECR on every run, which dominated deploy time even when only ingestion source changed. Serverless deploys can instead publish two PEX files -- deps.pex and source.pex -- and reuse a cached deps.pex when the resolved requirements are unchanged, so a source-only change uploads ~9MB rather than rebuilding a full image. Each workflow now runs actions/utils/prerun, which reports pex-deploy or docker-deploy from ENABLE_FAST_DEPLOYS. The PEX path is the default; the old image build survives verbatim in a second job reached by setting that variable to 'false', as the escape hatch for anything PEX cannot express (a dependency with no Linux wheel that also fails to build from source, a system package needing apt). Every dependency currently resolves to a linux/cp313 wheel, so neither case applies today. Two details are load-bearing: `--no-hashes` on the PEX export, and only there. The PEX builder unions requirements.txt with [project].dependencies from pyproject.toml, and those pins carry no hashes -- a hashed requirements.txt would put pip in --require-hashes mode, where every unhashed line is a hard error. Both sources resolve from the same uv.lock, so the duplicate pins agree. The Docker path keeps hashes; it feeds the file straight to pip install -r with nothing unhashed mixed in. python_version 3.13, against an action default of 3.8. The PEX files are resolved for one interpreter, and requires-python is >= 3.13. On the branch workflow, prerun also handles the closed-PR teardown: it runs `ci branch-deployment` to mark the deployment closed and reports skip, so the docker action's own closed-PR branch is no longer reached. dagster_cloud_post_install.sh is unchanged apart from its header. Only the Docker path still runs it; the source-pex builder performs the same `uv pip install --no-deps .` of this repository itself, which is what keeps db/ and domain/ importable from the process that executes a step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 78.59% total — gate is 75%. |
Contributor
Coverage✅ 78.60% total — gate is 75%. Coverage for the Python files changed in this PR
|
A successful deploy is weaker evidence than a successful run. The agent loading the code location proves the loader process can import the package; it says nothing about the process that executes a step, which is a different process with a different sys.path. That gap is why assets/heartbeat.py exists, and it matters more now that the code location ships as PEX files rather than an image, because the two package the repository by different routes. Nothing could launch that asset from CI. dagster-cloud-action's launch_job identifies what to run by job name and exposes no asset selection, so an asset reachable only through the implicit __ASSET_JOB is unreachable. So wrap it in a named job, ingestion_heartbeat_check, and add a workflow_dispatch workflow that launches it against a deployment given as an input -- a branch deployment id or prod -- with wait: true so the workflow result is the materialization result rather than just "a run was launched". Dispatch-only on purpose. It costs a Dagster+ run, and the interesting time to spend one is after a deploy that changed how the code location is packaged, not on every push. No retry policy on the job: a retry would mask exactly the failure it exists to surface, since an import that works at load time and fails at execution does so deterministically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dispatch-only smoke test cannot cover the case it was written for. GitHub offers workflow_dispatch only for workflows present on the default branch, so a PR that changes how the code location is packaged -- exactly when the check is worth running -- cannot run it. So run it from the deploy workflow, as the last step of the PEX path. That also fixes the ordering for free: a separate workflow would race the deploy and could launch the job before the agent has synced the new code location, or before the job exists in it at all. The deployment name has to be resolved rather than assumed. It is derived from the branch and build_deploy_python_executable does not report it, so ask for it with the same `ci branch-deployment` call the deploy makes internally; that call is idempotent and returns the existing deployment for the PR. smoke_dagster_location.yml stays, with its purpose narrowed to the cases the deploy workflow does not reach: prod, and branch deployments that predate this step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 78.60% total — gate is 75%. Coverage for the Python files changed in this PR
|
The previous commit claimed a green check meant the asset materialized. It did not. `launch_job`'s run.sh captures the CLI output in a command substitution and never checks the exit code, deciding success by whether it can regex a run id out of the text. Underneath, `dagster-cloud job launch --wait` reports a failed run with `ui.error(...)` -- and `ui.error` only returns an exception rather than raising it, so the CLI exits 0 as well. The action documents "fail if the run fails"; at neither layer can it. A failed materialization produced a passing step. Call the CLI directly and require the "finished successfully" line. Matching on output is not lovely, but it is the only signal either layer emits, and an assertion that can fail is worth more than one that reads better. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 78.60% total — gate is 75%. Coverage for the Python files changed in this PR
|
jirhiker
added a commit
that referenced
this pull request
Aug 19, 2026
Picks up the Dagster PEX branch-deploy work from #863. The branch-deploy CI step that materializes `ingestion_heartbeat_check` came from staging via the pull_request merge ref, but the job it launches is defined in automated_ingestion/defs/jobs/heartbeat.py, which this branch predated -- so the deployed code location had no such job and the launch failed with PipelineNotFoundError. Merging brings the definition along with the step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Both Dagster+ CD workflows built and pushed a container image to ECR on every run, which dominated deploy time even when only ingestion source changed. Serverless deploys can instead publish two PEX files —
deps.pexandsource.pex— and reuse a cacheddeps.pexwhen the resolved requirements are unchanged, so a source-only change uploads ~9MB rather than rebuilding a full image. See Fast deploys with PEX and Docker.What
Each workflow now runs
actions/utils/prerun, which reportspex-deployordocker-deployfromENABLE_FAST_DEPLOYS.project-repo/, export requirements,build_deploy_python_executable. No image build.ENABLE_FAST_DEPLOYS: "false". Escape hatch for anything PEX cannot express — a dependency with no Linux wheel that also fails to build from source, or a system package needing apt. Neither applies today.On the branch workflow,
prerunalso handles the closed-PR teardown: it runsci branch-deploymentto mark the deployment closed and reportsskip, so the docker action's own closed-PR branch is no longer reached.dagster_cloud_post_install.shis unchanged apart from its header. Only the Docker path still runs it; the source-pex builder performs the sameuv pip install --no-deps .of this repository itself, which is what keepsdb/anddomain/importable from the process that executes a step.Two details are load-bearing
--no-hasheson the PEX export, and only there. The PEX builder unionsrequirements.txtwith[project].dependenciesfrompyproject.toml, and those pins carry no hashes — a hashedrequirements.txtwould put pip in--require-hashesmode, where every unhashed line is a hard error. Both sources resolve from the sameuv.lock, so the duplicate pins agree. The Docker path keeps hashes; it feeds the file straight topip install -rwith nothing unhashed mixed in.python_version: 3.13, against an action default of3.8. The PEX files are resolved for one interpreter, andrequires-pythonis>= 3.13.Verification
actionlinton both workflowsx86_64-unknown-linux-gnu/ py3.13--no-buildsource.pexbuilt from a clean cloneocotilloapi==1.2.0installedautomated_ingestion/,db/,domain/,services/,core/,schemas/, plusworking_directory/root/Because all dependencies are wheels there is no sdist compile. The runner is
ubuntu-latest(24.04), so the action buildsdeps.pexinside apython:3.13-slimcontainer to match the serverless base image — but only when the dependency hash changes.ubuntu-22.04would skip that container too; it is not used here because that runner label is being retired.One layout difference worth knowing:
core/lexicon.jsonlands only underworking_directory/root/, not in the installedcore/package, soservices/lexicon_helper.py:121would not find it. Harmless —automated_ingestion/imports exactly one repo module,domain.van_essen, and never touchesservices/orcore/. The same gap already exists in the Docker image.Measured on this PR
This PR matches the branch workflow's path filter, so it deployed itself through the new PEX path three times.
skipping rebuild— cache hitSame dependency hash (
31ffa29d…) both times, and identical to what the same export reproduces locally.source.pexbuilds in ~5s.The heartbeat now materializes as the last step of the PEX path, so every ingestion PR proves a step process can execute and not merely that the agent could load the location. Verified on branch deployment
4206bfdf…: run3194f52c-5866-4daa-9533-a56f6dee4280finished successfully.One thing reviewers should know
launch_jobis not used, and the success assertion is hand-rolled, because the vendor's gate does not work. Itsrun.shcaptures the CLI output in a command substitution and never checks the exit code — it decides success by regexing a run id out of the text. Underneath,dagster-cloud job launch --waitreports a failed run viaui.error(...), andui.erroronly returns an exception rather than raising it, so the CLI exits 0 too. The action documents "fail if the run fails"; at neither layer can it. The workflow therefore requires thefinished successfullyline itself.Still open
A green heartbeat proves the step process runs user code under PEX. It does not prove
dbanddomainare importable from a step: the asset reportsdb_on_path/domain_on_pathviafind_specas metadata and does not fail when either isFalse. Both packages are present at the root ofsource.pex(verified by unzipping it), which is onsys.path, but the direct confirmation is those two metadata values on run3194f52c…in the Dagster+ UI. Worth a glance before merge.🤖 Generated with Claude Code