fix(ingestion): set code location env vars at a scope that reaches the container - #842
Merged
Conversation
…e container Dropping --global scoped variables to `OcotilloAPI` -- the project name -- rather than to `ocotillo-automated-ingestion`, the name in dagster_cloud.yaml. Dagster+ accepts the unknown location without complaint, so INGESTION_GCP_CREDENTIALS_JSON appeared correctly set in the UI while the container never received it, and the asset kept failing with DefaultCredentialsError as though no key had been set. `code_location_name` in [tool.dg.project] looks like the fix and is ignored for this command, so that avenue is closed and the attempt is not left in the file. Back to --global, which is broader than ideal: this deployment also hosts aqueduct_dagster_defs_definitions and die-orchestration, which can read these values. That is a real cost, recorded in the script along with the alternative -- scoping in the Dagster+ UI against the correct location name -- rather than traded away silently. A credential the container cannot read is worth less than one scoped more broadly than we would like. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLOUD_SQL_INSTANCE_NAME was set to `dataservices`, which every layer accepts until the connector tries to use it and raises a ValueError from inside the driver, well below anything this project wrote. The connector wants the full connection name, PROJECT:REGION:INSTANCE. The script now checks the shape before setting it, and points at the gcloud command that prints the right value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.27% total — gate is 75%. |
1 similar comment
Contributor
Coverage✅ 79.27% total — gate is 75%. |
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
database_connectivitykept failing withDefaultCredentialsErrorafter thekey was set and verified in the Dagster+ UI. The key was real; the container
never saw it.
Cause
Dropping
--globalfrom the setup script scoped variables toOcotilloAPI—the project name from
pyproject.toml— rather than toocotillo-automated-ingestion, thelocation_nameindagster_cloud.yaml.Dagster+ accepts an unknown location name without complaint. The variable shows
as set in the UI, is injected into nothing, and the failure is indistinguishable
from never having set it.
Confirmed by querying the deployment:
The variables set before I removed
--globalwere fine all along. Only whatcame after was affected — the credential, and
PYTHONPATH.code_location_namein[tool.dg.project]looks like it should fix this and isignored for this command; I tried it, confirmed dg still reported "location
OcotilloAPI", and removed it rather than leaving a setting that does nothing.
The trade
Back to
--global. That is broader than I would like — this deployment alsohosts
aqueduct_dagster_defs_definitionsanddie-orchestration, which can nowread these values. The alternative is scoping in the Dagster+ UI against the
correct location name, which the script now says explicitly.
I removed
--globalin the first place to keep credentials off a shareddeployment, which was the right instinct and the wrong mechanism. A credential
the container cannot read is worth less than one scoped more broadly than ideal,
but the cost is recorded rather than traded away silently.
To finish
Re-set the credential at deployment scope:
PYTHONPATHis already re-set. Environment variables apply on the next run, sono rebuild is needed.
🤖 Generated with Claude Code