fix(ingestion): grant bucket read, and name the pipeline after the bucket - #845
Merged
Conversation
…cket raw_san_acacia_locations failed with "Bucket does not exist: ocotillo-ingestion-production" against a bucket that exists. objectAdmin covers objects and says nothing about the bucket, so it omits storage.buckets.get. gcsfs checks a bucket exists before writing, that check was denied, and GCS reports denial as absence -- the same 404-for-403 shape as the Secret Manager failure earlier. legacyBucketReader adds buckets.get and objects.list and nothing else; storage.admin would also permit deleting the bucket. The same traceback showed a pipeline named san_acacia_staging writing to the production bucket. The name came from a run tag that was absent, defaulting to staging, while the bucket came from the environment: two sources of truth for one fact, free to disagree. The name is now derived from the bucket, so they cannot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.30% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
raw_san_acacia_locationsfailed withBucket does not exist: ocotillo-ingestion-productionagainst a bucket that plainly exists. Twoseparate bugs in one traceback.
objectAdmin does not include
storage.buckets.getIt covers objects and says nothing about the bucket itself. gcsfs checks a
bucket exists before writing to it, that check was denied, and GCS reports a
denial as absence — the same 404-for-403 shape as the Secret Manager failure
earlier in this work.
legacyBucketReaderaddsbuckets.getandobjects.listand nothing else. Itis the narrowest standard role that makes the check succeed;
storage.adminwould also permit deleting the bucket.
The pipeline name and the bucket could disagree
The same traceback shows a pipeline named
san_acacia_stagingwriting tothe production bucket. The name came from
context.run.tags.get("environment", "staging")— a tag that was absent, so it defaulted — while the bucket camefrom
INGESTION_GCS_BUCKET. Two sources of truth for one fact.The name is now derived from the bucket, so they cannot drift. A test asserts
it.
That mismatch was harmless here only because the load failed. Had the grant been
right, this run would have written production data under a pipeline named
staging, and the dlt state for the two environments would have been kept in the
same place.
Applying
The Terraform half needs an apply:
cd automated_ingestion/iac terraform apply -var project_id=waterdatainitiative-271000 -var cloud_sql_instance=dataservicesExpect 2 to add, nothing destroyed. The code half needs the merge to deploy.
Note on the failed run
dlt left a pending load package. On Serverless the pipeline working directory is
ephemeral, so it is discarded rather than retried — the message about
abort-packagesdoes not apply here.49 tests pass.
🤖 Generated with Claude Code