chore(ingestion): put the raw zone in us-west4, beside Cloud SQL - #843
Merged
Conversation
The buckets were created in US-CENTRAL1 from a default chosen before the database region was known. The dataservices instance is in us-west4, so every Mode B replay would have read across regions and paid egress to reach the loader. Bucket location is immutable, so applying this replaces both buckets. Verified empty first -- zero objects in each -- which makes now the only cheap moment to do it. After a backfill has landed, moving regions means copying objects and re-pointing the pipeline rather than editing a variable, which the comment on the resource now says. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.27% total — gate is 75%. |
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The buckets landed in
US-CENTRAL1, from a default I chose before the databaseregion was known. The
dataservicesinstance is in us-west4, so every Mode Breplay would read across regions and pay egress to reach the loader.
This replaces the buckets
Bucket location is immutable.
terraform applywill destroy and recreateboth.
That is safe right now and only right now — I verified both are empty before
making the change:
Once a backfill has landed, changing regions means copying objects across and
re-pointing the pipeline, not editing a variable. The resource now carries that
warning so the next person does not learn it from a plan output.
Applying
cd automated_ingestion/iac terraform apply -var project_id=waterdatainitiative-271000 -var cloud_sql_instance=dataservicesRead the plan before confirming. It should show two buckets replaced and
their IAM members recreated, and nothing else destroyed.
force_destroyisfalse, so if either bucket has acquired objects since I checked, Terraform willrefuse rather than delete data — which is the intended behaviour.
Passing
cloud_sql_instancealso creates the Cloud SQL grants and the IAMdatabase user, which
database_connectivityneeds next. That variable takes thebare instance name, unlike
CLOUD_SQL_INSTANCE_NAME, which needs the fullPROJECT:REGION:INSTANCEconnection string.🤖 Generated with Claude Code