feat(ingestion): schedule the San Acacia ingest weekly - #860
Merged
Conversation
One job over the san_acacia asset group, selected by group so a fourth asset joins the schedule without touching the job. Dagster orders the three steps from their dependencies. Weekly rather than the daily cadence the plan assumed. These are five-minute diver readings nobody watches in real time, the vendor's endpoint answers 500 when pushed, and the watermark makes the interval a question of freshness rather than correctness -- a run resumes from wherever the last one finished, so a missed week is caught up rather than lost. Mondays 05:00 America/Denver rather than UTC: the wells, the people reading the data and the working day are all in one timezone, so a schedule that shifts an hour twice a year would be the surprising choice. Stopped by default. Turning it on begins writing to Ocotillo, and the first run for the 24 wells without history fetches back to the floor. That is a decision somebody should take once, not a consequence of a merge. A test pins the selection to exactly the three ingest assets. A group name is a string, and a typo would leave a schedule that runs successfully and ingests nothing, which looks like everything working. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 77.97% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of task 4.4.
san_acacia_ingestover the wholesan_acaciaasset group,driven by
san_acacia_weekly.Weekly, not the daily cadence the plan assumed
These are five-minute diver readings nobody watches in real time, and the
vendor's endpoint answers 500 when pushed. The watermark makes the interval a
question of freshness rather than correctness: each run resumes from where
the last finished, so a missed week is caught up by the next run rather than
lost.
Mondays 05:00
America/Denver, not UTC. The wells, the people reading thedata and the working day are all in one timezone — a schedule drifting an hour
twice a year would be the surprising choice. After midnight so a run covers
whole days; early enough that a failure is visible at the start of the week
rather than discovered the following Monday.
Stopped by default
DefaultScheduleStatus.STOPPED. Turning it on starts writing to Ocotillo,and the first run for the 24 wells without history fetches back to
INITIAL_START. That should be a decision taken once, deliberately — not aconsequence of merging this.
Start it in the Dagster+ UI when you want it running.
Selected by group
So a fourth asset added to
san_acaciajoins the schedule without touching thejob. Dagster orders the three steps from their asset dependencies rather than
anything declared here.
A test pins the selection to exactly
raw_san_acacia_locations,raw_san_acacia_readingsandsan_acacia_observations, and assertsingestion_heartbeatanddatabase_connectivityare excluded. A group name isa string: a typo would leave a schedule that runs successfully and ingests
nothing, which looks exactly like everything working.
Retries
RetryPolicy(max_retries=2, delay=60)— enough for a dropped request or a tokenexpiring mid-run. Not more: a persistent 500 means the window is wrong or the
endpoint is unwell, and hammering it makes both worse.
Still open in 4.4
Observability and alerting — log bridge, failure notification, run metadata.
107 tests pass.
🤖 Generated with Claude Code