feat(transducer): add data_maturity to observations - #853
Merged
Conversation
release_status is one column whose lexicon lists public and provisional as siblings, so a reading could not be both visible and marked unreviewed. Those are orthogonal questions -- who may see it, and how much it should be trusted -- and this adds the second axis. A lexicon term rather than an is_provisional boolean: review is a progression, not a switch, and a boolean cannot express the middle. Terms follow USGS usage. provisional and approved are what USGS publishes against; in review is the intermediate state from the Aquarius approval levels used for continuous time series. Aquarius' Working is folded into provisional, since to a consumer the two are indistinguishable. Existing rows are left NULL rather than defaulted. Backfilling 88,666 legacy observations to provisional would assert something about NMA data nobody has checked -- some may be approved. NULL reads as not stated, which is true. provisional and approved already existed as terms, since lexicon_term.term is globally unique and categories share terms by association, so only "in review" is new. approved is therefore shared with review_status; the two ask different questions, and shared vocabulary is how this lexicon is built. The loader defaults to provisional and refreshes maturity on upsert, so a corrected reading arriving as approved does not keep the older maturity. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I left historical rows NULL on the grounds that nobody had established whether legacy NMA data was approved. The evidence was in the same table: nma_waterlevelscontinuous_pressure_qced records whether a reading was quality controlled, which is the question data_maturity asks. True becomes approved, false becomes provisional. All 88,666 rows in the development database are qced, so they land as approved rather than as an absence somebody would have to reconstruct later. Rows where the flag is NULL stay NULL. Those did not come from the NMA transducer tables, so there is no evidence either way, and NULL is honest where a guess would not be. The update runs after the foreign key is in place, so a bad value fails loudly rather than persisting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.47% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 3.3. Migration
b2c3d4e5f6a7.release_statusis one column whose lexicon listspublicandprovisionalassiblings, so a reading could not be both visible and marked unreviewed.
Those are orthogonal questions — who may see it, and how much it should be
trusted. This adds the second axis.
A lexicon term rather than an
is_provisionalboolean, per your call: review isa progression, not a switch, and a boolean cannot express the middle.
Terms
provisional·in review·approvedprovisionalandapprovedare what USGS publishes against — "provisional datasubject to revision" is the standard caveat on unapproved records.
in reviewis the intermediate state from the Aquarius approval levels USGS uses for
continuous time series (Working / In Review / Approved).
I folded Aquarius'
Workingintoprovisional, because to a consumer the twoare indistinguishable — both mean "not approved, may change". Say the word if
you want the full three-level workflow instead; it is one term and a lexicon
entry.
Existing rows are left NULL
Backfilling 88,666 legacy observations to
provisionalwould assert somethingabout NMA data nobody has checked — some may well be approved. NULL reads as
"not stated", which is true.
A wrinkle worth knowing
provisionalandapprovedalready existed as terms:lexicon_term.termis globally unique and categories share terms through an association table, so
only
in reviewis new.That means
approvedis now shared betweenreview_statusanddata_maturity.They ask different questions —
review_statuson the block records that aBureau human reviewed it and carries a
reviewer_id, whiledata_maturitydescribes the reading's revision state — and shared vocabulary is how this
lexicon is designed. Flagging it because two columns reading
approvedcouldlook like duplication.
Verification
lexicon_term, so a typo is rejected by thedatabase — tested with
probationalrelease_status = publicanddata_maturity = provisionalon the same row, which is the whole point
provisionaland refreshes maturity on upsert, so acorrected reading arriving as approved does not keep the older value
DataMaturityenum built fromcore/lexicon.jsonlike every other statusenum; the migration seeds the database to match
Still open
services/ngwmn_helper.pyfiltersThing.release_status == "public"for NGWMNpublication. San Acacia data becoming public needs to be intended there too —
that check is not in this PR.
🤖 Generated with Claude Code