Skip to content

docs: document BernCo HydroVu source schema (ST2DAT-129) - #46

Merged
crawson-icasa merged 1 commit into
mainfrom
ST2DAT-129-docs-bernco_hydrovu-document-BernCo-HydroVu-source-schema
Aug 25, 2026
Merged

docs: document BernCo HydroVu source schema (ST2DAT-129)#46
crawson-icasa merged 1 commit into
mainfrom
ST2DAT-129-docs-bernco_hydrovu-document-BernCo-HydroVu-source-schema

Conversation

@crawson-icasa

@crawson-icasa crawson-icasa commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Why

  • BernCo HydroVu is one of the next sources to onboard and had no required source mapping doc

How

  • Added docs/sources/bernco_hydrovu.md based on _mapping_template.md.
  • Confirmed the credentials. hydrovu_bernco is in Secret Manager and its {id, secret} payload already matches what _resolve_hydrovu_credentials() expects, so no credential code has to change
  • Surveyed all 53 locations against the live tenant instead of sampling one, covering device profiles, parameter and unit inventory, cadence, and history depth
  • Mapped all six entities, with every fixed value traced to sources/hydrovu/adapter.py and canonical/canonical_constants.py
  • Documented the unit conversion. parameterId="4" arrives as unitId="35" (metres) at every location, so PVACD's × 3.28084 carries over unchanged
  • Split the Sensor section into "exists in canonical_constants.py" and "applies to this source", because 11 of the template's 13 names are not yet defined in canonical_constants.py
  • Included sanitized, JSON-validated response examples for all three endpoints

Notes

Any special considerations, workarounds, or follow-up work to note?

  • deploy/30_dagster_gcp_auth.sh grants secretAccessor on hydrovu_pvacd only. The Dagster service account needs the same grant on hydrovu_bernco before this runs in production
  • Found two code discrepancies: dlt_pipeline.py:191-193 logs 404 as "no data endpoint" when it means no data at or after startTime, and 13 of BernCo's 14 404s return full history at startTime=0. dlt_pipeline.py:130-137 says the API has no endTime parameter, but the OpenAPI spec lists one
  • Three locations sit at In-Situ's Fort Collins factory-default coordinates, and one of them carries live DTW data. Two more return sentinel timestamps at epoch 0 and 960
  • A location's parameter set changes over time. 35 locations have carried DTW against 29 today, so any backfill allowlist has to come from historical coverage
  • Ten locations are VuLink gateways reporting only their own telemetry. Exclude them from the allowlist rather than filtering them at transform time
  • initial_start_date is still open. History runs back to 2009-05-18, so someone has to pick how much of it to load
  • Multi-tenant HydroVu stays unmodeled: one flat [sources.hydrovu] block, one registry entry. Open Questions Scaffold dagster_dlt(ST2DAT78) #1 proposes a design, this PR implements none of it

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
aqueduct_dagster_defs_definitions View in Cloud Aug 25, 2026 at 01:02 AM (UTC)

**Source timezone:** UTC (HydroVu `timestamp` is Unix epoch seconds). See the device-clock caveat below.
**Update frequency:** 4-hourly for most wells. Observed cadences across the 29 DTW locations: 4 h (19), 1 h (4), 6 h (3), 24 h (2), 1 min (1, `LVCC`)

**Credentials:** GCP Secret Manager, project `waterdatainitiative-271000` (project number

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would probably remove project and number from this doc, otherwise looks good

@crawson-icasa
crawson-icasa merged commit afd8391 into main Aug 25, 2026
6 checks passed
@crawson-icasa
crawson-icasa deleted the ST2DAT-129-docs-bernco_hydrovu-document-BernCo-HydroVu-source-schema branch August 25, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants