Skip to content

feat(ogc): add the water well field operations internal layer - #913

Closed
jirhiker wants to merge 2 commits into
stagingfrom
feat/amp-field-operations-ogc-layer
Closed

feat(ogc): add the water well field operations internal layer#913
jirhiker wants to merge 2 commits into
stagingfrom
feat/amp-field-operations-ogc-layer

Conversation

@jirhiker

@jirhiker jirhiker commented Sep 1, 2026

Copy link
Copy Markdown
Member

What

A new OGC API - Features collection, water_well_field_operations, on
/ogcapi-internal only. One feature per water well, carrying what a field crew
needs to plan and execute a visit.

Every other well layer answers a scientific question — water level, chemistry,
trend. This one answers an operational one: may we go to this well, what are
we allowed to do when we get there, and is it overdue?

86 columns: identity and location (latitude/longitude in decimal degrees, plus
how the elevation was obtained), construction detail, the current measuring
point, all five status types, the three standing landowner permissions,
monitoring frequency and group membership, manual water-level and
chemistry-sampling statistics, field-visit history, the installed logger and
its continuous record, landowner contacts, and access and directions notes.

Column-by-column sources: docs/water-well-field-operations-columns.md.
Design rationale: docs/water-well-field-operations-layer.md.

Two relations, not one

Relation Kind Holds
ogc_internal_water_well_field_operations_stats materialized, nightly refresh Aggregates over observation and transducer_observation
ogc_internal_water_well_field_operations plain view, served by pygeoapi Live join of thing, location, the four history tables, deployment, sensor, contact, notes, LEFT JOINed to the stats matview

The split is not tidiness. Staleness is dangerous on exactly the columns that
are cheap to read: a revoked sampling permission that still reads true
until the next nightly refresh sends a crew onto land they are no longer
welcome on.
The current-record rule is also written against CURRENT_DATE,
which inside a materialized view would freeze at refresh time — a permission
that expired this morning would still read current tonight.

Please review these three decisions

1. Internal-only, with no public twin — this layer publishes PII.
Landowner and operator contact details (name, organization, role, phone,
email) alongside staff-written access notes that routinely contain gate codes.
ogc_water_well_field_operations does not exist and must never be created.

Every credential the internal mount accepts now reaches personal information,
including the static desktop-GIS keys, which are shared secrets revocable
only by redeploy.
Notices added to CLAUDE.md and
docs/internal-ogc-desktop-gis.md. docs/api-key-management.md needs the same
notice — that file lives on feat/api-key-management and does not exist on
this branch, so it could not be edited here.

2. The three permission columns are three-valued. true = a current grant
allows it, false = a current grant refuses it, NULL = no permission is on
record
, which is not a refusal — it means nobody has asked the landowner yet.
Collapsing NULL to false would tell a crew a well is off limits when the truth
is unknown. Any consumer-facing rendering has to carry the third state; the
field descriptions say so, since that prose is the only place a consumer reads
it.

3. History reads honour end_date. ogc_actively_monitored_wells takes
the greatest start_date and ignores end_date, so a status closed in 2019
still reads as current there. Tolerable on a summary layer, not on one whose
job is to say what is true today. The divergence is written into
docs/ogc_conventions.md so the two views don't read as one being a mistake.

Naming

Requested as amp_field_well_operations and renamed before merge, per the
pre-merge check in docs/ogc_conventions.md. Three rules pushed the same way:
amp is an unexpanded abbreviation whose expansion is recorded nowhere in this
codebase; AMP labels the layer's audience rather than filtering its rows (the
row set is every water well, no group predicate), so the name would claim a
scope the data doesn't have; and Group B analytic layers prefer a
water_well_ prefix. The AMP crews stay named in the collection description,
where an audience belongs.

Consistency with the existing catalogue

  • Depth to water uses (value - COALESCE(measuring_point_height, 0)) — the
    convention shared with ogc_water_well_summary,
    ogc_latest_depth_to_water_wells and ogc_well_water_column.
  • elevation_method reuses water_well_summary's provenance lookup verbatim.
  • Multi-valued columns are comma-joined text, not arrays. This layer gets
    pulled into ArcGIS Pro and QGIS and exported to File Geodatabase and
    GeoPackage for offline field use, and neither format has a list type.
  • Unlike water_well_summary, wells with no measurements still appear — a
    well nobody has measured is exactly the well a crew needs to find.

Also in this PR

  • Twelve supporting indexes. None of the foreign keys these lookups traverse
    was indexed; Postgres does not index foreign keys on its own.
  • The stats matview joins the CLI refresh list. The nightly pg_cron job
    discovers matviews from the catalog, so no schedule change.
  • One unrelated black reformat in tests/test_cli_commands.py, applied by
    the repo's own pre-commit hook.

Testing

Full suite green: 1224 passed, 84 skipped, 6 xpassed. black and flake8
clean. Migration downgrade/upgrade round trip verified, and
REFRESH MATERIALIZED VIEW CONCURRENTLY confirmed working against the stats
view's unique index.

18 new tests in tests/test_ogc_water_well_field_operations.py cover the
load-bearing rules: unmeasured wells still appear, permission NULL ≠ false,
an expired grant is not current, a barometer is not a datalogger, the
measuring-point convention, coordinates agree with the geometry, provenance on
a different field doesn't leak into elevation_method, and the layer 404s on
the public mount.

Open before this ships

  • docs/api-key-management.md needs the PII notice (see decision 1).
  • Access Status has no values. status_type carries it, but the
    status_value lexicon has no access-related terms — all eleven map to Well,
    Monitoring, Open, or Datalogger Suitability. The column publishes and reads
    null until the vocabulary gains terms scoped to it. Worth a look at
    production data.
  • continuous_reading_count aggregates the largest table in the schema. If the
    nightly refresh proves too slow at production scale, the fallback is to
    source the three continuous columns from transducer_daily_data.

🤖 Generated with Claude Code

jirhiker and others added 2 commits August 31, 2026 19:23
Adds water_well_field_operations, an OGC API - Features collection on
/ogcapi-internal only: one feature per water well carrying what a field crew
needs to plan and execute a visit. Every other well layer answers a scientific
question -- water level, chemistry, trend. This one answers an operational one:
may we go to this well, what are we allowed to do when we get there, and is it
overdue?

86 columns: identity and location (including latitude and longitude in decimal
degrees and how the elevation was obtained), construction detail, the current
measuring point, all five status types, the three standing landowner
permissions, monitoring frequency and group membership, manual water-level and
chemistry-sampling statistics, field-visit history, the installed logger and
its continuous record, landowner contacts, and access and directions notes.

Two relations rather than one (migration e1f2a3b4c5d6):

  ogc_internal_water_well_field_operations_stats  materialized, nightly refresh
      Aggregates over observation and transducer_observation.

  ogc_internal_water_well_field_operations        plain view, served by pygeoapi
      Live join of thing, location, status_history, permission_history,
      measuring_point_history, monitoring_frequency_history, deployment,
      sensor, contact and notes, LEFT JOINed to the stats matview.

The split is not tidiness. Staleness is dangerous on exactly the columns that
are cheap to read: a revoked sampling permission that still reads true until
the next nightly refresh sends a crew onto land they are no longer welcome on.
The current-record rule is also written against CURRENT_DATE, which inside a
materialized view would freeze at refresh time.

Three decisions worth review:

* Internal-only with no public twin. The layer publishes landowner and operator
  contact details -- name, organization, role, phone, email -- alongside
  staff-written access notes. ogc_water_well_field_operations does not exist
  and must never be created. Every credential the internal mount accepts now
  reaches personal information, the static desktop-GIS keys included; notices
  added to CLAUDE.md and docs/internal-ogc-desktop-gis.md.

* The three permission columns are three-valued. NULL means no permission is on
  record, which is not a refusal -- it means nobody has asked the landowner
  yet. Collapsing it to false would tell a crew a well is off limits when the
  truth is unknown.

* History reads honour end_date, unlike ogc_actively_monitored_wells, which
  takes the greatest start_date and lets a status closed in 2019 read as
  current. Written up in docs/ogc_conventions.md.

Requested as amp_field_well_operations and renamed before merge per the
pre-merge check in docs/ogc_conventions.md: amp is an unexpanded abbreviation
whose expansion is recorded nowhere in this codebase, AMP labels the layer's
audience rather than filtering its rows (the row set is every water well), and
Group B analytic layers prefer a water_well_ prefix. The AMP crews stay named
in the collection description.

Multi-valued columns are comma-joined text rather than arrays: this layer is
pulled into ArcGIS Pro and QGIS and exported to File Geodatabase and GeoPackage
for offline field use, and neither format has a list type.

Depth to water uses the (value - COALESCE(measuring_point_height, 0))
convention shared with ogc_water_well_summary, ogc_latest_depth_to_water_wells
and ogc_well_water_column, and elevation_method reuses that summary's
provenance lookup verbatim, so the layers cannot disagree.

Twelve supporting indexes come with the migration; none of the foreign keys
these lookups traverse was indexed.

Design rationale in docs/water-well-field-operations-layer.md, column-by-column
sources in docs/water-well-field-operations-columns.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Coverage

79.35% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
services/materialized_views.py 1 0 100%
TOTAL 1 0 100%

@ksmuczynski

Copy link
Copy Markdown
Contributor

@jirhiker I was working on this today (see ticket Jira ticket 1202) and was about to push a PR. I'll take a look at your work as well and compare.

@jirhiker
jirhiker marked this pull request as draft September 1, 2026 03:30
@jirhiker

jirhiker commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@ksmuczynski oops. sorry i didn't see you were already working on this. ill close this PR, but you can still compare to feat/amp-field-operations-ogc-layer

@jirhiker jirhiker closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants