chore(ingestion): report duplicate observations without psql

[jirhiker]

The SQL version of the pre-migration duplicate check needs a psql client and a
database password. A machine with only gcloud has neither, which is how this
check stalled when it was needed.

This does the same counting through the Cloud SQL connector with IAM
authentication: the credential is the operator's own gcloud login, and nothing
is stored.

uv run --group ingestion python -m \
  automated_ingestion.scripts.report_duplicate_observations \
  --instance waterdatainitiative-271000:us-west4:dataservices \
  --database ocotillo-staging

It reports, it does not resolve

Groups whose value disagrees are conflicting measurements rather than
redundant rows. Collapsing them discards a reading somebody recorded, so which
copy survives is a judgement about the data and not something a script should
make.

The failure path is most of the value

A principal without a database user gets a Postgres error that reads exactly
like a wrong password:

FATAL: password authentication failed for user "..."

so the script names the gcloud sql users create command that fixes it. That
mattered in practice — being a project owner does not imply a database login,
and the error does not say so.

Note that a user still needs USAGE on public and SELECT on
transducer_observation; creating the IAM user alone is not enough, and that
grant needs someone who already has privileges.

Context

Written while checking #847's migration was safe to run. The check has since
come back clean for staging and production, and #847 is merged — this stays
because the next constraint migration will want it, and because it documents how
to query these databases from a machine with only gcloud.

🤖 Generated with Claude Code

[github-actions]

Coverage

✅ 79.45% total — gate is 75%.

Coverage for the Python files changed in this PR

Name	Stmts	Miss	Cover	Missing
automated_ingestion/scripts/report_duplicate_observations.py	51	51	0%	40-150
TOTAL	51	51	0%

[github-actions]

Your pull request is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
ocotillo-automated-ingestion		View in Cloud	Aug 19, 2026 at 06:24 AM (UTC)