Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,9 @@ jobs:
- name: Run tests
# --cov-fail-under is set here rather than in pyproject so that running a
# single test file locally does not fail on the whole-project total.
# --ignore=tests/transfers excludes the deprecated NM_Aquifer / NM_Wells
# transfer tests; those scripts are frozen and run by hand against SQL
# Server, so they must not gate a pull request. See transfers/README.md.
run: uv run pytest -vv --durations=20 --cov --cov-report=xml --cov-report=html --cov-report=term-missing --cov-fail-under="$COVERAGE_FAIL_UNDER" --junitxml=junit.xml --ignore=tests/transfers

- name: Write coverage summary
Expand Down
26 changes: 25 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,15 @@ POSTGRES_PASSWORD=<password>
```

### Data Migration
Both legacy transfer drivers are **deprecated** (see `transfers/README.md`); they
raise `DeprecationWarning` and take no new migrations, but stay runnable for
backfills.
```bash
# Transfer data from legacy AMPAPI (NM_Aquifer) to new schema
# NM_Aquifer (AMPAPI) -> new schema. Deprecated.
python -m transfers.transfer

# NM_Wells (geothermal) Phase-1 staging mirror. Deprecated.
python -m transfers.transfer_geothermal
```

## Architecture
Expand Down Expand Up @@ -285,6 +291,24 @@ GitHub Actions workflows (`.github/workflows/`):

## Legacy System Migration

**Deprecated.** Both legacy drivers are frozen -- `transfers/transfer.py`
(NM_Aquifer/AMPAPI) and `transfers/transfer_geothermal.py` (NM_Wells, with
`nmw_mirror_transfer.py`, `nmw_sql_dump.py`, `export_nmw_csvs.py`). Entry points
raise `DeprecationWarning`. Do not add new migrations to either. They remain
runnable because live API routes still read the `NMA_*` and `NMW_*` tables.
Read **`transfers/README.md`** before touching this layer.

Their tests live in `tests/transfers/` and **do not gate CI** --
`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`, and
`transfers/*` is omitted from coverage in `pyproject.toml`. Run them by hand:
`uv run pytest tests/transfers`. Tests for the `NMA_*`/`NMW_*` ORM models
(`db/nma_legacy.py`, `db/nmw_legacy.py`) stay in `tests/` proper and still gate
CI, since live routes depend on those models.

Still live, *not* deprecated: `services/scoped_transfer.py` and the
`oco scoped-transfer` command, which import the individual NM_Aquifer
transferers directly.

**Source**: AMPAPI (SQL Server, `NM_Aquifer` schema)
**Target**: OcotilloAPI (PostgreSQL + PostGIS)

Expand Down
2 changes: 1 addition & 1 deletion SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ T8|x|export_nmw_csvs.py pymssql export|I.export
T9|x|transfer_geothermal.py orchestrator|I.cli
T10|x|6 OGC collections in pygeoapi-config.yml|V6,I.ogc
T11|x|FK enforced via migration op.create_foreign_key; model index-only (resolved)|V2,V10
T12|x|add NMW_* mirror/loader/migration/OGC tests (tests/test_nmw_mirror.py, 19 tests); found+fixed CAST-unwrap bug B1|V1,V2,V3,V5,V6,V10,V11
T12|x|add NMW_* mirror/loader/migration/OGC tests (tests/transfers/test_nmw_mirror.py, 19 tests); found+fixed CAST-unwrap bug B1|V1,V2,V3,V5,V6,V10,V11
T13|.|verify alembic down path drops all views+tables (V3) on real db|V3
T14|.|run end-to-end load vs real dump, capture row counts per table|V2,V4
T15|.|finish PR #738 body (truncated at "- I ") + reviewer notes|-
Expand Down
5 changes: 5 additions & 0 deletions docs/nm_wells-migration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# NM_Wells → Ocotillo migration

> **Deprecated.** The Phase-1 loader described here is frozen and Phase 2 is not
> being pursued through this path; see [transfers/README.md](../transfers/README.md).
> Retained as the design record for the `NMW_*` mirror tables, which live API
> routes still read.

Migration of the legacy **NM_Wells** SQL Server database (and the related
Subsurface Library) into OcotilloAPI. Source of truth for table inventory and
field-level recommendations: `NM_Wells + Subsurface library.xlsx` (planning
Expand Down
11 changes: 9 additions & 2 deletions docs/nm_wells-transfer-runbook.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# NM_Wells 1:1 Mirror Transfer — Runbook

> **Deprecated.** The NM_Wells transfer path is frozen: no new migrations, no new
> features, and its tests no longer gate CI (they live in `tests/transfers/`,
> which `.github/workflows/tests.yml` ignores). The steps below stay accurate and
> the code stays runnable, because the `NMW_*` tables are still read by live API
> routes and so backfills and re-runs must remain possible. See
> [transfers/README.md](../transfers/README.md).

Operational steps to run the NM_Wells (geothermal) Phase-1 mirror transfer and verify it
worked. Phase 1 is a faithful, column-for-column copy of the legacy NM_Wells SQL Server
tables into the Postgres `NMW_*` staging mirror — no transform to the Ocotillo model.
Expand Down Expand Up @@ -225,8 +232,8 @@ psql "$DATABASE_URL" -c '\dv ogc_*' # expect 0
alembic upgrade head # recreate
```

Automated coverage for this lives in `tests/test_nmw_mirror.py` (19 tests):
`uv run pytest tests/test_nmw_mirror.py`.
Automated coverage for this lives in `tests/transfers/test_nmw_mirror.py` (19 tests):
`uv run pytest tests/transfers/test_nmw_mirror.py`.

---

Expand Down
2 changes: 1 addition & 1 deletion tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This directory contains automated tests (unit, integration, transfer, and API be

- `tests/unit/`: focused unit tests
- `tests/integration/`: cross-component tests
- `tests/transfers/`: transfer-focused tests
- `tests/transfers/`: transfer-focused tests for the deprecated `transfers/` scripts; excluded from CI (see `tests/transfers/README.md`)
- `tests/features/`: BDD-style feature tests

## Running tests
Expand Down
24 changes: 24 additions & 0 deletions tests/transfers/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Transfer tests

Tests for the legacy migration scripts in `transfers/` — the deprecated
NM_Aquifer (AMPAPI) and NM_Wells drivers.

## Excluded from CI

`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`, so
nothing in this directory gates a pull request. The transfer scripts they cover
are deprecated and run by hand against SQL Server; `transfers/*` is likewise
omitted from the coverage total in `pyproject.toml`.

Put new tests here only if they exercise `transfers/`. Tests for the
`NMA_*`/`NMW_*` ORM models (`db/nma_legacy.py`, `db/nmw_legacy.py`) belong in
`tests/` proper — those tables are still read by live API routes, so they stay
in CI.

## Running them

From the repo root, against the `ocotilloapi_test` database:

```bash
uv run pytest tests/transfers
```
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
from transfers.nmw_mirror_transfer import NMW_MIRROR_SPECS
from transfers.nmw_sql_dump import _parse_value, iter_table_rows

ROOT = os.path.dirname(os.path.dirname(__file__))
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

# DB relations created by the OGC-view migrations (d1e2f3a4b5c6, e2f3a4b5c6d7).
OGC_VIEWS = [
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
28 changes: 27 additions & 1 deletion transfers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,35 @@

This directory contains legacy-to-target ETL transfer logic.

## Status: deprecated

Both legacy migration drivers are frozen. Do not add new migrations to either:

- `transfers/transfer.py` -- the NM_Aquifer (AMPAPI, SQL Server) driver.
- `transfers/transfer_geothermal.py` -- the NM_Wells (geothermal) driver, plus
its `nmw_mirror_transfer.py`, `nmw_sql_dump.py`, and `export_nmw_csvs.py`
supporting modules.

Their top-level entry points raise `DeprecationWarning`. They are kept runnable
because the tables they populate (`NMA_*`, `NMW_*`) are still read by live API
routes, so backfills and re-runs must remain possible -- but they receive no new
features.

Consequently their tests live in `tests/transfers/` and do **not** gate CI
(`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`), and
`transfers/*` is omitted from the coverage total in `pyproject.toml`. Run them by
hand with `uv run pytest tests/transfers`.

Still live and *not* deprecated:

- `services/scoped_transfer.py` and the `oco scoped-transfer` command, which
import the individual NM_Aquifer transferers directly.
- `transfers/seed_geothermal.py`, a dev/test seeder that generates fake data
rather than reading a legacy source.

## Main orchestration

- `transfers/transfer.py`
- `transfers/transfer.py` (deprecated)

## Important supporting modules

Expand Down
13 changes: 12 additions & 1 deletion transfers/export_nmw_csvs.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
"""Export NM_Wells SQL Server tables to CSV files for the transfer pipeline.
"""DEPRECATED: export NM_Wells SQL Server tables to CSV for the transfer pipeline.

Part of the frozen NM_Wells migration path; see the deprecation note in
``transfers/transfer_geothermal.py``. Kept runnable for re-exports, but it gets
no new features and its tests no longer gate CI.

Connects to the NM_Wells SQL Server database and exports each source table to
transfers/data/nma_csv_cache/<table>.csv, which is where nmw_mirror_transfer.py
Expand All @@ -16,6 +20,7 @@
"""

import os
import warnings
from pathlib import Path

import pymssql
Expand Down Expand Up @@ -64,6 +69,12 @@ def export_table(cursor, table: str, out_path: Path) -> int:


def main():
warnings.warn(
"transfers.export_nmw_csvs is deprecated; the NM_Wells migration path "
"is frozen and receives no new migrations.",
DeprecationWarning,
stacklevel=2,
)
OUT_DIR.mkdir(parents=True, exist_ok=True)
print(
f"Connecting to {os.environ.get('NMW_HOST')} / {os.environ.get('NMW_DATABASE', 'NM_Wells')}"
Expand Down
14 changes: 13 additions & 1 deletion transfers/nmw_mirror_transfer.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ===============================================================================
"""Load the NM_Wells SQL dump into the ``NMW_*`` 1:1 staging mirror tables.
"""DEPRECATED: load the NM_Wells SQL dump into the ``NMW_*`` staging mirror.

Part of the frozen NM_Wells migration path; see the deprecation note in
``transfers/transfer_geothermal.py``. The ``NMW_*`` tables it populates are
still read by live API routes, so this loader stays runnable for backfills and
re-runs, but it gets no new features and its tests no longer gate CI.

Phase 1 of the NM_Wells migration (see db/nmw_legacy.py and
docs/nm_wells-migration.md). This is a faithful copy: each source table's CSV
Expand Down Expand Up @@ -47,6 +52,7 @@
import os
import tempfile
import uuid
import warnings
from dataclasses import dataclass

import pandas as pd
Expand Down Expand Up @@ -317,6 +323,12 @@ def transfer_nmw_mirror(session: Session, limit: int = None) -> tuple:
``(session, limit)`` signature as the other session-based transfers. Returns
``(num_tables_loaded, total_rows_inserted, errors)``.
"""
warnings.warn(
"transfers.nmw_mirror_transfer is deprecated; the NM_Wells migration "
"path is frozen and receives no new migrations.",
DeprecationWarning,
stacklevel=2,
)
limit = int(limit or 0)
dump = os.getenv(_SQL_DUMP_ENV)
out_dir = None
Expand Down
6 changes: 5 additions & 1 deletion transfers/nmw_sql_dump.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ===============================================================================
"""Parse a SQL Server data-dump ``.sql`` file into per-table CSVs.
"""DEPRECATED: parse a SQL Server data-dump ``.sql`` file into per-table CSVs.

Part of the frozen NM_Wells migration path; see the deprecation note in
``transfers/transfer_geothermal.py``. Kept runnable for re-runs of the NM_Wells
dump load, but it gets no new features and its tests no longer gate CI.

``INSERT [dbo].[<table>] (<cols>) VALUES (<vals>)[, (<vals>) ...]`` statements
(SSMS "Generate Scripts -> data" / bcp INSERT mode) are split with ``sqlparse``
Expand Down
18 changes: 15 additions & 3 deletions transfers/transfer_geothermal.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ===============================================================================
"""Standalone orchestrator for the NM_Wells (geothermal) migration.
"""DEPRECATED: standalone orchestrator for the NM_Wells (geothermal) migration.

Separate from the deprecated ``transfers/transfer.py`` (NM_Aquifer driver). This
script runs the NM_Wells Phase-1 staging migration:
Deprecated alongside ``transfers/transfer.py`` (the NM_Aquifer driver): both
legacy migration drivers are frozen. Do not add new migrations here. The
``NMW_*`` staging tables this loads remain in service -- live API routes still
read them -- so the loader is kept runnable for backfills and re-runs, but it
gets no new features and its tests no longer gate CI.

This script runs the NM_Wells Phase-1 staging migration:

1. Reference -> lexicon load (``ref_*`` lookups), gated by
``TRANSFER_GEOTHERMAL_REFERENCE`` (default True).
Expand All @@ -37,6 +42,7 @@
"""

import os
import warnings

from dotenv import load_dotenv

Expand All @@ -63,6 +69,12 @@

def run_geothermal_transfer(limit: int = None) -> dict:
"""Run the NM_Wells geothermal staging migration. Returns a summary dict."""
warnings.warn(
"transfers.transfer_geothermal is deprecated; the NM_Wells migration "
"drivers are frozen and receive no new migrations.",
DeprecationWarning,
stacklevel=2,
)
limit = int(limit if limit is not None else os.getenv("TRANSFER_LIMIT", 0) or 0)
summary: dict = {}

Expand Down
Loading