diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index fb68c00e7..632b9feac 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -100,6 +100,9 @@ jobs: - name: Run tests # --cov-fail-under is set here rather than in pyproject so that running a # single test file locally does not fail on the whole-project total. + # --ignore=tests/transfers excludes the deprecated NM_Aquifer / NM_Wells + # transfer tests; those scripts are frozen and run by hand against SQL + # Server, so they must not gate a pull request. See transfers/README.md. run: uv run pytest -vv --durations=20 --cov --cov-report=xml --cov-report=html --cov-report=term-missing --cov-fail-under="$COVERAGE_FAIL_UNDER" --junitxml=junit.xml --ignore=tests/transfers - name: Write coverage summary diff --git a/CLAUDE.md b/CLAUDE.md index 30549235f..147bab3d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -77,9 +77,15 @@ POSTGRES_PASSWORD= ``` ### Data Migration +Both legacy transfer drivers are **deprecated** (see `transfers/README.md`); they +raise `DeprecationWarning` and take no new migrations, but stay runnable for +backfills. ```bash -# Transfer data from legacy AMPAPI (NM_Aquifer) to new schema +# NM_Aquifer (AMPAPI) -> new schema. Deprecated. python -m transfers.transfer + +# NM_Wells (geothermal) Phase-1 staging mirror. Deprecated. +python -m transfers.transfer_geothermal ``` ## Architecture @@ -285,6 +291,24 @@ GitHub Actions workflows (`.github/workflows/`): ## Legacy System Migration +**Deprecated.** Both legacy drivers are frozen -- `transfers/transfer.py` +(NM_Aquifer/AMPAPI) and `transfers/transfer_geothermal.py` (NM_Wells, with +`nmw_mirror_transfer.py`, `nmw_sql_dump.py`, `export_nmw_csvs.py`). Entry points +raise `DeprecationWarning`. Do not add new migrations to either. They remain +runnable because live API routes still read the `NMA_*` and `NMW_*` tables. +Read **`transfers/README.md`** before touching this layer. + +Their tests live in `tests/transfers/` and **do not gate CI** -- +`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`, and +`transfers/*` is omitted from coverage in `pyproject.toml`. Run them by hand: +`uv run pytest tests/transfers`. Tests for the `NMA_*`/`NMW_*` ORM models +(`db/nma_legacy.py`, `db/nmw_legacy.py`) stay in `tests/` proper and still gate +CI, since live routes depend on those models. + +Still live, *not* deprecated: `services/scoped_transfer.py` and the +`oco scoped-transfer` command, which import the individual NM_Aquifer +transferers directly. + **Source**: AMPAPI (SQL Server, `NM_Aquifer` schema) **Target**: OcotilloAPI (PostgreSQL + PostGIS) diff --git a/SPEC.md b/SPEC.md index 3731fa02e..727c3ade7 100644 --- a/SPEC.md +++ b/SPEC.md @@ -61,7 +61,7 @@ T8|x|export_nmw_csvs.py pymssql export|I.export T9|x|transfer_geothermal.py orchestrator|I.cli T10|x|6 OGC collections in pygeoapi-config.yml|V6,I.ogc T11|x|FK enforced via migration op.create_foreign_key; model index-only (resolved)|V2,V10 -T12|x|add NMW_* mirror/loader/migration/OGC tests (tests/test_nmw_mirror.py, 19 tests); found+fixed CAST-unwrap bug B1|V1,V2,V3,V5,V6,V10,V11 +T12|x|add NMW_* mirror/loader/migration/OGC tests (tests/transfers/test_nmw_mirror.py, 19 tests); found+fixed CAST-unwrap bug B1|V1,V2,V3,V5,V6,V10,V11 T13|.|verify alembic down path drops all views+tables (V3) on real db|V3 T14|.|run end-to-end load vs real dump, capture row counts per table|V2,V4 T15|.|finish PR #738 body (truncated at "- I ") + reviewer notes|- diff --git a/docs/nm_wells-migration.md b/docs/nm_wells-migration.md index 35f335b5d..16cd97c6b 100644 --- a/docs/nm_wells-migration.md +++ b/docs/nm_wells-migration.md @@ -1,5 +1,10 @@ # NM_Wells → Ocotillo migration +> **Deprecated.** The Phase-1 loader described here is frozen and Phase 2 is not +> being pursued through this path; see [transfers/README.md](../transfers/README.md). +> Retained as the design record for the `NMW_*` mirror tables, which live API +> routes still read. + Migration of the legacy **NM_Wells** SQL Server database (and the related Subsurface Library) into OcotilloAPI. Source of truth for table inventory and field-level recommendations: `NM_Wells + Subsurface library.xlsx` (planning diff --git a/docs/nm_wells-transfer-runbook.md b/docs/nm_wells-transfer-runbook.md index 60731a8b5..9a78fea33 100644 --- a/docs/nm_wells-transfer-runbook.md +++ b/docs/nm_wells-transfer-runbook.md @@ -1,5 +1,12 @@ # NM_Wells 1:1 Mirror Transfer — Runbook +> **Deprecated.** The NM_Wells transfer path is frozen: no new migrations, no new +> features, and its tests no longer gate CI (they live in `tests/transfers/`, +> which `.github/workflows/tests.yml` ignores). The steps below stay accurate and +> the code stays runnable, because the `NMW_*` tables are still read by live API +> routes and so backfills and re-runs must remain possible. See +> [transfers/README.md](../transfers/README.md). + Operational steps to run the NM_Wells (geothermal) Phase-1 mirror transfer and verify it worked. Phase 1 is a faithful, column-for-column copy of the legacy NM_Wells SQL Server tables into the Postgres `NMW_*` staging mirror — no transform to the Ocotillo model. @@ -225,8 +232,8 @@ psql "$DATABASE_URL" -c '\dv ogc_*' # expect 0 alembic upgrade head # recreate ``` -Automated coverage for this lives in `tests/test_nmw_mirror.py` (19 tests): -`uv run pytest tests/test_nmw_mirror.py`. +Automated coverage for this lives in `tests/transfers/test_nmw_mirror.py` (19 tests): +`uv run pytest tests/transfers/test_nmw_mirror.py`. --- diff --git a/tests/README.md b/tests/README.md index 2593c5930..402c4c6de 100644 --- a/tests/README.md +++ b/tests/README.md @@ -6,7 +6,7 @@ This directory contains automated tests (unit, integration, transfer, and API be - `tests/unit/`: focused unit tests - `tests/integration/`: cross-component tests -- `tests/transfers/`: transfer-focused tests +- `tests/transfers/`: transfer-focused tests for the deprecated `transfers/` scripts; excluded from CI (see `tests/transfers/README.md`) - `tests/features/`: BDD-style feature tests ## Running tests diff --git a/tests/transfers/README.md b/tests/transfers/README.md new file mode 100644 index 000000000..e93578e83 --- /dev/null +++ b/tests/transfers/README.md @@ -0,0 +1,24 @@ +# Transfer tests + +Tests for the legacy migration scripts in `transfers/` — the deprecated +NM_Aquifer (AMPAPI) and NM_Wells drivers. + +## Excluded from CI + +`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`, so +nothing in this directory gates a pull request. The transfer scripts they cover +are deprecated and run by hand against SQL Server; `transfers/*` is likewise +omitted from the coverage total in `pyproject.toml`. + +Put new tests here only if they exercise `transfers/`. Tests for the +`NMA_*`/`NMW_*` ORM models (`db/nma_legacy.py`, `db/nmw_legacy.py`) belong in +`tests/` proper — those tables are still read by live API routes, so they stay +in CI. + +## Running them + +From the repo root, against the `ocotilloapi_test` database: + +```bash +uv run pytest tests/transfers +``` diff --git a/tests/unit/test_contact_transfer_email_utils.py b/tests/transfers/test_contact_transfer_email_utils.py similarity index 100% rename from tests/unit/test_contact_transfer_email_utils.py rename to tests/transfers/test_contact_transfer_email_utils.py diff --git a/tests/test_minor_trace_chemistry_transfer.py b/tests/transfers/test_minor_trace_chemistry_transfer.py similarity index 100% rename from tests/test_minor_trace_chemistry_transfer.py rename to tests/transfers/test_minor_trace_chemistry_transfer.py diff --git a/tests/test_nmw_mirror.py b/tests/transfers/test_nmw_mirror.py similarity index 99% rename from tests/test_nmw_mirror.py rename to tests/transfers/test_nmw_mirror.py index 4de183fe4..30552f734 100644 --- a/tests/test_nmw_mirror.py +++ b/tests/transfers/test_nmw_mirror.py @@ -38,7 +38,7 @@ from transfers.nmw_mirror_transfer import NMW_MIRROR_SPECS from transfers.nmw_sql_dump import _parse_value, iter_table_rows -ROOT = os.path.dirname(os.path.dirname(__file__)) +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # DB relations created by the OGC-view migrations (d1e2f3a4b5c6, e2f3a4b5c6d7). OGC_VIEWS = [ diff --git a/tests/test_sensor_transfer.py b/tests/transfers/test_sensor_transfer.py similarity index 100% rename from tests/test_sensor_transfer.py rename to tests/transfers/test_sensor_transfer.py diff --git a/tests/test_thing_transfer.py b/tests/transfers/test_thing_transfer.py similarity index 100% rename from tests/test_thing_transfer.py rename to tests/transfers/test_thing_transfer.py diff --git a/tests/test_transfer_legacy_dates.py b/tests/transfers/test_transfer_legacy_dates.py similarity index 100% rename from tests/test_transfer_legacy_dates.py rename to tests/transfers/test_transfer_legacy_dates.py diff --git a/tests/test_well_transfer.py b/tests/transfers/test_well_transfer.py similarity index 100% rename from tests/test_well_transfer.py rename to tests/transfers/test_well_transfer.py diff --git a/transfers/README.md b/transfers/README.md index 08e032349..2bac5b0b1 100644 --- a/transfers/README.md +++ b/transfers/README.md @@ -2,9 +2,35 @@ This directory contains legacy-to-target ETL transfer logic. +## Status: deprecated + +Both legacy migration drivers are frozen. Do not add new migrations to either: + +- `transfers/transfer.py` -- the NM_Aquifer (AMPAPI, SQL Server) driver. +- `transfers/transfer_geothermal.py` -- the NM_Wells (geothermal) driver, plus + its `nmw_mirror_transfer.py`, `nmw_sql_dump.py`, and `export_nmw_csvs.py` + supporting modules. + +Their top-level entry points raise `DeprecationWarning`. They are kept runnable +because the tables they populate (`NMA_*`, `NMW_*`) are still read by live API +routes, so backfills and re-runs must remain possible -- but they receive no new +features. + +Consequently their tests live in `tests/transfers/` and do **not** gate CI +(`.github/workflows/tests.yml` runs pytest with `--ignore=tests/transfers`), and +`transfers/*` is omitted from the coverage total in `pyproject.toml`. Run them by +hand with `uv run pytest tests/transfers`. + +Still live and *not* deprecated: + +- `services/scoped_transfer.py` and the `oco scoped-transfer` command, which + import the individual NM_Aquifer transferers directly. +- `transfers/seed_geothermal.py`, a dev/test seeder that generates fake data + rather than reading a legacy source. + ## Main orchestration -- `transfers/transfer.py` +- `transfers/transfer.py` (deprecated) ## Important supporting modules diff --git a/transfers/export_nmw_csvs.py b/transfers/export_nmw_csvs.py index 51daf403e..6ee6cc106 100644 --- a/transfers/export_nmw_csvs.py +++ b/transfers/export_nmw_csvs.py @@ -1,4 +1,8 @@ -"""Export NM_Wells SQL Server tables to CSV files for the transfer pipeline. +"""DEPRECATED: export NM_Wells SQL Server tables to CSV for the transfer pipeline. + +Part of the frozen NM_Wells migration path; see the deprecation note in +``transfers/transfer_geothermal.py``. Kept runnable for re-exports, but it gets +no new features and its tests no longer gate CI. Connects to the NM_Wells SQL Server database and exports each source table to transfers/data/nma_csv_cache/.csv, which is where nmw_mirror_transfer.py @@ -16,6 +20,7 @@ """ import os +import warnings from pathlib import Path import pymssql @@ -64,6 +69,12 @@ def export_table(cursor, table: str, out_path: Path) -> int: def main(): + warnings.warn( + "transfers.export_nmw_csvs is deprecated; the NM_Wells migration path " + "is frozen and receives no new migrations.", + DeprecationWarning, + stacklevel=2, + ) OUT_DIR.mkdir(parents=True, exist_ok=True) print( f"Connecting to {os.environ.get('NMW_HOST')} / {os.environ.get('NMW_DATABASE', 'NM_Wells')}" diff --git a/transfers/nmw_mirror_transfer.py b/transfers/nmw_mirror_transfer.py index d59ef4eaf..1d42fef6c 100644 --- a/transfers/nmw_mirror_transfer.py +++ b/transfers/nmw_mirror_transfer.py @@ -13,7 +13,12 @@ # See the License for the specific language governing permissions and # limitations under the License. # =============================================================================== -"""Load the NM_Wells SQL dump into the ``NMW_*`` 1:1 staging mirror tables. +"""DEPRECATED: load the NM_Wells SQL dump into the ``NMW_*`` staging mirror. + +Part of the frozen NM_Wells migration path; see the deprecation note in +``transfers/transfer_geothermal.py``. The ``NMW_*`` tables it populates are +still read by live API routes, so this loader stays runnable for backfills and +re-runs, but it gets no new features and its tests no longer gate CI. Phase 1 of the NM_Wells migration (see db/nmw_legacy.py and docs/nm_wells-migration.md). This is a faithful copy: each source table's CSV @@ -47,6 +52,7 @@ import os import tempfile import uuid +import warnings from dataclasses import dataclass import pandas as pd @@ -317,6 +323,12 @@ def transfer_nmw_mirror(session: Session, limit: int = None) -> tuple: ``(session, limit)`` signature as the other session-based transfers. Returns ``(num_tables_loaded, total_rows_inserted, errors)``. """ + warnings.warn( + "transfers.nmw_mirror_transfer is deprecated; the NM_Wells migration " + "path is frozen and receives no new migrations.", + DeprecationWarning, + stacklevel=2, + ) limit = int(limit or 0) dump = os.getenv(_SQL_DUMP_ENV) out_dir = None diff --git a/transfers/nmw_sql_dump.py b/transfers/nmw_sql_dump.py index f7010b849..e0b758539 100644 --- a/transfers/nmw_sql_dump.py +++ b/transfers/nmw_sql_dump.py @@ -13,7 +13,11 @@ # See the License for the specific language governing permissions and # limitations under the License. # =============================================================================== -"""Parse a SQL Server data-dump ``.sql`` file into per-table CSVs. +"""DEPRECATED: parse a SQL Server data-dump ``.sql`` file into per-table CSVs. + +Part of the frozen NM_Wells migration path; see the deprecation note in +``transfers/transfer_geothermal.py``. Kept runnable for re-runs of the NM_Wells +dump load, but it gets no new features and its tests no longer gate CI. ``INSERT [dbo].[
] () VALUES ()[, () ...]`` statements (SSMS "Generate Scripts -> data" / bcp INSERT mode) are split with ``sqlparse`` diff --git a/transfers/transfer_geothermal.py b/transfers/transfer_geothermal.py index 6945ea01d..5071d32aa 100644 --- a/transfers/transfer_geothermal.py +++ b/transfers/transfer_geothermal.py @@ -13,10 +13,15 @@ # See the License for the specific language governing permissions and # limitations under the License. # =============================================================================== -"""Standalone orchestrator for the NM_Wells (geothermal) migration. +"""DEPRECATED: standalone orchestrator for the NM_Wells (geothermal) migration. -Separate from the deprecated ``transfers/transfer.py`` (NM_Aquifer driver). This -script runs the NM_Wells Phase-1 staging migration: +Deprecated alongside ``transfers/transfer.py`` (the NM_Aquifer driver): both +legacy migration drivers are frozen. Do not add new migrations here. The +``NMW_*`` staging tables this loads remain in service -- live API routes still +read them -- so the loader is kept runnable for backfills and re-runs, but it +gets no new features and its tests no longer gate CI. + +This script runs the NM_Wells Phase-1 staging migration: 1. Reference -> lexicon load (``ref_*`` lookups), gated by ``TRANSFER_GEOTHERMAL_REFERENCE`` (default True). @@ -37,6 +42,7 @@ """ import os +import warnings from dotenv import load_dotenv @@ -63,6 +69,12 @@ def run_geothermal_transfer(limit: int = None) -> dict: """Run the NM_Wells geothermal staging migration. Returns a summary dict.""" + warnings.warn( + "transfers.transfer_geothermal is deprecated; the NM_Wells migration " + "drivers are frozen and receive no new migrations.", + DeprecationWarning, + stacklevel=2, + ) limit = int(limit if limit is not None else os.getenv("TRANSFER_LIMIT", 0) or 0) summary: dict = {}