Skip to content

Commit a0fca89

Browse files
jirhikerclaude
jirhiker
and
claude
committed
feat(cm): mirror the critical-minerals workbook into CM_legacy
Phase-1 staging mirror of the Earth MRI critical-minerals chemistry workbook (McLemoreMasterChem, NMBGMR) into CM_* tables, plus a reconciliation report for the source-data owner. No transform into the Ocotillo data model yet. Seven mirror tables. The ChemicalData, GIS and QAQC sheets share a column set, so they land in one CM_ChemicalData behind a source_sheet discriminator: the first two are byte-identical in header text and order, and QAQC is those columns minus MapSymbol/Pd/Pt with latitude/longitude capitalized. Rows are keyed on (source_sheet, source_row) rather than SAMPLE, which repeats across 258 names. GIS is a stale hand-maintained fork of ChemicalData, not a location-enriched copy: it carries the same mixed datums and the same ~876 rows with no latitude, and 1704 of the 4848 shared rows disagree in both directions (GIS holds 533 Chem Lab File No., 485 Laboratory and 170 FeO/Fe2O3 values ChemicalData lacks; ChemicalData holds 633 Total, 184 Area and 18 appended Pearce (2020) samples GIS lacks). Neither sheet is authoritative, so both are mirrored in full and reconciliation is deferred to a per-column ruling by V.T. McLemore. The loader warns whenever the two row counts diverge. Every column is a String. The workbook stores censored analyte values as text (1154 '<' values in Au alone), carries '#VALUE!' errors, and mixes real dates with year-only text; parsing belongs to the transform. Column names are derived mechanically because sheet headers are not SQL identifiers, with analytes carrying the unit the workbook declares for them - which also keeps As and In off the Python and SQL keyword lists. Loading is idempotent per sheet and asserts the layout instead of guessing it: a moved header row or an unmapped column aborts the load rather than silently dropping cells. scripts/cm_reconciliation_report.py builds the decision workbook - the sheet-to-sheet drift plus 4276 integrity findings across 13 issue types, among them 724 values impossible for their declared unit (F = 27700 in a % column), 259 Au detection limits that are ppm in a ppb column, 300 non-numeric analyte tokens, and 64 of 84 analytes reported against more than three distinct detection limits. Verified by loading the delivered workbook: 10,100 rows, and 708 cells across 6 random rows compared against openpyxl with zero mismatches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent e425fa1 commit a0fca89

10 files changed

Lines changed: 3423 additions & 0 deletions

‎CLAUDE.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,23 @@ Still live, *not* deprecated: `services/scoped_transfer.py` and the
315315
`oco scoped-transfer` command, which import the individual NM_Aquifer
316316
transferers directly.
317317

318+
### Critical minerals mirror (`CM_legacy`)
319+
320+
`db/cm_legacy.py` + `services/cm_legacy_mirror.py` mirror the McLemore Earth MRI
321+
critical-minerals chemistry workbook into `CM_*` tables
322+
(`oco load-critical-minerals-workbook --file ...`). Phase 1 staging only; no
323+
transform into the Ocotillo model yet. Every column is a `String` because the
324+
workbook stores censored analyte values as text (`<0.1`) and carries `#VALUE!`
325+
errors.
326+
327+
The workbook's `ChemicalData`, `GIS` and `QAQC` sheets share a column set and
328+
mirror into one table behind a `source_sheet` discriminator. **`GIS` is a stale
329+
fork of `ChemicalData`, not a location-enriched copy** — each sheet holds values
330+
the other lacks, so reading one `source_sheet` alone silently drops data.
331+
Reconciliation is deliberately deferred. Read
332+
**`docs/critical-minerals-legacy-mirror.md`** before querying or extending this
333+
layer.
334+
318335
**Source**: AMPAPI (SQL Server, `NM_Aquifer` schema)
319336
**Target**: OcotilloAPI (PostgreSQL + PostGIS)
320337

‎alembic/versions/d4e5f6a7b8c9_cm_legacy_mirror_tables.py‎

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,296 @@
1+
"""CM_legacy staging mirror tables
2+
3+
Revision ID: d4e5f6a7b8c9
4+
Revises: c3d4e5f6a7b8
5+
Create Date: 2026-08-20
6+
7+
1:1 staging mirror of the McLemore critical-minerals chemistry workbook
8+
(Earth MRI, NMBGMR; see db/cm_legacy.py and
9+
docs/critical-minerals-legacy-mirror.md). Faithful copies of the workbook
10+
sheets; the transform into the Ocotillo data model is a later phase.
11+
12+
ChemicalData / GIS / QAQC -> CM_ChemicalData (source_sheet discriminator)
13+
DetectionLimits -> CM_DetectionLimits
14+
References -> CM_References
15+
MineralSystems -> CM_MineralSystems
16+
world -> CM_WorldComparisons
17+
world_ref -> CM_WorldReferences
18+
General Information / MetaData / DefinitionOfFields
19+
-> CM_WorkbookMetadata
20+
21+
Every data column is a String: the workbook stores censored analyte values as
22+
text ("<0.1"), carries Excel error text ("#VALUE!"), and mixes dates with free
23+
text. Parsing belongs to the transform.
24+
25+
The ChemicalData, GIS and QAQC sheets disagree with each other and none is
26+
authoritative, so all three are mirrored and reconciliation is deferred. See
27+
the module docstring in db/cm_legacy.py.
28+
"""
29+
30+
from typing import Sequence, Union
31+
32+
from alembic import op
33+
import sqlalchemy as sa
34+
35+
revision: str = "d4e5f6a7b8c9"
36+
down_revision: Union[str, Sequence[str], None] = "c3d4e5f6a7b8"
37+
branch_labels: Union[str, Sequence[str], None] = None
38+
depends_on: Union[str, Sequence[str], None] = None
39+
40+
41+
def upgrade() -> None:
42+
op.create_table(
43+
"CM_ChemicalData",
44+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
45+
sa.Column("source_sheet", sa.String(), nullable=False),
46+
sa.Column("source_row", sa.Integer(), nullable=False),
47+
sa.Column("sample", sa.String(), nullable=True),
48+
sa.Column("project", sa.String(), nullable=True),
49+
sa.Column("area", sa.String(), nullable=True),
50+
sa.Column("reference", sa.String(), nullable=True),
51+
sa.Column("date_collected", sa.String(), nullable=True),
52+
sa.Column("date_analyzed", sa.String(), nullable=True),
53+
sa.Column("chem_lab_file_no", sa.String(), nullable=True),
54+
sa.Column("laboratory", sa.String(), nullable=True),
55+
sa.Column("latitude", sa.String(), nullable=True),
56+
sa.Column("longitude", sa.String(), nullable=True),
57+
sa.Column("coordinate_system", sa.String(), nullable=True),
58+
sa.Column("county", sa.String(), nullable=True),
59+
sa.Column("state", sa.String(), nullable=True),
60+
sa.Column("lithology", sa.String(), nullable=True),
61+
sa.Column("mineral_system", sa.String(), nullable=True),
62+
sa.Column("deposit_types", sa.String(), nullable=True),
63+
sa.Column("map_symbol", sa.String(), nullable=True),
64+
sa.Column("method_collected", sa.String(), nullable=True),
65+
sa.Column("sample_source", sa.String(), nullable=True),
66+
sa.Column("mineralogy_deposit_type", sa.String(), nullable=True),
67+
sa.Column("depth_legnth_ft", sa.String(), nullable=True),
68+
sa.Column("mine_id", sa.String(), nullable=True),
69+
sa.Column("location_notes", sa.String(), nullable=True),
70+
sa.Column("comments", sa.String(), nullable=True),
71+
sa.Column("paste_ph", sa.String(), nullable=True),
72+
sa.Column("paste_conductivity", sa.String(), nullable=True),
73+
sa.Column("tds_mg_l", sa.String(), nullable=True),
74+
sa.Column("sio2_pct", sa.String(), nullable=True),
75+
sa.Column("tio2_pct", sa.String(), nullable=True),
76+
sa.Column("al2o3_pct", sa.String(), nullable=True),
77+
sa.Column("fe2o3t_pct", sa.String(), nullable=True),
78+
sa.Column("mno_pct", sa.String(), nullable=True),
79+
sa.Column("mgo_pct", sa.String(), nullable=True),
80+
sa.Column("cao_pct", sa.String(), nullable=True),
81+
sa.Column("na2o_pct", sa.String(), nullable=True),
82+
sa.Column("k2o_pct", sa.String(), nullable=True),
83+
sa.Column("p2o5_pct", sa.String(), nullable=True),
84+
sa.Column("loi_pct", sa.String(), nullable=True),
85+
sa.Column("f_pct", sa.String(), nullable=True),
86+
sa.Column("s_pct", sa.String(), nullable=True),
87+
sa.Column("so3_pct", sa.String(), nullable=True),
88+
sa.Column("so4_pct", sa.String(), nullable=True),
89+
sa.Column("c_pct", sa.String(), nullable=True),
90+
sa.Column("co2_pct", sa.String(), nullable=True),
91+
sa.Column("total_pct", sa.String(), nullable=True),
92+
sa.Column("feo_pct", sa.String(), nullable=True),
93+
sa.Column("fe2o3_pct", sa.String(), nullable=True),
94+
sa.Column("feo_star_pct", sa.String(), nullable=True),
95+
sa.Column("h2o_plus_pct", sa.String(), nullable=True),
96+
sa.Column("h2o_minus_pct", sa.String(), nullable=True),
97+
sa.Column("au_ppb", sa.String(), nullable=True),
98+
sa.Column("ag_ppm", sa.String(), nullable=True),
99+
sa.Column("as_ppm", sa.String(), nullable=True),
100+
sa.Column("b_ppm", sa.String(), nullable=True),
101+
sa.Column("ba_ppm", sa.String(), nullable=True),
102+
sa.Column("be_ppm", sa.String(), nullable=True),
103+
sa.Column("bi_ppm", sa.String(), nullable=True),
104+
sa.Column("br_ppm", sa.String(), nullable=True),
105+
sa.Column("cd_ppm", sa.String(), nullable=True),
106+
sa.Column("cl_ppm", sa.String(), nullable=True),
107+
sa.Column("co_ppm", sa.String(), nullable=True),
108+
sa.Column("cr_ppm", sa.String(), nullable=True),
109+
sa.Column("cs_ppm", sa.String(), nullable=True),
110+
sa.Column("cu_ppm", sa.String(), nullable=True),
111+
sa.Column("ga_ppm", sa.String(), nullable=True),
112+
sa.Column("ge_ppm", sa.String(), nullable=True),
113+
sa.Column("hf_ppm", sa.String(), nullable=True),
114+
sa.Column("hg_ppm", sa.String(), nullable=True),
115+
sa.Column("in_ppm", sa.String(), nullable=True),
116+
sa.Column("li_ppm", sa.String(), nullable=True),
117+
sa.Column("mo_ppm", sa.String(), nullable=True),
118+
sa.Column("nb_ppm", sa.String(), nullable=True),
119+
sa.Column("ni_ppm", sa.String(), nullable=True),
120+
sa.Column("pd_ppm", sa.String(), nullable=True),
121+
sa.Column("pb_ppm", sa.String(), nullable=True),
122+
sa.Column("pt_ppm", sa.String(), nullable=True),
123+
sa.Column("rb_ppm", sa.String(), nullable=True),
124+
sa.Column("re_ppm", sa.String(), nullable=True),
125+
sa.Column("sb_ppm", sa.String(), nullable=True),
126+
sa.Column("sc_ppm", sa.String(), nullable=True),
127+
sa.Column("se_ppm", sa.String(), nullable=True),
128+
sa.Column("sn_ppm", sa.String(), nullable=True),
129+
sa.Column("sr_ppm", sa.String(), nullable=True),
130+
sa.Column("ta_ppm", sa.String(), nullable=True),
131+
sa.Column("te_ppm", sa.String(), nullable=True),
132+
sa.Column("th_ppm", sa.String(), nullable=True),
133+
sa.Column("tl_ppm", sa.String(), nullable=True),
134+
sa.Column("u_ppm", sa.String(), nullable=True),
135+
sa.Column("v_ppm", sa.String(), nullable=True),
136+
sa.Column("w_ppm", sa.String(), nullable=True),
137+
sa.Column("y_ppm", sa.String(), nullable=True),
138+
sa.Column("zn_ppm", sa.String(), nullable=True),
139+
sa.Column("zr_ppm", sa.String(), nullable=True),
140+
sa.Column("la_ppm", sa.String(), nullable=True),
141+
sa.Column("ce_ppm", sa.String(), nullable=True),
142+
sa.Column("pr_ppm", sa.String(), nullable=True),
143+
sa.Column("nd_ppm", sa.String(), nullable=True),
144+
sa.Column("sm_ppm", sa.String(), nullable=True),
145+
sa.Column("eu_ppm", sa.String(), nullable=True),
146+
sa.Column("gd_ppm", sa.String(), nullable=True),
147+
sa.Column("tb_ppm", sa.String(), nullable=True),
148+
sa.Column("dy_ppm", sa.String(), nullable=True),
149+
sa.Column("ho_ppm", sa.String(), nullable=True),
150+
sa.Column("er_ppm", sa.String(), nullable=True),
151+
sa.Column("tm_ppm", sa.String(), nullable=True),
152+
sa.Column("yb_ppm", sa.String(), nullable=True),
153+
sa.Column("lu_ppm", sa.String(), nullable=True),
154+
sa.Column("tree_ppm", sa.String(), nullable=True),
155+
sa.Column("mn_pct", sa.String(), nullable=True),
156+
sa.Column("fe_pct", sa.String(), nullable=True),
157+
sa.Column("al_pct", sa.String(), nullable=True),
158+
sa.Column("ca_pct", sa.String(), nullable=True),
159+
sa.Column("na_pct", sa.String(), nullable=True),
160+
sa.Column("k_pct", sa.String(), nullable=True),
161+
sa.Column("mg_pct", sa.String(), nullable=True),
162+
sa.Column("p_pct", sa.String(), nullable=True),
163+
sa.Column("si_pct", sa.String(), nullable=True),
164+
sa.Column("ti_pct", sa.String(), nullable=True),
165+
sa.PrimaryKeyConstraint("id"),
166+
sa.UniqueConstraint(
167+
"source_sheet", "source_row", name="uq_cm_chemical_data_source_row"
168+
),
169+
)
170+
op.create_index("ix_CM_ChemicalData_area", "CM_ChemicalData", ["area"])
171+
op.create_index("ix_CM_ChemicalData_sample", "CM_ChemicalData", ["sample"])
172+
op.create_index(
173+
"ix_CM_ChemicalData_source_sheet", "CM_ChemicalData", ["source_sheet"]
174+
)
175+
176+
op.create_table(
177+
"CM_DetectionLimits",
178+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
179+
sa.Column("source_row", sa.Integer(), nullable=False),
180+
sa.Column("method", sa.String(), nullable=True),
181+
sa.Column("element", sa.String(), nullable=True),
182+
sa.Column("lower_reporting_limit", sa.String(), nullable=True),
183+
sa.Column("unit", sa.String(), nullable=True),
184+
sa.PrimaryKeyConstraint("id"),
185+
sa.UniqueConstraint("source_row", name="uq_cm_detection_limits_source_row"),
186+
)
187+
op.create_index("ix_CM_DetectionLimits_element", "CM_DetectionLimits", ["element"])
188+
189+
op.create_table(
190+
"CM_References",
191+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
192+
sa.Column("source_row", sa.Integer(), nullable=False),
193+
sa.Column("citation", sa.String(), nullable=True),
194+
sa.PrimaryKeyConstraint("id"),
195+
sa.UniqueConstraint("source_row", name="uq_cm_references_source_row"),
196+
)
197+
198+
op.create_table(
199+
"CM_MineralSystems",
200+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
201+
sa.Column("source_row", sa.Integer(), nullable=False),
202+
sa.Column("system_name", sa.String(), nullable=True),
203+
sa.Column("synopsis", sa.String(), nullable=True),
204+
sa.Column("deposit_types", sa.String(), nullable=True),
205+
sa.Column("principal_commodities", sa.String(), nullable=True),
206+
sa.Column("critical_minerals", sa.String(), nullable=True),
207+
sa.Column("references", sa.String(), nullable=True),
208+
sa.Column("phase_2", sa.String(), nullable=True),
209+
sa.Column("phase_3", sa.String(), nullable=True),
210+
sa.Column("phase_4", sa.String(), nullable=True),
211+
sa.PrimaryKeyConstraint("id"),
212+
sa.UniqueConstraint("source_row", name="uq_cm_mineral_systems_source_row"),
213+
)
214+
215+
op.create_table(
216+
"CM_WorldComparisons",
217+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
218+
sa.Column("source_row", sa.Integer(), nullable=False),
219+
sa.Column("area", sa.String(), nullable=True),
220+
sa.Column("deposit", sa.String(), nullable=True),
221+
sa.Column("reference", sa.String(), nullable=True),
222+
sa.Column("la", sa.String(), nullable=True),
223+
sa.Column("ce", sa.String(), nullable=True),
224+
sa.Column("pr", sa.String(), nullable=True),
225+
sa.Column("nd", sa.String(), nullable=True),
226+
sa.Column("sm", sa.String(), nullable=True),
227+
sa.Column("eu", sa.String(), nullable=True),
228+
sa.Column("gd", sa.String(), nullable=True),
229+
sa.Column("tb", sa.String(), nullable=True),
230+
sa.Column("dy", sa.String(), nullable=True),
231+
sa.Column("ho", sa.String(), nullable=True),
232+
sa.Column("er", sa.String(), nullable=True),
233+
sa.Column("tm", sa.String(), nullable=True),
234+
sa.Column("yb", sa.String(), nullable=True),
235+
sa.Column("lu", sa.String(), nullable=True),
236+
sa.Column("tree", sa.String(), nullable=True),
237+
sa.Column("sc", sa.String(), nullable=True),
238+
sa.Column("y", sa.String(), nullable=True),
239+
sa.Column("metric_tons", sa.String(), nullable=True),
240+
sa.Column("grade_pct", sa.String(), nullable=True),
241+
sa.Column("total_ree", sa.String(), nullable=True),
242+
sa.Column("cutoff_grade_pct", sa.String(), nullable=True),
243+
sa.Column("la2o3", sa.String(), nullable=True),
244+
sa.Column("ce2o3", sa.String(), nullable=True),
245+
sa.Column("pr6o11", sa.String(), nullable=True),
246+
sa.Column("nd2o3", sa.String(), nullable=True),
247+
sa.Column("sm2o3", sa.String(), nullable=True),
248+
sa.Column("eu2o3", sa.String(), nullable=True),
249+
sa.Column("gd2o3", sa.String(), nullable=True),
250+
sa.Column("tb4o7", sa.String(), nullable=True),
251+
sa.Column("dy2o3", sa.String(), nullable=True),
252+
sa.Column("ho2o3", sa.String(), nullable=True),
253+
sa.Column("er2o3", sa.String(), nullable=True),
254+
sa.Column("tm2o3", sa.String(), nullable=True),
255+
sa.Column("yb2o3", sa.String(), nullable=True),
256+
sa.Column("lu2o3", sa.String(), nullable=True),
257+
sa.Column("y2o3", sa.String(), nullable=True),
258+
sa.PrimaryKeyConstraint("id"),
259+
sa.UniqueConstraint("source_row", name="uq_cm_world_comparisons_source_row"),
260+
)
261+
262+
op.create_table(
263+
"CM_WorldReferences",
264+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
265+
sa.Column("source_row", sa.Integer(), nullable=False),
266+
sa.Column("citation", sa.String(), nullable=True),
267+
sa.PrimaryKeyConstraint("id"),
268+
sa.UniqueConstraint("source_row", name="uq_cm_world_references_source_row"),
269+
)
270+
271+
op.create_table(
272+
"CM_WorkbookMetadata",
273+
sa.Column("id", sa.Integer(), nullable=False, autoincrement=True),
274+
sa.Column("source_sheet", sa.String(), nullable=False),
275+
sa.Column("source_row", sa.Integer(), nullable=False),
276+
sa.Column("label", sa.String(), nullable=True),
277+
sa.Column("value", sa.String(), nullable=True),
278+
sa.PrimaryKeyConstraint("id"),
279+
sa.UniqueConstraint(
280+
"source_sheet", "source_row", name="uq_cm_workbook_metadata_source_row"
281+
),
282+
)
283+
284+
285+
def downgrade() -> None:
286+
op.drop_table("CM_WorkbookMetadata")
287+
op.drop_table("CM_WorldReferences")
288+
op.drop_table("CM_WorldComparisons")
289+
op.drop_table("CM_MineralSystems")
290+
op.drop_table("CM_References")
291+
op.drop_index("ix_CM_DetectionLimits_element", table_name="CM_DetectionLimits")
292+
op.drop_table("CM_DetectionLimits")
293+
op.drop_index("ix_CM_ChemicalData_area", table_name="CM_ChemicalData")
294+
op.drop_index("ix_CM_ChemicalData_sample", table_name="CM_ChemicalData")
295+
op.drop_index("ix_CM_ChemicalData_source_sheet", table_name="CM_ChemicalData")
296+
op.drop_table("CM_ChemicalData")

‎cli/cli.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1359,6 +1359,40 @@ def import_project_area_boundaries_command(
13591359
)
13601360

13611361

1362+
@cli.command("load-critical-minerals-workbook")
1363+
def load_critical_minerals_workbook_command(
1364+
file_path: str = typer.Option(
1365+
...,
1366+
"--file",
1367+
exists=True,
1368+
file_okay=True,
1369+
dir_okay=False,
1370+
readable=True,
1371+
help="Path to the McLemoreMasterChem .xlsx critical-minerals workbook.",
1372+
),
1373+
):
1374+
"""
1375+
mirror the McLemore critical-minerals workbook into the CM_legacy staging
1376+
tables. Every sheet is copied unchanged -- including the ChemicalData, GIS
1377+
and QAQC sheets, which disagree with each other -- and each sheet's rows
1378+
replace whatever was loaded for it before. Values are not parsed; see
1379+
docs/critical-minerals-legacy-mirror.md.
1380+
"""
1381+
from db.engine import session_ctx
1382+
from services.cm_legacy_mirror import load_cm_workbook
1383+
1384+
with session_ctx() as session:
1385+
result = load_cm_workbook(file_path, session)
1386+
session.commit()
1387+
1388+
typer.echo(f"Mirrored {result.total_rows} row(s) from {file_path}:")
1389+
width = max(len(sheet) for sheet in result.rows_by_sheet)
1390+
for sheet, count in result.rows_by_sheet.items():
1391+
typer.echo(f" {sheet:<{width}} | {count:>6}")
1392+
for warning in result.warnings:
1393+
typer.echo(f"warning: {warning}", err=True)
1394+
1395+
13621396
if __name__ == "__main__":
13631397
cli()
13641398

‎db/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@
5858
from db.thing_aquifer_association import *
5959
from db.thing_geologic_formation_association import *
6060
from db.aquifer_type import *
61+
from db.cm_legacy import *
6162
from db.nma_legacy import *
6263
from db.nmw_legacy import *
6364
from db.transducer import *

0 commit comments

Comments
 (0)