From ef6058bc99e40cf372072cf04b8600d2f10b7d3c Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Fri, 11 Sep 2026 22:18:46 -0500 Subject: [PATCH] Correct the scaling claim the case study could not support The case study said the 10-year run cost ~46 s per year of data, "dead linear, which confirms I/O bound with no meaningful per-call overhead." Its own two measurements say otherwise: one year takes 29.1 s and ten take 464.4 s, so a line through them has a negative intercept and each year after the first costs about 48 s. Throughput agrees -- 129 MB/s on the single-year read against 81 MB/s across ten. The likely cause is a warm page cache on the 1979 file, which was read repeatedly while the case study was written, so both numbers are now given with ~48 s/year as the one to plan with and the cache named as a suspicion rather than a result. Two other numbers were wrong. The advertised tool count is 33, not 31, in both write-ups. The demo page still carried a 112 KB PNG and a 320,000:1 ratio from an earlier render; the measured artifact is 177,737 bytes, so the ratio is ~210,000:1. Adds what the case study was missing: the five stages the worker runs, pointers to the full tool menu and the architecture doc, and a reproducing section covering scripts/reproduce_conus_case_study.py and what a rerun showed. That rerun landed on the same PBS job as the original, so its byte-identical PNG is a determinism result and not evidence of reproducibility across allocations -- which the new section says outright. Also links the case study from the root README, where it was not mentioned at all. --- README.md | 9 ++ .../conus-precipitation-gdex/README.md | 95 ++++++++++++++++--- docs/demo-gdex-conus-precip.md | 26 +++-- 3 files changed, 111 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 7e8cd05..1b3efa3 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,15 @@ You are most likely one of: 5. **Just trying it out, or running an agent harness** — you don't want to install a scientific Python stack at all. → [Docker](#docker). +**Want to see what this is for before installing anything?** Read the worked +case study: [**a 10-year CONUS precipitation map from 37 GB that never +moved**](case-studies/conus-precipitation-gdex/README.md). One paragraph of +plain English on a laptop, ten years of 6-hourly CESM output on the NCAR +filesystem, all the compute on a Casper worker, and a 178 KB PNG plus an +auditable provenance record coming back. It covers what MCP is, what actually +ran where, why it took 7.7 minutes, what it cost in tokens, and how to set the +same thing up yourself. + --- ## Docker diff --git a/case-studies/conus-precipitation-gdex/README.md b/case-studies/conus-precipitation-gdex/README.md index 9236a95..202f6fe 100644 --- a/case-studies/conus-precipitation-gdex/README.md +++ b/case-studies/conus-precipitation-gdex/README.md @@ -73,6 +73,7 @@ No data was downloaded. The laptop never held more than a 178 KB PNG. - [What it cost](#what-it-cost) - [Set it up yourself](#set-it-up-yourself) - [Methods: what was used](#methods-what-was-used) +- [Reproducing it](#reproducing-it) - [Honest limitations](#honest-limitations) --- @@ -257,11 +258,25 @@ so a wrong plot is *falsifiable* rather than merely pretty. 4. **The function is serialized and shipped.** Globus Compute's `AllCodeStrategies` sends the *source* of the worker function, so the code does not have to be pre-installed on Casper. -5. **A Casper worker runs it.** It opens the ten files with - `ux.open_mfdataset`, subsets to the CONUS bounding box, takes the time mean, - scales to mm/day, renders a choropleth, and overlays Natural Earth coastlines - from the worker's cartopy cache. -6. **A PNG and a JSON record come back.** ~180 KB total. +5. **A Casper worker runs it.** One function, five stages, all inside the NCAR + filesystem boundary: + + | # | stage | what it does | + |---|---|---| + | 1 | open | `ux.open_mfdataset` over the ten annual files and the SCRIP mesh | + | 2 | subset | `Grid.subset.bounding_box([-125,-67], [24,50])` — **before** the reduction, cutting 777,602 cells to 23,510 | + | 3 | reduce | mean over 14,600 time steps, then `× 86,400,000` for m/s → mm/day | + | 4 | render | HoloViews `polygons()` on the matplotlib backend, plus Natural Earth coastlines, borders and state lines from the worker's cartopy cache | + | 5 | package | `min` / `mean` / `max`, `n_nonfinite`, `n_face_subset`, image byte count, and the PBS provenance block | + +6. **A PNG and a JSON record come back.** ~180 KB total, in one response. + +That is one tool call, not a pipeline you assembled. The server publishes 33 +tools in total — vector calculus (curl, divergence, gradient), remapping and +regridding including YAC, zonal means and anomalies, cross-sections, subsetting, +session and result handles. [`docs/tools.md`](../../docs/tools.md) is the full +menu; [`docs/architecture.md`](../../docs/architecture.md) explains how a front +door like `plot_dataset` dispatches to the operation underneath. ### The one design decision that matters @@ -317,8 +332,19 @@ Three reasons, in order of size: [`docs/ucar.md`](../../docs/ucar.md)). Login-node I/O is shared and throttled; compute nodes read faster. -Marginal cost is **~46 s per year of data**, dead linear, which confirms I/O -bound with no meaningful per-call overhead. +**The per-year cost is not constant, and the gap is worth naming.** One year +takes 29.1 s; ten take 464.4 s. That is 16× the wall clock for 10× the data — +about 29 s for the first year against roughly 48 s for every year after it. Fit +a straight line through the two points and the intercept comes out *negative*, +which is not what fixed per-call overhead looks like. Throughput tells the same +story from the other side: 129 MB/s on the single-year read, 81 MB/s across ten. + +The likely explanation is the page cache. The 1979 file was read again and again +while this case study was being built, so its 29 s is a warm number and the +ten-year figure is closer to the cold cost. That is a suspicion and not a +measurement — nothing here dropped the cache between runs. Plan with **~48 s per +year**; treat 29 s as what you get on a file the filesystem already holds in +memory. **If you need it faster:** compute the ten annual means once and average those — the repeat becomes seconds. Or parallelize the file reads. Or move the provider @@ -364,14 +390,17 @@ costs are worth separating, because only one of them is interesting. | item | approximate tokens | |---|---| -| **standing cost:** the uxarray tool menu, 31 tools, sent on *every* request | **~10,600** | +| **standing cost:** the uxarray tool menu, 33 tools, sent on *every* request | **~10,600** | | the paragraph you typed | ~200 | | the returned map (1000×560 image) | ~750 | | the returned metadata + provenance JSON | ~750 | -**The standing cost dominates.** Publishing 31 tools means ~10.6k tokens of -schema ride along with every message, whether or not you use any of them. The -per-call cost of actually doing the science is small by comparison. +**The standing cost dominates.** Publishing 33 tools means ~10.6k tokens of +schema ride along with every message, whether or not you use any of them +(counted with `tiktoken`, `cl100k_base`, over the registered tool list — not +estimated). Nearly half of that is tool *descriptions*: full docstrings with +`Args:`, `Returns:` and `Examples:` sections, shipped raw. The per-call cost of +actually doing the science is small by comparison. Two honest consequences: @@ -571,6 +600,50 @@ standard approximations, not billing records. --- +## Reproducing it + +Every number above is produced by a script in this repository, so you do not +have to take the article's word for any of them: + +```bash +uv run python scripts/reproduce_conus_case_study.py --act 1 # capabilities, ~3-20 s +uv run python scripts/reproduce_conus_case_study.py --act 2 # 1-year mean, ~30 s +uv run python scripts/reproduce_conus_case_study.py --act 3 # 10-year mean, ~7.8 min +uv run python scripts/reproduce_conus_case_study.py --all +``` + +It asserts the published values rather than printing them, so a drift in the +data, the mesh, or the reduction fails the run instead of quietly producing a +different map. You need the Casper endpoint configured as in +[Set it up yourself](#set-it-up-yourself); Act III needs `timeout_seconds: 2400`. + +### What a rerun actually showed + +Run on 2026-09-11/12 against `ucar-uxarray-yac`: + +| quantity | published | rerun | +|---|---|---| +| faces / nodes / edges | 777,602 / 780,456 / 2,329,471 | identical | +| time steps in the 10-year mean | 14,600 | 14,600 | +| CONUS subset | 23,510 of 777,602 | 23,510 of 777,602 | +| min / mean / max mm/day | 0.111 / 2.232 / 8.584 | 0.110901 / 2.232262 / 8.584175 | +| non-finite values | 0 | 0 | +| PNG size | 177,737 B | 177,737 B, `cmp` byte-identical | +| wall clock, 10 years | 462.1 s | 467.9 s | + +> [!IMPORTANT] +> Read that table for what it is. The rerun carries the same PBS job ID, +> `5890924.casper-pbs`, as the original — it landed on the same worker process, +> on the same node, with the same library versions and very likely the same page +> cache. A byte-identical PNG under those conditions demonstrates that the +> pipeline is **deterministic**. It is not evidence that the result reproduces +> across a fresh allocation, a different node, or another cartopy or matplotlib +> build, and the byte-for-byte match in particular would be the first thing to +> break on any of those. The scientific values — cell counts, step counts, +> statistics — are the durable part. + +--- + ## Honest limitations Things a demo usually hides. They are here because a case study that only shows diff --git a/docs/demo-gdex-conus-precip.md b/docs/demo-gdex-conus-precip.md index e0209df..74eaaec 100644 --- a/docs/demo-gdex-conus-precip.md +++ b/docs/demo-gdex-conus-precip.md @@ -4,6 +4,11 @@ Run-of-show for demonstrating the uxarray MCP server against data that never leaves NSF NCAR. Every number below was measured on 2026-09-11 against endpoint `ucar-uxarray-yac` (casper02). +This page is the presenter's run-of-show. The same run written up for a reader — +with the MCP background, the setup walkthrough, and the limitations — is the +worked case study at +[`case-studies/conus-precipitation-gdex/README.md`](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/case-studies/conus-precipitation-gdex/README.md). + --- ## The claim being demonstrated @@ -15,7 +20,7 @@ leaves NSF NCAR. Every number below was measured on 2026-09-11 against endpoint Three properties the audience should walk away with: 1. **The data never moves.** 10 files x ~3.75 GB = ~37 GB of 6-hourly CAM output stays - on the GDEX filesystem. What crosses the wire is a ~110 KB PNG and a JSON record. + on the GDEX filesystem. What crosses the wire is a ~178 KB PNG and a JSON record. 2. **The compute is where the data is.** uxarray runs on a casper worker via Globus Compute. The laptop has no uxarray-scale memory and never needs it. 3. **Every answer is auditable.** Each call returns `_provenance`: tool name, argument @@ -244,33 +249,38 @@ regression test — a longitude-convention bug would put the Sierra in Kansas. | capability query (Act I) | ~20 s | | 1 year, direct compute function | 29.1 s | | 1 year, full MCP front door | 51.4 s / 77.5 s | -| 10 years, full chain | 464.4 s | +| 10 years, direct compute function | 464.4 s | +| 10 years, full MCP front door | 462.1 s | | cold worker penalty (PBS queue) | add 1-4 min if not pre-warmed | -Per-year marginal cost is ~46 s, so the 10-year run is dominated by I/O over the -~37 GB of source data, not by the mesh operation. +The run is I/O bound on the ~37 GB of source data, not on the mesh operation. +Do not read the two rows as a linear scaling law: one year costs 29.1 s and ten +cost 464.4 s, so each year after the first runs about 48 s, not 29 s. The +single-year file was read repeatedly while this demo was built and is almost +certainly cache-warm, which is a suspicion rather than a measurement. Quote +~48 s per year. ### Bytes over the wire | direction | payload | |---|---| | laptop → casper | the serialized function + arguments, a few KB | -| casper → laptop | 112 KB PNG (base64: ~150 KB) + ~3 KB JSON | +| casper → laptop | 177,737-byte PNG + ~3 KB JSON | | what stayed put | ~37 GB of 6-hourly CAM output | -**This ratio is the demo.** ~37 GB read, ~115 KB returned — a factor of ~320,000. +**This ratio is the demo.** ~37 GB read, ~180 KB returned — a factor of ~210,000. ### Token cost (approximate, per request) | item | tokens | |---|---| -| uxarray MCP tool schema, 31 tools | ~10,600 on *every* request | +| uxarray MCP tool schema, 33 tools | ~10,600 on *every* request | | returned map, 1000x560 | ~750 image tokens | | returned metadata + provenance JSON | ~750 | | the prompt you paste | ~200 | Honest framing for the room: the tool schema dominates. The interesting per-call -cost is small; the standing cost of having 31 tools registered is not. This is a +cost is small; the standing cost of having 33 tools registered is not. This is a real argument for the front-door design — `run_analysis` and `plot_dataset` cover most of the surface, and a narrower registered tool list would cut the standing cost several-fold.