Reach YAC remapping from run_analysis and fix what a first run trips on - #144
Merged
Merged
Conversation
The SDK backend requested a data_access scope for the local collection, which Globus Connect Personal collections do not have, so every transfer failed once a local collection was configured. The CLI already tracks which collections need which scopes, so this holds no auth code at all and `uxarray-mcp transfer setup` walks the seven prerequisites instead.
Observed a task report nice_status TIMEOUT on STOR and then finish SUCCEEDED eighteen minutes later, so the previous advice to blame the VPN and retry would have abandoned a transfer that was working.
Driving the server as a new user against the local uxarray test meshes and the UCAR endpoint turned up one missing feature and ten defects. The feature: YAC was documented and installed on the ucar-uxarray-yac worker but unreachable from any client. run_analysis now takes backend and yac_method, and method accepts conservative, nnn, dnn and average directly. One resolver in domain/remap_backend.py serves the local path; the worker payloads inline the same rules under a drift test. Coverage records the method it judged, so the not-conservative warning names the method that ran. Verified locally (YAC 3.20) and on casper: identical field means. The defects: gradient on a grid without sphere_radius was unreachable, each repair pointing at the other, and acknowledging the refusal failed the output schema (status "overridden" was not in the enum); sphere_radius now reaches gradient/curl/divergence locally and remotely, with a radius_basis block. The azimuthal r=0 ring was counted as a missed bin. The variable-plot colorbar sat over the map. analyze_dataset inlined 82 KB of base64 into a JSON object. Remote remap returned no source_coverage. Remote analyze recommended operations that refuse use_remote. lat_step gives JSON callers the (start, stop, step) tuple. mesh_geo options are exposed on plot_dataset and the plot note names them. README: extras and the plain uv sync footgun, install-claude --config-path, 33 tools, missing CLI rows, sphere_radius instead of scale_by_radius=False.
Through the MCP transport the variable plot still arrived as 78 KB of base64 inside the summary: the local plot path records the stored file only under _provenance.artifacts and never set image_uri for an inline figure, so the summary saw no URI and kept the bytes. Both the plot metadata and the summary now read the URI off the artifact record. The summary dropped from 87,663 to 9,592 bytes on outCSne30.
Every existing plot type reads one file and draws the whole mesh, so a multi-year regional mean had no path through the server. temporal_mean, subset_bbox and anomaly all refused use_remote=True, and the advice they gave was to pass a locally-readable path -- not available when the data lives on someone else's filesystem, which is the case this endpoint exists for. remote_temporal_mean_map opens the files with open_mfdataset, applies the bounding box before the reduction so only the kept cells are carried through it, scales units, and renders a choropleth with Natural Earth geography when the worker has cartopy. It reports n_time_steps, reduced_dims, n_face_subset against n_face_total and n_nonfinite: a PNG cannot say what was averaged away, so the response has to. plot_type="variable" now refuses lon_bounds/lat_bounds rather than ignoring them. It never honored a box, but it accepted one and returned a global map with nothing saying the box had been dropped, so a regional request came back looking answered. The refusal names the plot types that do honor a box. Verified on the UCAR worker: ten annual files of 6-hourly CESM ne120 PRECT, 14,600 steps reduced to a CONUS map in 462 s, matching a direct call to the compute function to sixteen digits.
The three site scripts hardcoded USERNAME="jain" at the top under a banner telling the reader to edit it, so the first run by anyone else either failed on a path they did not own or, worse, worked against somebody's directory. Usernames now default to $USER and every site detail -- conda env, endpoint name, YAC version -- takes an environment override, so nothing in the files needs editing. YAC becomes opt-in via WITH_YAC, defaulting off. It buys conservative remapping and nothing else, needs a separate build, and its absence is already reported rather than crashed on, so requiring it to bring an endpoint up was a barrier for no gain. configure refuses a profile named for YAC unless WITH_YAC=1, since a profile named ucar-uxarray-yac that came up without it would look healthy and quietly lack the one method it is named for. scripts/endpoint.sh is a site-agnostic version that assumes no facility, for people standing up a first endpoint; each site doc now leads with it and keeps the site-specific script as the tuned path. Both notes claiming the repo need not be cloned on the cluster are corrected: it must be, because the scripts live in it. What must not happen is uxarray_mcp landing on the worker's PYTHONPATH.
case-studies/conus-precipitation-gdex/ follows one plain-English prompt through to a map: what was asked, the tool call it became, the result, the provenance block, and the numbers. Ten annual files of 6-hourly CESM ne120 PRECT from GDEX d651007, 14,600 steps, 23,510 of 777,602 cells, 462 s on casper02, ~37 GB read and 178 KB returned. Written for people who have the data and the allocation but have not run an MCP server before, so it explains what MCP is before assuming it, and carries laptop and Casper setup end to end. It records what usually gets left out: the run is I/O bound at roughly 81 MB/s and the CONUS subset saves no reading at all, since the kept cells are scattered through ncol and chunks are read whole; the reported mean is over the bounding box and includes ocean; the ensemble member is .002 because .010 is not staged in the six-hourly tier. Timings and cell counts are measured, token figures are estimates, and the text says which is which. docs/demo-gdex-conus-precip.md is the run-of-show for presenting it live, including the two config mistakes that cost the most time: an endpoint timeout of 300 s against a 462 s job, and path_prefixes missing /gdex/, which routed GDEX paths to a cluster at another facility.
test_every_dispatch_offers_the_same_input_kinds counted two substrings and required the counts to match. That tied the guard to formatting: an extension list the formatter wrapped over three lines read as zero shapefile branches, and a nested if naming the HEALPix prefix twice read as two HEALPix branches. remote_temporal_mean_map is both, so it was reported as offering HEALPix without shapefiles while opening shapefiles two lines away. The same miscount could cancel out and let a genuine omission through. Read the branch kinds off the AST and assert both are present, which is the property the docstring already claimed to protect. A new test pins the guard itself: it must flag a payload that drops the shapefile read, and must judge the one-line and wrapped spellings of the extension test alike. Bump the three deliberate copy counts for the new remote function, and ship the case-study reproduction script with its unused import, formatting and two None-safety errors fixed.
The docs job builds with `-W`, so two warnings from the new demo page failed it. The page was never added to a toctree, and its `_provenance` excerpt was fenced as `json` while being neither a complete object nor valid JSON -- the `...` placeholders that make it readable are what the lexer choked on. Fenced as `text`, which is what it is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drove the server as a regular user against the local
~/uxarray/test/meshfilesset and theucar-uxarray-yacendpoint, then fixed everything that failed or misled.Feature: YAC remapping was unreachable
docs/ucar.mdpromised conservative remapping and YAC imports fine on casper (check_yacgreen), butrun_analysisnever forwardedbackend, andremap_variable/regrid_datasethad no such argument. Now:run_analysistakesbackend(uxarray|yac) andyac_method(nnn|dnn|average|conservative);method="conservative"etc. selects YAC on its own.domain/remap_backend.py) for the local path; worker payloads inline the same rules under a test that compares the two.method="conservative"(1.0000000019841813) onoutCSne30 -> QU 1920km. Known upstream limit surfaced with a hint:to_rectilinear(backend="yac")fails whentarget_lonspans 360°.Defects fixed
gradienton any grid withoutsphere_radiuscould not be obtained: repair A said passscale_by_radius=False, which was refused by repair B saying set itTrue.sphere_radiusnow reachesgradient/curl/divergence(local + remote) and is reported inradius_basis; both repairs name it.acknowledgecrashed the envelope:preconditions.status="overridden"was not in the output-schema enum (MCP error -32602).remap_variable/regrid_datasetreturned nosource_coverage.analyze_datasetinlined 82 KB base64 into a JSON object; stored figures are now referenced by URI.PROFILE_COVERAGE_PARTIAL.mesh_geoplot note told callers to "ask" for optionsplot_datasetdid not expose; they are exposed and named.analyze_datasetrecommendedsubset_bbox/cross_section, which refuseuse_remote.(start, stop, step)latitude range: addedlat_step.Docs
README:
--extra hpc --extra transferand the plain-uv sync-removes-them footgun (howNo module named 'globus_compute_sdk'happens),install-claude --config-path, 33 tools not 31, CLI rows foropenapi/endpoints remove/transfer setup,sphere_radiusinstead of the now-refusedscale_by_radius=False, clone-pathuv --directory ... runcommand, env vars.docs/tools.md: remap backends and every new parameter. CHANGELOG updated.Tests
uv run pytest: 1254 passed. New:tests/test_remap_backend.py,tests/test_regular_user_fixes.py.pre-commit run --all-filesclean.I have tested and take responsibility for all content in this PR.