Fix ICON unstructured-grid GRIB loading by bypassing earthkit grid en…

[icedoom888]

What this fixes

Reading ICON-CH1/CH2 GRIB (baselines and ML-forecaster output) crashed during verification with a misleading TypeError: 'ValueError' object is not callable. This makes verification_metrics_baseline and forecaster verification work again.

Root cause

ICON GRIB is on an unstructured (triangular) grid (gridType=unstructured_grid, GRIB2 template 101). earthkit-data v1.0's to_xarray unconditionally builds a TensorGrid, which for unstructured fields needs the GRIB gridSpec. On our cluster gridSpec can't be obtained: the bundled eckit-geo codec can't parse ECMWF's ICON .ek grid file (eckit::codec::InvalidRecord: version not found) and the auto-downloader is broken. The underlying ValueError is then masked by an earthkit lazy-handler bug, surfacing as the confusing TypeError. No to_xarray profile flag avoids the grid build, and meteodatalab.grib_decoder is import-incompatible with this earthkit version.

Fix

Detect unstructured fields by gridType and route only those through a new _unstructured_fieldlist_to_xarray helper that builds the Dataset directly from field values + metadata and attaches latitude/longitude from the local ICON grid file (meteodatalab.icon_grid.load_grid_from_balfrin, imported lazily). Regular grids keep the existing earthkit path, untouched.

Key property: coordinates come from the local grid file, so there is no runtime network dependency and no eckit-geo involvement — which is what makes it robust on compute nodes.

The unstructured output is structurally identical to the regular path (flat values dim, forecast_reference_time/step dims, valid_time coord, {param}_{level} naming for pressure levels, global attrs), so all downstream code — map_forecast_to_truth, the metrics, and TOT_PREC de-accumulation — is unchanged.

Scope

• Touches only src/data_input/__init__.py (load_from_grib_file + new helpers).
• Fixes both ICON baselines and forecaster verification (ML outputs are ICON-unstructured too).
• workflow/scripts/data_extract_baseline.py has separate, unrelated breakage on this earthkit version and is intentionally left out of scope; the new helper is reusable for a follow-up.

Testing

Validated on balfrin against a full evalml experiment run (ICON-CH1/CH2 hindcast, Jan–Jun 2025):

• Before: every verification_metrics_baseline job crashed on ICON-CH1/CH2 GRIB with the masked TypeError.
• After: all 342/342 baseline verification jobs succeed and produce metrics; the same path also reads the forecaster ICON GRIB output (forecaster verification produces metrics over the full run).
• Standalone checks: ICON-CH2 (0/120/6) and ICON-CH1 (0/33/6) baselines load with finite latitude/longitude and physically sensible fields; end-to-end verification_metrics.py against KENDA-CH1 yields the expected lead-time skill degradation (e.g. T_2M MAE 0.48 K @ step 0 → 1.42 K @ step 120, CORR 0.985 → 0.923); TOT_PREC de-accumulates with no spurious negatives.
• A regular rotated_ll GRIB still routes through the unchanged earthkit path; existing tests/unit/test_data_input.py pass.